There are many recipes for preparing CI / CDs on the Web for solving various problems and organizing processes for specific needs. In this article, we will describe another one, the essence of which is to prepare a process as close as possible to the classical approach, despite the fact that it is intended for the development of QCD, and to solve the problem of organizing the work of a large team.

Goals

• Save developers from the endless transfer of functionality from one environment to another.
• Create a versioning system to solve the problem of dependency in QCD development.
• The process must be ACID-compliant, which means that a faulty patch must not break the entire repository.
• Development standardization.
• Organization of a repository of patches for Continuous Delivery.
• Organization of the work of a large team.

We decided on the problems. Now we need to find solutions for them.

Technology stack

As a result, the following configuration was chosen:

We chose GitLab as our repository. Oracle as a DBMS, SonarQube is a static code analyzer, Ansible is a configuration manager that allows you to parallelize operations across multiple instances. Nexus Repository is an artifact repository. Finally, Jenkins manages all of this.

Before you start working on automation, you need to bring all patches to a single format. The main requirements are:

• Rigid directory structure. The division into different environments, schemes, and types of objects in them.
• Makefile specifying the instance to install and the path to the script.
• A file with dependencies for each patch in the form of a list of other patches that the current one depends on.
• Bash script for installing a patch on any environment. This script makes a snapshot of the databases that are affected by the patch, through sqlplus it executes all the scripts that are specified in the makefile, and if any of them was executed with an error, then it returned the database to the state before installation.
• And the required release_notes.

Also, to develop a competent CI / CD process, it was necessary to deploy several contours:

• DEV – Patches are developed in this circuit.
• CI – this circuit is used to check the validity of patches.
• TEST – functional testing happens here.
• PrePROD is a circuit in which the customer conducts his own testing before sending it to prod.
• PROD is a productive environment.
• HOTFIX is a complete copy of sales for critical bug fixes.

Continuous integration

We decided on the stack, the patches led to the same form. Now you can tackle automation.

This part of the process has two stages: verification and installation. For each stage, a separate pipeline has been created in Jenkins. A test pipeline is triggered by a Merge Request trigger in GitLab, picks up the branch they want to merge, and tries to install the patch. Before installation, a restore point is created in the database, this is a snapshot of the database, to which it returns when an error occurs when installing the patch. Rollback is performed automatically after each unsuccessful patch installation.

Installation can be performed on one or several instances. This happens in parallel using Ansible. The scripts are executed by means of the sqlplus utility.

At the stage of checking the patch, the source code is checked by means of SonarQube, and if the patch is installed without errors, then the code reviewer receives a notification that the patch is valid and can be sent to the develop branch.

The next step is installation. When we realized that the patch is ready to be formed and transferred to functional testing, we need to finally install the changes in this patch in the CI loop and prepare for installation on other environments.

At this stage, the trigger is the execution of the Merge Request and the process goes on without any checks. A zip archive is generated and sent to the Nexus artifact repository.

Versioning system

For this process to work, we need to understand at the time of installation what state the contour is in, what objects it contains, and which ones need to be installed. A patch can contain a view or a function that refers to a table that is created in another patch, and before installing, we need to check if it is installed.

Since the dependencies apply to the entire contour, then it is necessary to check the dependencies in the entire contour.

After installing each patch, we make an entry into the historical table. Thanks to this simple solution, we know which patches are installed in each circuit.

Continuous Delivery

So we solved two more problems: excluded the developer from the patch generation process and made a system in which the process can control the implementation of patch dependencies. Now we need to automate the transfer of this patch to further environments.

Now the functionality we have developed lies in the Nexus artifact repository. We need to push it further into the FT circuit. Each patch has a timestamp there, by which we determine the most recent version of the patch. The process is simpler and faster than in CI, the source and destination are different and there are not many checks.

After testing, the patch goes either to the register ready for transfer to the customer or to the beginning of our process for revision.

Telegram notifications

The work is already going much faster, patches walk along the contours at an incredible speed, the team does not waste time at all and does their own thing, namely, they overwrite each other's functionality, create conflicts in the GitHub, and spam the code reviewer with requests to freeze their changes.

The above problems led to the fact that developers' hours of work were lost during rewriting, DevOps wasted time resolving conflicts in the GitLab, and the code viewer was looking for the necessary merge request among the merge requests.

The simplest solution would be to notify developers and code reviewers via mail. We have done. These emails were ignored at best and sent straight to the trash can by the soulless Outlook rules at worst. Everyone continued to use Telegram as a subject of coordination. Then we decided to inform them there.

The telegram bot seemed to us to be an excellent solution to this problem, although, in reality, it did not immediately become such. Its development was also associated with difficulty – the entire stack was on the corporate network. We were lucky to have internet access there.

We solved this problem by placing the code of the executable bot (python) inside the corporate network.

In the first version, the bot simply informed the developers about the build results. It lasted until it was required to organize a queue to work on one branch.

This is how the second version appeared, which was tied to a database with team members and could independently update branches from GitLab to organize a queue to it.

The principle of interacting with the bot was extremely simple. The developer contacts the bot, choosing from the list the number of the branch that he wants to take into work. And the bot answers him whether someone is working on it now or not. If not, then he offers to take her to work, and if she is busy, he offers to get in the queue (waiting in the queue rarely exceeded 30 minutes). The bot will notify you when the branch is free.

As far as security is concerned, we tried not to insert any specific data into messages. They were purely technical in nature, and the bot itself is not particularly talkative. He is only allowed to respond to members of the development team.

Outcome

This solution helped developers to do development, testers to do testing, code reviewers to review the code and monitor its quality. That is, they removed the person from the code delivery process, which significantly reduced the number of errors.

Thanks to the versioning system, it is always clear what state this or that circuit is in, but for a person, it is not even that important. If he wants to bring the contour to a certain form, then he will understand what needs to be installed for this.

Since the setting on the contours is the same, then when adding new ones, this system is easily scalable. It is just the contour for installation that changes.

We no longer have conflicts in GitLab. All developers use Telegram and do not interfere with each other's work.

We have accelerated the delivery of functionality, which means we have reduced the time to market due to the early detection of errors.