There are a lot of materials on how to establish continuous delivery, but there are not enough articles on why this is necessary.

Let’s assume that the goal of a normal person’s life is to write more interesting code and throw it into production. From this perspective, the importance of continuous delivery is clear. Alas, it turned out that there are completely different people (you can recognize them by such strange expressions as “product quality”, “resources”, “speed of error correction”, “labor costs”), which are alien to normal values. To make it easier to reach them and to always have a short memo on defending the only correct point of view at hand, and this text was written.

Okay, in some situations it is really nearly impossible to arrange continuous delivery. You yourself will understand this when you find yourself in a similar situation. Just in case, near the end of this material, there are cases in which the benefits of continuous delivery clearly do not cover the cost of organizing it.

Definitions

In this material, we will use the following three definitions.

Continuous Integration – when a product is regularly (several times a day) built from the source and a substantial part of the automated tests is run against it, for example, all unit tests. If automated tests run for a long time, then they can be run less often, for example, once a day. The standard approach for organizing continuous integration is to start TeamCity or Jenkins, which will download the source code from the source control system, build it and run tests. Other notable solutions: Travis CI, GitLab, Space, GitHub, BitBucket.

Continuous Delivery – the product is always in an "assembled" state and ready to be put into production, even taking into account the latest changes made by the developers to the code. Most likely, continuous delivery will be implemented using the same technologies like continuous integration.

Continuous Deployment – when product updates are regularly (for example, after each change made by developers) are put into production, and all the processes of the domain continue to work without interruption. In a way, continuous deployment is a highly automated and frequently performed continuous delivery.

It seems that in a professional environment, they often use the generalizing all of the above term CI / CD with the necessary clarifications, so as not to think about which of the three terms includes regular automatic product deployment in development and testing environments.

Two villages

Let's look at the example of two villages Villarriba and Villabajo: both villages live in typical rural labor – they produce software. The residents of the first village put their product updates into commercial operation every day, well, maybe, except for Fridays (that is, they practice continuous delivery or deployment), and the residents of the second one every two weeks. Using their example, we will consider several parameters of the standard process of manufacturing and maintaining software products. This will help you better understand what the approaches are.

Error elimination speed

Villabajo residents roll out changes every two weeks. For two weeks they make a lot of new functionality, so when they make a mistake in one of the new products, they either have to 1) roll back all the changes at once and, if necessary, pick out the defective change from the version, build the version again and install it (in a hurry, of course), or 2) poke around at once in all the changes that they rolled out, try to understand which of these changes led to errors, and debug it "in battle".

Villarriba residents are ready to roll out changes at least a few times a day. The more often they roll out updates, the fewer changes they accumulate. The fewer changes are simultaneously put into commercial operation, the easier it is to roll back erroneous changes or, if possible, correct them directly in the working product, because the changes are isolated from each other and, therefore, have less impact on each other.

It would seem that Villarriba residents make mistakes as often as Villabajo residents, so the number of defects in an industrially exploited product is the same! However, Villarriba residents, on average, fix each defect faster than Villabajo residents, because it is easier for them to understand the causes of the error, it is easier for them to roll back erroneous changes and it is easier to roll out the fixed version.

But here, too, not everything is so simple: the residents of Villabajo do not want to install updates, often not because they are so angry, but because they have a poorly automated process for publishing their product. As a result, unlike the Villarriba residents, they also suffer from product bugs caused by an incorrect version installation.

Installation costs of the new version

Villabajo residents roll out changes every two weeks. Unlike serious b2b offices, the residents of Villabajo really have a clearly planned volume of each sprint, and they also learned how to check whether everything they need was included in the version (and whether garbage got there), in a semi-automatic mode.

However, every Villabajo resident regularly spends several hours to:

• check the version for the completeness of the included changes (yes, in semi-automatic mode), include in the version what you forgot and throw out what got by mistake, agree with the rest of the villagers and with the customer that it's time to install the update;
• do what you need to do before installing the version, for example, run the "scripts", debug and fix them, because they were not completely ready to run "in battle";
• do what needs to be done during the installation of the version, for example, run the "scripts", debug and fix them, because they were again not completely ready to be launched "in battle";
• do what needs to be done after installing the version, for example, run the "scripts", debug and fix them, because for the third time they were not completely ready to run "in battle".

Villarriba residents are ready to roll out changes once a day. They have to keep the product ready for commercial operation almost all the time, taking into account the latest updates. They do not have “scripts” to run before, during, or after the new version is installed. They are developing in such a way that they can introduce new functionality gradually, one small controlled block per version. It makes no sense for Villabajo residents to do this: they have the same version every two weeks.

What is the bottom line? Villabajo residents spend every two weeks the first half of their working day installing the version, and the second half watching YouTube because after stress they can not work productively. Villarriba residents work, as usual, only sometimes they are distracted to roll back the latest changes from the combat server.

The effect under consideration – reducing the cost of installing the version – is quite obvious and is confirmed by real cases. For example, continuous delivery helped one of the Hewlett-Packard teams reduce overall development costs by 40%.

One study shows how continuous delivery eliminated the need for company engineers to spend days preparing and installing a version. This effect, obviously, also includes the decrease in the cost of fixing errors noted in the same place in the very process of preparing and installing the version.

In another study, some of the companies surveyed cited a reduction in the amount of manual labor required to manage the entire development and delivery cycle as a significant advantage of “continuous approaches”.

Enabling and disabling new functionality

Villabajo residents roll out changes every two weeks. If their customer wants some functionality, he and the villagers of Villabajo agree on a date when he will see this functionality, and the villagers of Villabajo prepare it right on time. Moreover, in a sense, the customer cannot refuse the planned functionality: at first, this functionality is not on the production server, then there is a day of installation of the version, and then this functionality is “cast in granite”.

To roll back this functionality (either because Villabajo residents made a mistake, or because the customer himself or other systems interacting with the Villabajo product are not ready), you need to install the previous version, remove the unnecessary functionality from the new version and install this revised new version. Especially the residents of Villabajo become discouraged in such cases when the new version has managed to create data that cannot be processed by the old version: you have to clean or reformat this data.

Villarriba residents are ready to roll out changes once a day. If their customer wants some functionality, then, after it is ready for publication, he can turn it on at any time. Villarriba residents also made it so that the customer could go back to the previous version until he was fully convinced that he was satisfied with the new functionality. This approach looks most natural when the main branch of the source code must be ready for publication at any time.

Unlike residents of Villabajo, residents of Villarriba spend money on maintaining the code in an "adequate" state and, possibly, on functionality switches, but 1) they do not have to spend money on "rolling back" unnecessary changes and "rolling back" fixed versions; 2) their customers do not need to agree on the moment at which new functionality becomes an integral part of the system.

Villarriba's customers are satisfied. This is not surprising: the positive impact of this approach on the relationship between the customer and the contractor is confirmed by scientific research.

Duration of the cycle "development – publication – adjustment"

Villabajo residents roll out changes every two weeks. In most cases, they do not have the opportunity to test how the new functionality will behave on the production server. Unlike serious b2b offices, they have many deployment environments, and one of them almost coincides with the combat one in terms of its parameters.

Still, the differences between combat and near-combat environments make it difficult for Villabajo residents to accurately predict the impact of the release of the new version. You can roll out the new functionality a little at a time, but who will wait until these two-week "little" is typed into the ordered functionality? Here the customer will forget why he needs it, and the developers will remember every two weeks after installing the version what they were doing and what needs to be adjusted in the upcoming functionality.

Villarriba residents can afford to spit out new functionality in small chunks that hit the battlefield every day. After the release of the version (and they don't even really follow it, for them the release of the version is just a regular working day), the villarriba residents watch how their small changes affected the system and its users, and, depending on this, adjust the upcoming functionality, according to informing your customer of the changes if necessary.

As you can see, Villarriba residents enjoy a short development-publish-tweak cycle, so they don't have, like Villabajo residents, problems shipping literally the wrong functionality.

Functional and Regression Testing

And this is a bit of a special point, which addresses the main pain of the QA department: manual functional and regression testing. First, let's see why this is needed.

• Obviously, during functional testing, it is checked whether the work performed corresponds to the setting described and agreed with the client. In which case, corrections are made.
• At this time, an acceptance scenario for the client can also be prepared.
• Errors are detected in those places that are not covered by automatic tests.
• During regression testing, functionality is tested that is "on the side" and was not directly affected during development. Here we can also mention the verification of how the changes made in the framework of individual tasks are "friendly" with each other: whether unexpected and undesirable effects appear when these changes are combined.
• Stress testing may also be required.

Companies that release a new version every 2-3 weeks often practice lengthy manual testing, at the beginning of which all tasks planned for the upcoming version must be ready. After testing and deploying the new version in a combat environment, a new cycle of working out technical specifications, implementation, and testing begin. Thus, if developers are pouring out-of-the-box changes into the main codebase too often, the QA department will not have enough time until the set of changes is frozen enough to allow regression testing.

Functionality switches solve this problem. With their help, you can keep the changes made "in hibernation" in the test environment (you can also in the production environment, just now we are talking about the test environment) and enable them at the same time when the department is going to start regression testing. During testing, developers make changes related to other tasks, for example, for the next version. These changes are also dormant by default and are waiting for testing of the next version to begin. It is clear that functionality switches help with acceptance testing as well.

Returning to the story of two villages, let's say this: Villabajo residents combine in one test environment in advance all those and only those changes that need to be tested within a specified time period, and Villarriba residents can flexibly change the number of tested changes and the time of testing. The residents of Villarriba did not get this convenience for free (they should at least have implemented functionality switches), but it is clear that here it is hardly worth looking for benefits without cost.

Generalization

Now let's structure this information a bit. In the following list, the benefits of continuous delivery come first, then those that come with both continuous delivery and continuous deployment (new each time), then those that only continuous deployment provides. It's the same with costs. At the end of this section, the main ideas are summarized in the technology tree of the software company.

Benefits

• The installation of updates is almost completely automated, so errors no longer appear due to incorrect installation of the update.
• Installing updates is almost completely automated, so you don't spend anything on installing a new version.
• Changes are more quickly integrated and validated against the background of an overall stable system. Interested employees receive faster feedback and better understand their role and tasks in the context of the product as a whole (as a system). Therefore, there are fewer problems at the junction of areas of responsibility; “Someone else has to take care of this” symptoms disappear. Quality is born locally, not a separate process.
• The installation of updates is almost completely automated, so employees do not "burn out" during the installation of a new version.
• Installation of updates is almost completely automated, and changes are installed in small portions, so you can quickly fix errors and restore the product to normal operation.
• The installation of updates is almost completely automated, and the changes are installed in small chunks, so the mistakes you make are less often catastrophic.
• The installation of updates is almost completely automated (like the entire development cycle), so very little time passes from an idea to its implementation in industrial operation. Thanks to this, the customer and the contractor can easily test their hypotheses about new functionality in practice, quickly implement useful and remove unclaimed functionality.
• Updates are installed automatically and continuously, so you spend less on coordinating the installation of a new version and when to enable new functionality. If desired, the customer can do it himself.
• Updates are installed automatically, continuously, and in small portions, so you and the customer begin to better understand each other; problems of the form "did not do what they wanted" disappear.
• Updates are installed automatically and continuously. Interested employees receive faster feedback and a better understanding of their role and tasks in the context of a commercial product. Therefore, employees are more involved in the enterprise (both in the process of creating a product and in its subject area). They associate themselves more strongly with both the product and the company in general.

Expenses

• To set up semi-automated publishing, you must purchase and configure the appropriate software. If you already have continuous integration, then you can probably reuse its infrastructure.
• To get semiautomatic publishing to work, you must make organizational changes. For example, if a single team has access to the desired server, you need to make sure that all interested people have access to the server.
• In order to keep the main source branch "compiled", you should not include potentially destructive changes in it. For example, if the new functionality requires storing data in a new format, then your product should be able to work with data in both the old and the new format for a long time.
• If you or the customer are manually performing regression testing on a collection of changes as a whole, then you will need to configure the functionality switches. This allows you to keep many changes hidden in the production environment and enable them only after you have tested everything in the test environment.
• In order to make these changes, you will need to break the conservative corporate culture at yourself and at the customer. This obviously means that a sufficiently high level of trust must be established between you and the customer.
• In order to be able to publish product updates several times a day, you must change the processes so that 1) the installation of a new version does not require separate approval from the customer, or so that 2) the customer can manage the enabling and disabling of the new functionality.

Obstacles

There are times when implementing continuous delivery or deployment is nearly impossible.

• Some domains do not support continuous deployment well, such as telecommunications software and embedded medical equipment. And in the general case, it makes little sense to talk about continuous delivery if the developers do not manage the combat environment, that is, in the case when the product is running on the user's device.
• Problems can also arise when it comes to a large product that is not covered by automated tests or that is difficult to integrate with other systems.

Conclusion

One continuous delivery evangelist noted in 2014 that many leaders are not convinced of the benefits of this approach. We hope that today, at least the relative number of such leaders has decreased and that this material will help to convince the rest in a reasoned and constructive manner. Also, don't forget about the more general methods of working with people.

The State of DevOps report argues that in 2016, successful companies published updates much more often than others (Amazon and Netflix showed extreme values ​​- several thousand times a day), and their changes fell through the pipeline faster from inclusion in the source code to introduction. into industrial operation. But only in this case, it is better to admit right away that a direct causal relationship cannot be established here. Most likely, continuous deployment is only one of the characteristics of a successful company.