Testing is one of the processes that can be automated within CI/CD. Reliable and thorough automated testing allows you to be confident in new builds, reduces production costs and improves product quality. But even in large companies on large projects, it is not always available.

Usually they start investing in testing when there are too many bugs, customer dissatisfaction grows, and business money starts to burn. Together with Alexander Dovnar, Lead DevOps at Naviteq, we analyzed what continuous testing is and what role it plays in CI/CD. We also considered the case, how companies come to its implementation, and what this implementation gives.

Continuous Integration (CI) and Testing

Almost all approaches to software development are iterative. This means that teams move in 2-3 week sprints, with each sprint adding new features to the product. Work on different tasks is carried out in parallel, and developers’ codebases can overlap. So that this does not lead to bugs and breakdowns, a CI process is being introduced.

The main task of CI is to make sure that as you work on the same or different codebases in parallel, you eventually get a version of the application that can be delivered to the environment. To do this, each piece of code must be versioned , tested and packaged in an appropriate format.

If a developer works on a local branch and does not affect the rest of the team in any way, basic checks that his code is adequate are sufficient. If, on the other hand, a developer creates a pull request with the master branch and could potentially affect their colleagues or end customers, a more complex testing model is needed. In this case, it is important to check how correctly the modified component works with other components. To do this, they run unit tests, integration tests, and security tests to make sure that there are no basic violations.

Continuous Delivery (CD) and Testing

There are two options for decoding the abbreviation CD – continuous delivery and continuous deployment.

Continuous delivery or continuous delivery is a process after which we can click on a button and deliver a new version of the application to our customers.

In continuous deployment or continuous deployment of a button and a manual step, we do not have. Everything goes automatically. But since it is difficult to roll out changes to clients in the background without our participation, continuous deployment is only possible with a testing strategy and a monitoring system. Then if suddenly something goes wrong, we will know about it before the customers.

What the CD consists of: after CI, we have a working version of the application, we deploy it in different environments and run a test suite. Usually there are three such environments (but it all depends on your scheme of working with the GIT version control system and the release process):

• In the dev environment (development environment), we roll out all new versions in which development has already been completed – these are not releases yet, but simply new functionality. This is where we do at least integration testing and smoke testing to make sure our application doesn’t break.
• Next, the branch where the changes were made merges with the main branch. And we deploy the version of the application resulting from this merger in the staging environment . At this stage, we run E2E testing (end-to-end tests to check the full functionality), plus, in addition, a manual tester usually works, who will test our application as a real user. A variety of security tests can also be carried out.
• Upon completion of the tests, we say that the version of the application is stable, free of bugs, and ready to be deployed in the production environment . Then it all depends on which release process is set in the team. We either expand everything manually by a button, or it happens automatically upon the occurrence of some event. At the end, we run a series of tests and ask our testers and/or real users to check the functionality (for example, if they are ready to be Beta testers and we have Feature toggling in the application). Next, we monitor the number of errors in the monitoring systems and logs of our applications in order to catch potential problems.

There may be more environments, for example, if we are going to conduct performance and extended security tests. Running them on one of the three listed environments means losing it for a while, so separate environments are allocated for them.

Continuous Testing (CT)

Continuous testing is what works within CI/CD in general. Before organizing continuous testing as a process, we need to form a testing strategy: select groups of tests that we will run at different times in order to check how well the application works.

In the CI phase, it is important that the build is fast, so lightweight test types are most often used:

• unit tests – to test individual components;
• integration tests – to check the integration of these components;
• basic linting – to check the code for compliance with the way we agreed to write it inside the team;
• statistical code analysis – to catch potential vulnerabilities in our code;
• smoke tests – to check if the application has deployed and if its basic components are working correctly.

You can do more tests, but usually these 5, provided they are well written, are enough. By running them, we will already prevent about 90% of bugs at the CI stage.

At the CD stage, there is continuous testing of specific environments, so there are more tests, and they are longer in time:

• E2E tests – to check the correctness of the UI;
• performance tests – to check if there are performance drops;
• regression tests – to make sure we don’t revert to bugs fixed in the past before releasing to production environments;
• security tests that “break” our application – to check that we do not have vulnerabilities;
• penetration tests – to check the resistance of the application to hacking and repelling DDoS attacks.

The depth and set of tests varies from environment to environment. For example, we run E2E testing before going into production, when we want to make sure that everything works correctly. If we have not only a web application, but also versions for iOS and Android, such testing can take up to two days in automatic mode.

Developing a testing strategy is the work of not only experienced testers, but also DevOps engineers. Tests are part of CI/CD, so DevOps implementers should also be involved in their creation. We still cannot do without business representatives, who should give money for all this. Developing continuous testing is long and expensive, but it delivers incredible results in the long run.

How to know when it’s time to test

Imagine a startup in a vacuum, which is created by two programmers. The first writes the front end, the second writes the back end, and from everything related to DevOps, they only have Git. They use their only production to show potential investors the product, and they test any changes locally.

Gradually, they find investments, and their MVP begins to acquire advanced functionality. It becomes difficult to develop with two people, they hire 4 front-end developers and 6 back-end developers. A team of 10 developers can work in parallel on different functionality, so at this stage of development it is logical to introduce containerization and CI. Then the local stands on which the functionality is tested will reflect the real situation in the final environments.

The startup develops further, it manages to get the first customers. But customers are willing to pay for a subscription only if, conditionally, they receive new functionality every month. The developers say that once a month they will release new functionality, but promises to customers do not always coincide with the possibilities – deadlines are burning, errors appear, etc.

There are two development paths:

• either work very hard and constantly fix spontaneously appearing bugs so as not to lose customers;
• or allocate one developer each from the front-end and back-end teams so that they begin to cover the existing code base with tests. And also agree internally that now all new functionality will be covered by at least unit tests.

The second option allows you to protect yourself from self-mistakes that often appear during development, and contributes to development – the team understands the benefits of unit tests and gradually introduces integration testing, improves the stability and quality of releases.

The startup continues to scale, the staff grows, new teams appear. In addition to the web version, the product is getting versions for iOS and Android, and more tests are needed. Developers do not like to test code, and then engineers who specialize in such tasks come to their aid – testers. They start catching bugs manually and implementing automated tests.

Of course, this is an ideal picture in a vacuum, which is not always possible to achieve in life. But when the project is actively scaling, without such an approach it is difficult to keep the high quality of the product. And continuous testing with the right strategy just allows you to reach the level when you are no longer afraid to roll out new releases.

Briefly about the main thing: why you need continuous testing

Teams using continuous testing can achieve the following results:

• Fast and stable releases . With continuous feedback loops, you find bugs early and fix them quickly. Time to develop and release a release is reduced.
• High efficiency . Due to automation, team resources are spent more rationally. Developers can focus on development, and testers can move away from repetitive tasks in favor of tasks that require critical and creative thinking. As a result, the coverage and quality of tests can be increased.
• Risk Reduction . Automation eliminates the human factor. Robots perform repetitive tasks with lots of data better and faster than humans. Risks in the field of automatic testing are reduced.
• Cost reduction . Test automation allows you to detect bugs and errors earlier. And the sooner they are discovered, the easier and cheaper it is to fix them.