Continuous Testing in Agile Development 

As the term implies, continuous testing involves continuously testing the application throughout the entire development process. This is best achieved by integrating automated tests into the software delivery process (CI/CD pipelines) and combining the code deployment and testing processes. This practice ensures that the automated tests will be executed immediately after each new build or deployment, giving us fast feedback about the quality of the code and detecting bugs and defects early in the development process. 

There are various ways to implement continuous testing. When specifying the test coverage, “the more, the better” is usually the approach that is often taken into consideration, including all automated test cases. This has its own benefits but may also have a negative impact (as we will see further down the article). Either way, carefully selecting the test cases is undoubtedly a key point when we are considering continuous testing. 

Key aspect in the continuous testing technique is the automated test types.
If we follow the initial practice and implement continuous testing at earliest development phase, we must consider the implementation of unit and integration tests, these tests can be implemented at the earliest phase of development. 

As the application grows, we can introduce functional testing, which, depending on the application’s nature, can be API, UI, or both. In addition to this we can add up regression and smoke tests. 

Performance and Security tests should also be implemented once that is possible; these tests should be implemented at a later phase in the development process as we would need to have a stable version of the application to get valid results of the performance and security tests. 

A variety of testing types will increase the application’s testing coverage and the chance of bug detection. 

Choosing the correct tools is a very important aspect of the continuous testing process. The nature of the application needs to be analyzed properly along with the version control system and the CICD pipelines in order to choose the correct tools. Each tool involved in this workflow needs to be compatible with each other.

In which state of the CI/CD process do we trigger the automated tests? Usually this is set after deployment in the testing environment.  Once the tests are executed, further deployment is stopped (deployment to stage/production) until the tests pass. When all relevant tests pass, the next stage (deployment to stage/production) is triggered automatically. 

If any of the relevant tests failed, then further deployment (to stage/production) is blocked. However, there should be an option to trigger the deployment manually event if there is a failure.  

This can be helpful in situation where there is a “hotfix” or an important emergency updates that must be released to the production environment, and we are sure that the failure reason is false-positive or we are simply willing to accept the risk of deploying a bug to production because the hotfix or the emergency updates have greater priority. 

The development and QA team should monitor the reports generated from continuous testing. The generated report must be easily accessible and should provide data about the testing result. If any test has failed there should be a proper alerting system so the issue can be addressed immediately. Also, the report should contain information about the failure (ideally a recording of the failed test). 

The tests created must be regularly updated and maintained in order to match any new requirements or code changes. The already created tests will likely require maintenance if there is still active development. 

One of the continuous testing challenges is the execution time of the tests. The more testing coverage we have, the higher the execution time. This will become a problem when the Continuous Testing flow is integrated into the Continuous Integration and Continuous Delivery process, and it will cause a delay in the deployment process.  
To tackle this issue, parallel test execution should be introduced. This approach will significantly lower the execution time and avoid bottlenecks in the deployment process.

If we are developing a web application and have to execute tests on the browser, headless browser execution is another consideration.  
Headless browser execution also lowers the execution time significantly, as the browser operates in the background without rendering the graphical interface. 

Once a continuous testing process is implemented, it is crucial to track metrics such as total execution time, current test coverage of the application, test types currently implemented, and the failure rate.
Why do we need to track all this? To
improve the process we need to detect its weaknesses.  

• If the execution time is too high and is causing a delay in the deployment process, then we need to look into parallel test execution or locate some other factors that contribute to high execution time.  
• If the test coverage is too low or we need to include some crucial test types/cases, we need to increase our resources to create more automated tests and include those in the process.  
• If some specific test cases have very high failures, not because of bugs but because of other factors such as bad data, then we need to improve the test and lower the failure rate to avoid false positives. 
• If some specific test cases have a high failure rate because of bugs, then we need to inform the development team to pay more attention to that part of the code as we may have repeating issues.

Both standard automated testing and continuous testing have automated scripts at their core. In both workflows, we rely on their execution and reporting. We must prepare a specific testing environment and set of data to execute the tests. 

In such a general overview both terms look very similar, so what is the difference then?
The key difference and the main advantage of continuous testing is the way of setting up the automation workflow. 

In regular automation the scripts are executed manually when that is required (as part of functional or regression testing) and the results must be shared with other team members, so we still have a human factor in the entire process.  

On the other hand, the continuous testing workflow aims to remove the human factor as much as possible. The automation tests are triggered automatically throughout the entire software delivery process. The tests are integrated into the CI/CD pipelines and are part of the deployment process. On each new build a whole set of different automated tests will be triggered, upon execution if any test has failed there should be an alerting system to notify the developers about the failure and to stop the deployment process preventing faulty code to move further in the delivery process.