Continuous Testing | The Key to Efficient Testing in the DevOps Era

Author of this article: Cheng Shengcong-CODING Product Manager

Challenges brought by the times to testing

Testing has always been an important guarantee for the quality of the software R&D process. In the traditional R&D model, testing activities are always at a relatively lagging link in the software life cycle. I believe we will not be unfamiliar with the following dialogue scenarios:

1. planning the needs:
   "The X function of this version is very important, and the test must not be missed!"
   "Okay, we have made a note in the requirement list."
2. development:
   "Is the function finished this time?"
   "The development is complete, but the test has not yet been completed."
3. approaching the release date:
   "Is the feature online?"
   "It's the pre-release environment, the regression test is still in progress, and later comparative analysis will take time."

The questions mentioned above: "What has been tested", "Did the test finished", and "Is the test fast?" are the "three soul tortures" that teams often face. With the promotion of the Agile & DevOps model in the software industry, more frequent delivery has aggravated the industry’s concerns about testing. Insufficient testing efficiency often becomes the primary reason for delivery delays, and the testing process has become the enterprise's DevOps The biggest bottleneck of transformation.

In order to cope with such challenges, the concept of "continuous testing" (or "agile testing") has been proposed and has gradually become an inevitable pursuit of the industry. Continuous testing can meet the following core demands:

1. What was tested: Under the premise that 100% regression test coverage cannot be achieved, test subsets are divided based on business value.
2. The test is over: In the process of continuous delivery, test on demand and provide quick feedback.
3. Is the test fast: let the test execute fast enough.

What is continuous testing?

Definition from Wikipedia: The process of performing automated testing in the software delivery pipeline in order to obtain immediate feedback on the business risks of pre-release software. The Mobius ring of continuous delivery is shown in the following figure:

It is true that the above definition fully emphasizes the importance of automated testing, which is the basis of continuous testing. But returning to the ultimate goal of "increasing efficiency through continuous testing", merely improving the single-point efficiency of test execution is not enough to reflect the nature of the change in testing concepts brought about by continuous testing. From the perspective of overall test efficiency, the "in-sprint testing" or "agile testing" described in the "agile" model of Project foundation and continuous activities throughout the entire software delivery cycle . Images from the Jenkins community better reflect this concept:

How to achieve continuous testing?

Continuous testing changes the traditional post-testing work mode, allowing testing activities to extend to each stage of the software development life cycle.

1. In the demand planning stage, plan the test as early as possible, and strategically define the test subset.
First of all, from the stage of requirements analysis, start to plan tests in advance and write test cases to achieve appropriate requirements coverage. This requires helpful practices including ATDD and BDD, especially for teams whose TDD is difficult to implement can try ATDD. Secondly, must have the awareness of optimizing test coverage . Testing should not blindly pursue 100% coverage, but test based on business risk and value testing strategies (Risk-based Testing). “100% coverage of high priority requirements” is far more effective than “80% coverage of all requirements” value.

2. During the iteration, push the test to shift left (Shift-left) to realize the parallel work of test and development.
Test execution should be advanced to the early stage of the software development life cycle. A variety of engineering practices can help the team move to the left: for example, attaches importance to test review, basic guarantee through unit testing, development and automated testing based on interface definitions, and introduction of code scanning Determine whether to meet the coding standards and engineering standards . In this way, in the iterative cycle, the integration test cases can be continuously written around the requirements, and the progress of the development is maintained in coordination, and the necessary test support is provided for the development, so that the testing and development work can be carried out simultaneously.

3. The iteration is in progress, providing a complete test environment and correct test data in a convenient way.
The creation of a test environment close to production and the rapid preparation of desensitized data have always been two important challenges faced by the team. Now with the maturity of cloud native technology, especially the development of Docker technology, it is possible to build and destroy the environment on demand . However, the management of test data is still a difficult problem. Basic data such as account information and environmental information, which are easily standardized, have already had better solutions in the industry. This is already a major advancement. However, business data has always lacked sufficient business abstraction due to the variability of scenarios, and is still in the basic stage of relying on the framework for process specification. Development based on interface definitions to implement mock services can also improve process efficiency.

4. After the application is deployed, focus on the test shift-right.
The traditional waterfall model regards deployment as the next stage of testing, which means that the end of the test is after the application is released and the function is quickly verified. The continuous test does not consider that the release is completed and the test is exited. It emphasizes that after the version is launched, continue to pay attention to the data monitoring and early warning of the production environment, find problems in time and follow up to solve them, and reduce the scope of impact to the lowest . And the use of production data can bring tangible value to the development process: for example, copy production data for desensitization to prepare test data, and the results of analysis of service access data can also provide optimized guidance for testing in the development process. Adjust testing and form better smoke and regression testing strategies, etc. The practice of shifting right includes data analysis, gray/canary release, online real-time monitoring, user feedback tracking process and so on.

In addition, we should pay attention to the precipitation of data in the process of practicing continuous testing, and then continuously optimize our behavior based on data indicators, so as to realize the team culture of continuous improvement advocated by DevOps. With the continuous spread of agile and DevOps culture in the software industry, the R&D team will inevitably expect to achieve a shorter iteration cycle, more effectively improve the quality of software development, and deliver business value faster.

CODING adheres to the concept of providing one-stop DevOps solutions for enterprise R&D teams. Through powerful test management functions, it R&D teams to use testing as a basic activity throughout the entire process of software delivery 160b9e493c4c84, which greatly shortens the software delivery cycle and allows testing Iterate synchronously with R&D, achieve continuous testing, and help the team return to high-quality delivery.