The road to efficient testing in the era of DevOps

On October 22, the 2021 DevOps International Summit was successfully held in Beijing. Top technical experts from home and abroad discussed the DevOps system and methods, processes and practices, tools and technologies. Cheng Shengcong (Diego), Product Director of Coding Testing and R&D Process Management, accepted an interview with CCTV's "Attitude" column group at the conference site, and also shared his theme for everyone in the DevOps Best Practices and Solutions Special- DevOps Era The road to testing.

Testing is becoming the biggest bottleneck of continuous delivery

The industry’s anxiety about testing has been around for a long time. Beginning in 2017, the industry began to form a consensus on the testing dilemma, that is, testing not fast enough has become the main reason for delays in delivery. After that, the industry began to emphasize automated testing, and proposed concepts such as creating a fast feedback loop, assembly line workflow, and engineering culture.

In this context, Diego believes that continuous testing is the key to efficient testing in the DevOps era, . As a basic and continuous activity, testing should run through the entire software delivery cycle. Continuous testing changes the working mode of the traditional post-test, allowing testing activities to extend to each stage of SDLC.

Efficient testing requires automation as the cornerstone of practice

From a survey on test time spent, it can be known that test execution accounts for 40% of the total time. Therefore, if you want to improve the overall test efficiency, the first thing is to improve the efficiency of test execution through automation. However, the testing team found that even if the automated code base is configured in CI, the set of use cases to be executed is fixed. As the coverage of automated tests increases, more and more automated codes, the running time becomes longer and longer, so the running frequency But it is getting lower and lower, and the effect is not satisfactory.

Diego pointed out that it is not enough to write automated code silently. requires "smarter execution" . If you can manually associate requirements (Req), use cases (Case), and automation functions (Func) together, flexible determination of test subsets will make testing more free. If an automated function fails, a problematic requirement can be quickly located, and the team can then determine whether it can continue to publish based on the priority and impact of the problem.

New era, new process

In the new era, advocate new processes. Continuous testing is not only automated testing, but also global optimization of the process , which requires us to extend from the single point (execution) efficiency improvement to the global efficiency improvement. Tools/techniques improve the efficiency of the process, and the process will point to the right results (effectiveness).

So, what kind of testing process is needed in the DevOps era?

First of all, Diego believes that the four "above"s of the Agile Manifesto clearly point out that R&D should focus on behaviors and results that bring value to users, rather than products in the process.

However, although collaboration is important, it does not mean that processes, documents, and plans are not important. As long as the complexity of the software remains, the processes, documents, and plans that help the smooth delivery of the software are all necessary. It just needs to change accordingly and keep pace with the times-shift to light processes, concise documentation, early planning, or "evolutionary" processes/documents/plans.

DevOps needs to be lighter and more emphasis on the process of role collaboration . The testing team first needs to carry out testing around requirements, strategically define subsets, and determine risk-based testing strategies. Second, the team needs to push the test to the left, so that the test and development work in parallel. At this stage, the work that needs to be completed includes unit testing, test review, code scanning, as well as development and automated testing based on interface definitions. Finally, it is necessary to move the test to the right in practice, so that "testing will never exit"-after deployment, online real-time monitoring is also required to ensure that problems are quickly discovered and dealt with.

In addition, DevOps also requires robust specifications and fast feedback . Diego believes that R&D "consistency" is the basis for efficiency improvement, and norms can reduce the subjective complexity caused by humans in collaboration and avoid unnecessary contextual understanding. And quick feedback is not only to reduce the "labor cost" of manual operation, but more importantly, to reduce the "heart loss" of members, so that members can focus on their own work, without having to switch back and forth between different platforms. The artistic component of software development should be kept in a cage, and the artistic part is not needed in software development, such as mechanical judgment and selection, and collaborative operations, which should tend to be scientific and quantified.

Sharing of Tencent Cloud's internal landing practices

Finally, Diego shared with the audience the internal landing practices of the Tencent Cloud CODING team, showing how efficient testing of R&D integration can be achieved.

In the research and development process, there are various activities. Different roles in the R&D team, such as product, development, testing, and operation and maintenance, generally only focus on the activities they are responsible for, and then there are correlations between the activities of different roles, and the activities generated by collaboration form a complete workflow. On this basis, the addition of entry and exit regulations can make collaboration more robust, and automatic operations can make flow more efficient. CODING is based on this logic to design the product CODING Compass for the efficient management of the R&D process.

However, DevOps shouldn't just be an area where the R&D team is happy, we also need to be connected with business value. By mapping R&D activities to stages familiar to business personnel: planning, development, testing, deployment, and maintaining the Lead Time, Process Time,% of Completeness & Accuracy, etc. Attention and continuous improvement of indicators form value stream management.

Next, Diego introduced to you how the Tencent Cloud CODING team embed the testing activities in it.

First of all, during the iterative planning phase, tests will be carried out around requirements, and a subset of tests will be strategically defined. Testing and development work closely together, and at the same time claim the same required tasks. When the iterative Backlog is determined, it means that the test scope is determined, the test plan can be created, and the requirements acceptance criteria can be directly converted into function point use cases.

In the iterative process, testing and development work in parallel. While the developers are coding the single test, the use cases are further refined to make them complete and reviewed in time. When writing automation use cases, we can use the help of the CODING platform to realize the correlation between automation use cases and functional use cases.

Thus, when the test execution is initiated from the test plan, the associated automation use case will be designated for execution.

Through the practice of high-efficiency testing of R&D integration, the team’s high-frequency release has been firmly supported. The monthly average number of successful releases is 400+, the required test coverage is 96%, and the automated use case set covering the backbone is basically through continuous refactoring. Maintain the number of 1400 (the smoke automation use case must be executed within 15 minutes).

Finally, Diego concluded: The reason why CODING can achieve efficient testing, the transformation of the three aspects of the concept is particularly critical.

The first is to break the functional silos and achieve goal sharing. In the past, the goals of developers and testers were usually separated or even opposed to each other. Developers looked at how many lines of code were rewritten, and testers focused on how many bugs were found. The correct concept should be to give the team the same goal, let different roles work around business needs, and jointly deliver high-quality software.

The second is to change the "stage transfer" to "small steps and fast walking" working mode. Development and testing should deal with requirements in the same priority order, so that when they are delivered, they get finished products instead of "semi-finished products." On the one hand, the overall delivery time is shortened by controlling the number of work-in-progress, and on the other hand, the single point of acceleration is achieved through automated means in the local area.

Finally, it is to realize the real empowerment for team members. Empowerment does not require everyone to be all-rounders, but to allow team members to collaborate with others or use the tools and services provided by others. In order to eliminate doubts and ensure the smoothness of the collaboration process, the consistency of activities is improved through the setting of standards and specifications, which has a positive impact on the improvement of team effectiveness.