Automated test system construction in the mist

Author of this article: Cheng Shengcong-Product Director of Coding Test Domain

In the industry's raging DevOps transformation process, automated testing has always been one of the hot spots. After all, providing fast quality feedback is the key to achieving DevOps goals. As a result, as the "crown" of the testing field, the implementation of automated testing has always attracted people's attention. However, various problems and even controversies have arisen during the implementation, which has been enduring for a long time, and has hidden layers of fog and confusion on the construction of the automated test system, which makes people wonder. Below we will discuss some of the "pits" we have stepped on, hoping that these experience sharing will help to uncover the fog and see the direction.

Do you want to do automated testing?

In software engineering theory, testing is an "assistant" role relative to development, and the product delivered by software does not have to include the output of testing. With the increase in the complexity of the software development process, although it is still an auxiliary role, and its value is still difficult to measure individually, the role of testing in quality assurance has been deeply rooted in the hearts of the people, so that since more than 10 years ago, no one will challenge testing as research and development. The basic role in the system exists.

However, similarly, as an "auxiliary aid", automated testing in the testing field has begun to face the same question: Is there any value in automated testing? This question began to be frequently asked more than 10 years ago, and it is difficult to answer. As an input in an auxiliary position, people will definitely pay close attention to its cost and its comparison with benefits. Automated testing first appeared in order to replace repetitive manual operations, thereby saving the labor cost of regression testing, so the prerequisite for obtaining positive benefits is enough times of execution:
Automation benefit = number of iterations x (manual execution cost-use case maintenance cost)-use case writing cost

Therefore, under the frequently released test scenarios in the DevOps era, the value of automated testing has been fully demonstrated. The question of whether or not to do automated testing is now no longer a problem, because the industry has now formed a consensus: automated testing is the foundation of continuous testing and an indispensable practice in the DevOps era. In addition, due to the high execution efficiency of automated testing, the time cost advantage reflected is even more obvious, and even more than the cost advantage can poke the pain points of this rapid release era: because otherwise, we will be more and more difficult to deal with Quick and effective verification required for short-term release.

Carry out automated testing strategically

The construction of a test system requires hierarchical strategy guidelines, and interface testing is often the highest priority

Rome was not built in a day. In order to achieve the goal of automation, the construction of an automated test system requires investment of resources and manpower. Therefore, in the actual landing process, we need to fully consider the ROI to design a path to achieve the goal that is in line with the actual situation. Automated testing is indeed of great value, but it does not mean that we should invest in various types of automated testing uncontrollably: automated testing is to verify whether the established logic meets expectations, and the maintenance cost of automated code in scenarios with frequent changes in requirements Not to be underestimated. So we need a suitable strategy to guide the implementation of automated testing-the pyramid model.

Many people's first impression of the pyramid model is that it gives advice on the proportion of investment in three types of tests: unit testing is the most, interface testing is centered, and UI testing is the least, such as 70%, 20%, and 10%. But more importantly, Mike Cohn proposed the concept of layering testing, and gave the advantages and disadvantages of each level of testing: the closer to the high-level testing of the user interface, the coarser the granularity and the lower the efficiency. There should be fewer tests; on the contrary, the closer to the testing of the underlying code, the finer the granularity, the higher the efficiency, more writing should be done, and more frequent execution.

In practice, each company faces different scenarios and different investment situations. For example, the actual situation may not be a pyramid but a spindle shape, and the middle interface test accounts for the highest proportion. Various practices have shown: In the initial stage of automated testing construction, interface testing is often the team's first choice for automated testing. This is because interface testing has the advantages of both execution efficiency and business value. It is easier for the technical team and the business team to invest in resources in this field. And because the stability of the interface definition is also high, its maintenance cost is also controllable. So relative to unit testing and UI testing, the input-output ratio of interface testing can be said to be the highest.

Interface testing is based on interface definition management. Synchronous reminders and mocks of contract changes are the key

The interface test achieves the goal of test verification by calling the interface, including the interface between the system and the system, and the interface between the various sub-modules within the same system. The focus of interface testing is to check the logical dependencies between systems/modules, and the accuracy of interactive data transmission. Interface testing is a type of black box testing, but it is the black box testing closest to white box testing. Therefore, it is also close to unit testing in terms of early detection of defects and execution efficiency, and is often called "gray box testing."

Use cases for interface testing generally include single-interface use cases and multi-interface use cases that integrate different interfaces based on business scenarios. The single-interface use case is the foundation and is also required for the development and debugging process. It is more popular in the industry to use Swagger for interface document management. Swagger predefines code annotations related to mainstream programming languages. You can get the update of the interface document after the interface realizes the code automatically reflects the interface change to maintain the automation use case. That is very important.

Multi-interface use cases are designed by testers based on their understanding of required functions. These use cases fully demonstrate the business value of automated testing. Because this part of the use cases is relatively complex, the team will need to prepare a basic framework for it, and even build scaffolding to improve writing efficiency. In the implementation, it will generally include the ability of interface specification definition, code management of calls between interfaces, storage management of test data, execution scheduling platform, and structured statistical reports. So many efficiency tools in this field have appeared in the industry, such as Postman, ReadyAPI! , Robot Framework, and "low-code" platforms apifox, Eolinker, etc.

With the interface contract definition, you can implement mocks (test doubles or baffles) for interfaces that are not online, so that you can build scenario test cases without relying on specific development implementations, which is beneficial to test development or different developers Parallel collaboration between. There are mature frameworks for building Mock solutions, such as Mockito, EasyMock, etc., or platform-based tools such as postman and apifox can easily build Mock Server.

In general, with the interface definition specifications that can be followed, plus the information synchronization of interface changes, and the provision of mock services to the interface, the team can implement parallel development, debugging and testing based on the API-first approach.

Is there a "shortcut" to efficiently write automation use cases?

Nowadays, the functions of the system are becoming more and more powerful, and more and more complex, writing a good automated test case is not a simple matter, this piece is also an investment that cannot be ignored. So in the practice of automation construction, we will naturally pursue more efficient methods. There is no problem in pursuing high efficiency, but how can we avoid going wrong? Below we will discuss some common "efficiency-improving tools", hoping that they can be used for reference in actual practice.

Coding to the left, low-code platforms to the right?

The interfaces of today's systems are often complicated. To do a good job of interface testing requires a deep understanding of the logical relationships between systems/modules at the business level, as well as a good grasp of various testing frameworks and related coding techniques at the technical level. It can be said that the construction of interface test automation is still facing great challenges, and people will naturally seek efficient ways to implement it. Accordingly, there is a demand for a low-code platform for automation use cases. Therefore, out of "realistic considerations for testing the technical capabilities of students", the R&D Leader often "takes a responsible attitude" and seeks to "get into the pit" to pursue a low-code platform. The concept of low code has been so popular in the past two years, which has caused this topic to be repeatedly mentioned. So when we write automated test cases, should we follow the way of writing code honestly, or should we boldly adopt a low-code platform to quickly increase coverage?

What is the advantage of a low-code platform-that is, by lowering the technical threshold of automated writing, so that people with weak coding skills can also participate, so as to quickly increase the coverage of automation from 0. Generally, low-code platforms are designed based on the principle of separation of data and code for interface test automation: first abstract the commonly used interface operation methods and encapsulate them (called keywords in Robot Framework), and then use the interface The drag-and-drop components are combined into a logical process, coupled with the data transmission drive to form a complete business scenario. Therefore, the experience of using a low-code platform is to realize automated use cases through the operation of forms, which feels "no programming ability required".

However, when using low-code platforms for complex business testing in practice, the requirements for data dictionary, abstraction of operations (keywords), and design capabilities are still very high. Otherwise, changes to the underlying logic will require changes. Many use cases. Judging from the practice of many teams, low-code platforms will be unsustainable in the mid-to-long-term maintenance in the face of changeable business scenarios: business (or business testing) members who lack technical abstraction and non-engineering backgrounds are difficult to sustain" Refactoring" existing use cases. In order to ensure the accuracy of use cases and overall operating efficiency, the reconstruction of automated use cases is inevitable. Faced with a series of "tabular" documents, engineers generally believe that this method is fragile and inefficient, so they are unwilling to take over.

Therefore, uses a low-code platform depends on our expectations for business development: For non-core businesses, if the modules are expected to be relatively solidified and do not need to evolve, then you might as well use the low-code platform to carry out automated tests with rapid coverage. The code platform does alleviate the "automation construction anxiety" of team managers and helps the team take the first step in automated testing; for the core business, modules that are destined to follow the development of the business and undergo technological evolution, then they need to be fully considered To achieve the purpose of improving the efficiency of automated writing for a long time. In principle, we should admit that automated test cases actually has an "engineering threshold", and it does require an "engineering threshold". Even if you use a low-code platform, you need to have "engineer thinking" and consider the encapsulation and maintenance of "keywords" and data structures. In general, engineers should be able to write automation use cases more efficiently by increasing visualization and reducing repetitive work; rather than "unrestricted lowering the threshold", in order to hope that they can write robust and high-quality without training Of automated test cases.

Can the flow recording and playback mode replace manual writing automation use cases?

Regarding interface testing, there is still a situation in reality: that is to "patch" the missing interface testing of the existing system, and it is necessary to supplement the coverage of many existing interface scenarios with test cases. If the traditional testing method is adopted, it is to start from the interface definition, manually sort out the system interface documents, enter the test cases, write logical assertions, and then construct the corresponding test data after in-depth understanding of the interface. This is a conventional method. , Is also the correct way, but it is undoubtedly a slow and meticulous job. When we need to complete a round of supplementation in a relatively short period of time, this method obviously cannot be done. So a new solution appeared: Automatically collect real traffic and form interface test cases.

Traffic "recording" refers to intercepting and recording online traffic requests and returns, and then recording them to form test cases; while "playback" refers to copying the online recorded requests and returns to a quasi-production environment service , Testing whether the new functions and services meet the requirements. Reality (data) and high efficiency are the two major advantages of traffic recording and playback methods, and its main application scenarios include:

1. When an online failure occurs, the recorded real traffic can be played back to the development/test environment for debugging and analysis. This is the use of "real" scenarios and the basic value of the traffic recording and playback function;
2. Copy and enlarge the recorded real traffic and apply it to the pre-release environment as a pressure test use case. This is also a "real" scenario. Real traffic is indeed a good supplement to pressure testing;
3. As mentioned above, batches form the first set of regression automation use cases, which is a scenario where "efficient" is used.

The mainstream ways to achieve traffic diversion recording include: Nginx mirror copy, GoReplay directly monitors the network interface to capture Http traffic, log-based analysis, and self-developed copy diversion functions for application services. The most popular tools in the industry are GoReplay and TCPCopy. GoReplay is particularly easy to use and has no code intrusion. The impact on online applications is negligible.

So with the function of traffic recording and playback, it is no longer necessary to manually write automation use cases? Definitely not, we also need to write automation use cases steadily. First of all, traffic recording and playback is a post-installed "potting behavior". No one wants to wait until the interface is online before testing. is often a one-time job; secondly, traffic recording also takes a long time to reach a higher level. For use case coverage, we still need to manually design important scenes for very useful boundaries; again, for scene use cases that build complex multi-interface combinations, the method of traffic recording is still difficult: perform recording on the recorded traffic. For secondary processing, it may be better to use your brain to realize it manually.

Is the recording method of UI test cases reliable?

The method of UI testing is very intuitive and it is easy to see the business value. However, the rapid iterative characteristics of the UI interface result in a short effective life cycle of test cases and extremely high maintenance costs. Therefore, compared to unit testing and interface testing, UI automation testing, which is fragile, complex, and low in input-output ratio, should not be our main input area. It is generally used to cover critical and UI-in-steady businesses.

UI automation use cases can also be quickly generated by recording. Nowadays, the more popular automated test tools that provide recording methods include: QTP for desktop programs, Ranorex, Selenium IDE for Web pages, Chrome DevTools-Recorder, Ali's UI Recorder, Xinghai "Whale Hung", WeTest UITrace, etc. for mobile terminal programs.

Now there is an interesting phenomenon in the industry, that is, although the recorded UI automation use cases are basically not maintainable, many people still want to use recording to achieve automation. The bottom line is that everyone’s expectations for UI testing are different from those of general automated testing: admit that the variability of the UI interface leads to the fact that the maintenance cost of automation use cases is very high, so that they can be regarded as “short-lived” and “disposable”. Regression test case- As long as the recording is simple and fast enough, it will not be re-recorded after a period of time (the next version).

From the point of view of "make the best use of everything," it seems understandable to make such a choice, but we should also pay attention to the acceptance of this approach by team members. After all, repeatedly doing things that are destined to give up and start all over again, the morale blow to people is also obvious. In the long run, we still hope to cover UI testing from a relatively stable level, such as the component level, that is to say, move forward to the unit testing link. After all, the front-end framework is now very mature, and it is possible to perform fine-grained unit tests on the UI based on the framework. For end-to-end UI testing, you should still invest carefully.

Use cases for "smart" execution automation

refer to integration level tests, excluding unit tests

Writing good use case code is an important first step in automated testing, but only when it is executed can its value be demonstrated. As stated in the automated revenue formula, the more automated test cases are executed, the better. In practice, when we cannot execute the full set of regression use cases every time, we need to consider the balance between test scope and efficiency, and implement automated use cases strategically. The following two points are about the core principles of automation value, which need to be kept in mind in the construction of the automated test system:

Principle 1: The root of the value of automated testing lies in the business. It should be based on business-driven automated testing, and always pay attention to the tests corresponding to the highest-priority requirements.

Principle 2: The amplifier of automated test value is the test execution frequency. Only when the number of runs is large, can the cost be earned; and only a single round of execution is fast enough to ensure a higher execution frequency.

What "pits" will you encounter when running automated tests in CI/CD?

Starting from the waterfall model era, the traditional automated test execution method is nightly-run: set the task to run all return cases after get off work every day, and then deal with the case of execution failure after the results are obtained the next day. The reason why it is impossible to run automated tests in real time is because after a certain number of automated use cases have accumulated, it takes several hours to complete a round. However, in the DevOps era, there will not be such "ample" time for regression testing. It is expected that automated testing will be embedded in the CI/CD pipeline to provide timely quality feedback, such as the following situations:

1. Automated testing is embedded in CI, that is, every time the code of a new feature is merged (Merge Request) to the main branch, in addition to running a full unit test, it often runs a smoke test case set;
2. Automated testing is embedded in the CD. Before each deployment action in the production environment, confirm to run a full regression test case set in the pre-release environment; after the deployment is completed, a smoke test case set will be run in the production link;
3. In the case that the team can write automated use cases in a timely manner within the iteration, the CI process also needs to execute the use cases corresponding to the completed features to ensure compatibility between code changes;
4. Even within an iteration, members of the team are free to create CIs to execute any selected set of automation use cases.

Generally speaking, CI/CD is very sensitive to running time, so the integration test part of the pipeline does not consume energy for too long. If the scope of the test case set is expanded uncontrollably and integrated, the running time will inevitably be out of control.

Imagine that as more and more codes are written, it takes longer and longer to run automated use cases. In the end, it can only reduce the frequency of executing automated tests... This is relatively speechless, it is clearly written. Too much code, but can not play the value, then we still need to work hard to write more automation use cases? I believe many people have encountered such a "pit". Automated testing was originally very powerful, but now it is locked in a cage. How to solve this dilemma? Since it is not advisable to execute a large number of test cases step by step each time, you need to find a better way to achieve "smarter" execution of automated use cases.

To execute faster, the first solution that comes to mind is naturally to execute in parallel and distribute the use cases to a distributed machine environment. To do this, you need to set rules to ensure the independence between use cases, and manage the test data so that the data after each use case execution change can be restored. Just doing so will greatly increase the complexity, and it will be difficult to troubleshoot problems after they occur. In addition to this, there is another very important question: each run is full, mixed with a large number of invalid execution use cases, so what can the general pass rate thus obtained explain? Unless a simple standard is set, it is to pass all of them, but in reality, delays caused by network jitter, complicated data updates, etc., will lead to sporadic failures. Therefore, parallel execution cannot completely solve the problem. We also need to find another way: If you run automated test cases, you can determine the scope of the use case according to the changes, instead of covering the entire number of use cases each time, can you control the time of test execution? ? The answer is precision testing- establishes the correspondence between changes and tests, so as to verify changes more effectively and reduce unnecessary global regressions. precision testing roughly includes two types in terms of implementation: determining the use case set based on code changes, and determining the use case set based on demand changes.

To automatically determine the use case set based on code changes is to establish a logical mapping relationship between test cases and program code, reflecting the test coverage of the code. This goal is very ambitious, but the current industry solutions are not yet mature, and a large number of manual checks are required. To determine the set of use cases based on changes in requirements is to establish a mapping relationship between test cases and business requirements, reflecting the test coverage of requirements. This method does not sound so high technically, it is only achieved through management constraints, but it has been able to achieve our goals very well. CODING is based on this logic to create an automation use case library. The goal is to allow the practices of 1) and 2) to be smoothly implemented, and even to realize the free execution of a set of automation use cases in 3) and 4).

Coding helps realize "business-driven automated testing"

Coding test products promote "business-driven testing": everything starts from business needs, and by establishing the relationship between automation and business needs, automation can be targeted to expand coverage, instead of writing test codes out of thin air, resulting in repeated invalid tests .

At the same time, the coding automation use case library analyzes the automation test code, matches the obtained function list to the corresponding function use case, and forms the mapping relationship between the automation use case and the demand function based on the relationship between the existing function use case and the demand:

Based on this relationship, we can follow the vines, find out which demand has the problem according to the failure of an automated function, and then decide whether to continue with the next step according to the priority of the demand or the scope of influence, whether to return to fix the problem or decide to continue publishing online.

First of all, establish the matching relationship between automation code and functional use cases, and the automation coverage is "clear at a glance."

Coding provides sample code import to help users fill in the function use case ID in the comments/comments according to the rules in the code, or manually judge, so that the parsed automation code function matches the existing function use cases, and gradually improves the automation of the function use cases Coverage rate (also lays the foundation for accurate testing):

Then, during the iteration, a specific set of automation use cases can be selectively executed.

In the ordinary test plan, you can circle some functional use cases to execute the corresponding automation use cases; in the iterative test plan, after the specific requirements are circled, the corresponding functional use case list is also generated, thereby realizing the execution of the business-driven automation use cases . Coding will continue to strengthen the ability to execute automated tests flexibly, and provide a smoother "smart" execution scenario.

Summarize

The construction of an automated test system is a systematic project, involving people, technology, processes and other aspects. We especially need to think about the overall situation and weigh the pros and cons, "there is no best, only the most suitable way."

First, the goals of automated testing should be aligned at the team level. Because quality assurance has a cost, it cannot be unlimited investment. Without a clear quality expectation, it is impossible to start to formulate an implementation path. Whether it is the capacity building of team members or the selection of automated testing technology frameworks, the design of the overall workflow must be based on this.

Secondly, the selection of technical framework/tools should be adapted to the existing or target workflow. For medium and large teams that want to build an integrated R&D work model, cross-functional collaboration needs to be reflected in the selection of tools and process development, and it must be aligned with the business, and the continuous efficiency of the team must be fully considered. For example, let automated testing become a part of DevOps continuous integration/continuous deployment. Automated testing should reflect business value (ensure coverage of business requirements), and then focus on the maintainability of test scaffolding to form a reconstruction of automated use cases/test data. System specifications for timely tuning of test execution efficiency.

Finally, the development of people, business changes bring about the requirements of technological and process changes, the final foothold still depends on the growth of people. When we formulate goals, design processes, and select tools, we will take into account the team’s suitability. Team members will also have changes in responsibilities and even skills upgrade requirements. They need to pay attention to personal experience, and guide and train them to achieve a "warm" transformation. once-popular "de-testing" trend shows that the role of testing will not disappear. Test development will ultimately be testing, only to "return" the role of testing to the roots of engineers.

Click to open an efficient cloud R&D workflow