Technical dry goods | Practice and exploration of automated testing in ToB business scenarios

Introduction : In the process of rapid iteration of ToB business and new version release, development and QA testing are faced with a large number of manual testing scenarios. So how to implement intelligent regression testing and achieve accurate recording and two-way traceability from test cases to code logic? How to improve test efficiency more efficiently? This is the key exploration of automated testing platforms.

Text | Zhao Chong

NetEase Yunxin Senior Mobile Development Engineer

Background

business scenario complicates

With the increasing number of B-side solution delivery scenarios, the version needs to be iterated quickly. The automation of the single-end platform has not been able to improve the efficiency. The manual regression is more about subjectively evaluating the impact of changes. In order to ensure the stability of the version, full regression testing will lengthen the B-side SDK and solutions. Release plan . The traditional automated test verification result is only a single verification. The API and UI automation scenarios on the mobile terminal are complex, the old and new versions are compatible with use case management, and the scripted implementation of use case management is inefficient. Therefore, for NetEase Yunxin, the test of the multi-terminal interoperability platform based on the mobile terminal is indispensable.

Thoughts and Challenges

abstracts the basic capabilities of the platform and supports full-process automated testing in the form of plug-in capabilities.

 

Actuator Link Environment

• Executor information used to run the test case;
• A build link for running the execution product of the test case;
• On-hook test equipment for running test cases is scheduled;

Cloud Assertion Rules

• The verification of the execution result is based on the full word-end comparison of the rules;
• The definition of comparison rules for unified execution results is supported by keywords;
• Persistent assertion parameters, and calculate verification results based on multi-terminal data;

Meta-use-case model

• According to the automatically generated interface metadata attributes, basic use cases and AI use cases are recommended and identified;
• Build business scenario models based on fragmented reuse combinations and custom use cases;
• Multi-terminal and multi-platform use case parameters are performed upstream and downstream based on platform persistent data;
• Execution set modular configuration resource package;

Result Statistical Model

• Execute task result defect location classification report;
• coverage analysis;
• Historical data aggregation model analysis;

Solution

Based on the background and thinking, the design strategy of the automation platform includes: actuator link environment, cloud assertion rules, upstream and downstream meta-use cases, scenario modeling, coverage analysis, defect location and reporting.

Platform Architecture Solution

Meta-Case Extraction Process

Automatic extraction of code, mapping corresponding basic use cases.

By configuring resource packages and rule files, combined with tools to analyze the product to be tested, test code and corresponding interface use cases are automatically generated. The method and custom attributes of the element to be tested are selected on the platform to form a meta-use case. The method attribute of a meta-use case cannot be edited. It is the mapping product of the precise test code. The cornerstone of the subsequent business model construction is the metadata based on the use case. The client interface cannot update the interface behavior in the same way as the server. must be a fixed interface behavior.

Building a business model

Refine the scenario business, do the baseline precipitation, and extract the test execution set.

Run link process

Combined with the QA local debugging process, the link of the local debugging use case is the need to support custom device resources for debugging use cases.

smart use case recommended

Persistent data logs of real users, lock the execution track of the use case link through the keyword log, and generate the corresponding use case.

 

 

Practice

Shortening the time between code completion and user feedback, minimizing the cost of ongoing work, which is also presenting the final value of the code to the customer, requires a faster and safer delivery version to maximize the benefits of the version.

Costs and Benefits (ROI) of

The benefits of automation are proportional to the number of iterations. Here is a simplified formula for a new product:

自动化的收益 = 迭代次数 * (全手动执行成本 - 维护成本) - 首次自动化成本

project

Currently implemented in engineering line solutions and instant messaging (IM1 and IM2).

Daily on-hook test

Build tasks are completed by upstream jobs, and automated tasks are performed by downstream jobs. Do not block the packaging engineering task. If the upper and lower jobs cannot be triggered, you can write the node node execution in the business execution pipeline.

Future Outlook

In the early stage, it was only aimed at the exploration and practice of multi-platform mobile terminals in the field of ToB business solutions. In the future, we hope to improve the usability and scalability of the platform while achieving real efficiency improvement. I hope that through this article, other teams can be inspired, and we will continue to explore together in the next time.

Author introduction

Zhao Chong, senior mobile terminal development engineer of NetEase Yunxin, is keen on researching cross-platform development technology and engineering efficiency improvement. Currently, he is mainly responsible for the related research and development of video conferencing component SDK.