Tencent's best practices for improving quality and efficiency: the exploration and construction of intelligent automated testing

The status quo and challenges of automated testing

With the continuous improvement of people's requirements for quality of life, the frequency of product updates on the market has also increased, which is not a small challenge for product developers. And we explore the meaning of automated testing to help businesses occupy the market with better quality and faster speed.
Compared with manual testing, automated testing has higher test coverage, and the test efficiency is higher when facing TOP300 and TOP500 models. At the same time, automated testing can cooperate with continuous integration tools to achieve a fast response version. Once a version is reached, a regression test can be performed, and even a code submission can perform a regression test. And automated testing can also integrate multiple special tests.
In recent years, within Tencent, the demand for automated testing has also continued to rise. Judging from the data from 2019 to 2020, the number of automated test cases for Tencent's head projects has increased by 50%, and the frequency of automated testing has increased by 200%. This article will comprehensively analyze how Tencent's internal automated testing is explored and constructed from multiple perspectives.

Exploration of Intelligent Automated Testing

Tencent has a large number of internal products and a wide range of coverage types. There are differences in access methods, test scenarios, and test requirements. An excellent automated test platform needs to help the business solve the following problems:

1. Efficiency: Business access is convenient and fast, and supports multiple access methods
2. Available: Stable execution of test tasks: 99.9%+
3. Effect: easy problem analysis and positioning, easy regression
   So how does WeTest, as a one-stop testing platform recognized by many well-known products of Tencent, build a technical solution for intelligent automated testing?
   01
   Full link test capability construction
   The WeTest automated testing platform provides a series of automated testing capabilities from script writing to task triggering, task execution, task reporting, and finally to problem debugging. At the same time, it also provides access methods at different levels from IAAS, PAAS, and SAAS. At the IAAS layer, WeTest provides a series of hardware facilities including signal shielding cabinets, constant temperature laboratories, and domestic and foreign network dedicated lines; at the PAAS layer, it provides different access methods. The SAAS layer provides automated testing capabilities for the entire link. Services can use these underlying hardware and devices only through the IAAS layer access, or they can connect to the entire test platform through the PAAS layer, and they can also use these automated test capabilities directly through the SAAS layer.
   02
   Multiple access mechanisms to meet the docking of various systems
   Facing different business access requirements, WeTest provides different access methods. For example, the CI plug-in, as a link in the CI automation test, can smoke or return to the version build on a daily basis; in the face of a business with a test platform, WeTest can provide a REST API for platform-level docking; For different needs on the business side, WeTest also provides automated tests that can be submitted on the web page with one click, and command-line tools that can submit automated tests by executing a simple command.
   03
   Mirrored test framework deployment
   When performing automated testing, the automated testing technologies of different business teams are very different, and the script frameworks, scripting environments, and terminal equipment requirements they use are different, and some user scripts may even have some security risks. .
   In order to solve such problems, WeTest provides a mirrored deployment plan. When the business team deploys the test framework, it only needs to generate the corresponding image to use it. The scripts or test framework libraries that need to be relied on in the mirrors only need to be packaged into mirrors by writing DockerFile, and then uploaded to the mirror warehouse of the platform. When a user submits an automated test, the platform system will pull the image required by the user from the user's mirror warehouse, and then run in the form of the pull in the entire K8S cluster.
   Using the K8S cluster can also automatically perform load balancing, so that the containers for task execution are evenly distributed on different nodes, and it is also very convenient for expansion.
   04
   Configurable test scheduling
   Equipment, use cases, and scripts are all resources that can be configured and scheduled. For example, when a user needs to schedule a use case, he only needs to configure the required execution use case and the equipment execution conditions required by the use case, and the platform will follow the historical execution time of the user use case, the conditions required by the use case, the estimated result of the use case, and the performance of the use case. The execution method is optimally allocated to the device, thereby accelerating the execution speed of the use case. For the scheduling of models, the platform will select devices that are both idle and can meet the user's test requirements based on a certain algorithm, thereby improving test efficiency.
   05
   Centralization, networking, clustering
   At present, in most automated test platforms in the industry, the hardware connected to the platform facilities is a splicing of consumer products, resulting in the availability of mobile phones depending on the availability of all hardware in the entire link. When one of the hardware is unavailable, the phone cannot be used. This will not only make a single point of failure become the bottleneck of the entire mobile phone, but the overall control link will also be very long. Faced with such problems, WeTest proposed a new plan of "hardware data centering, mobile phone control networking, and execution machine clustering".
   06
   Comprehensive test report
   In order to enable the business to quickly and conveniently find problems, locate problems and perform regression tests after the business completes the test, WeTest provides a full range of test reports. Including equipment logs, performance data, network data, screenshot data, error stack information, playback of the entire test process, etc. In response to the problems of individual mobile phones in the report, the platform also provides remote debugging functions, supporting high-definition picture and audio input and output.
   07
   Proprietary Cloud
   For some special needs of business confidentiality and equipment timeliness, the platform can provide a proprietary cloud exclusively for this business. Not only can it meet the confidentiality requirements and the timeliness of the device with zero waiting, it can also perform special configuration and deployment of the device to meet the needs of user-specific application pre-installation, non-uninstallation, permanent login status, automatic chat, automatic audit and other demand scenarios.

Intelligent automated testing

In order to better guarantee the quality of users' products, and give users a better platform experience. We have been researching, exploring and thinking about automated testing technology, and we are constantly pursuing technological progress.
For the areas explored by low-code tools and AI, we then developed a new generation of low-code automation tools for script recording-WeAutomator. This low-code tool contains a variety of script recording functions, including the most basic integrated recording through image, space and coordinates, and can also provide intelligent recording based on image edge algorithms. And this tool supports scene and code mode, supports Android and iOS.

Case Studies

King Glory skin automated test
The screen on the left is the real game screen of the user, and the screen on the right is a special test map provided by Honor King for automated testing of this screen. It changed the time into a logical title, removed the noise of the map, turned the bottom surface into a uniform blue, and concealed the heroes, blood volume, hero names, etc. that do not need to be tested by the enemy.

During the entire automated test process, the system will record. After the recording is completed, each frame of each video will be extracted through OCR recognition, and then all the picture frames will be saved. The pictures between different iterative versions will be compared through the feature matching of the ORB and the precise comparison of the pictures, and the picture comparison library will be used to screen out the different pictures for annotation, and then determine whether there is a UI abnormality. The bug discovery ability of the entire automated test is very strong, equivalent to 5 times the manual ability. At the same time, because King Honor has more component resources, an automated test can save 160 hours compared to manual testing.
WeChat client automation test case
WeChat-related end-to-end UI automation testing mainly involves: WeChat basic client, payment UAT, applet basics, search and other business tests. Generally, UI automation testing needs to go through four steps: select UI-driven tools, select use case organization tools, write use cases and use case deployment. The WeChat client project team chose 3 opportunities to participate in automated testing when deploying use cases based on its own business conditions.
1. Check in
Compilation and startup problems will be checked every time the development submits MR. This time is mainly to check whether WeChat start/message is successful, and use 3 use cases + login to test.
2. Red version test
On this version, it is necessary to ensure that the basic functions of the application are stable and available, and to allow new features or some scenes with bugs. After compiling the package, intervene in a selected use case (about 40 use cases) for testing to verify whether the basic functions are executed successfully, mainly to detect whether there are some problems that obviously affect the experience.
3. P0 test
The requirements that have passed the test are merged into the release branch to wait for release, and the package will be triggered after the merge. Intervene to perform P0 automated tests (about 400 use cases) after compiling the package, and the test can be guaranteed to run in about 26 minutes.

After WeChat is connected to WeTest’s automated testing, it can deliver use case distribution, use case result storage, test reports and other functions to WeTest, and then have more energy to focus on the business itself. WeTest provides highly stable equipment and automated equipment operation. With dimensional monitoring, the test stability reaches 99.68%, and the test efficiency is increased by 40%, allowing users to inspect product quality more efficiently and conveniently without worrying about equipment problems.

Explore the results
1. Equipment dimension: The equipment has high stability, and the equipment success rate and test task execution rate are as high as 99.9% under the condition of an average monthly test volume of 100w+ sets
2. Use case latitude: use case coverage is high, through the use of dynamic distribution of use cases, shorten the test time, support flexible use case distribution mode
3. Efficiency dimension: more resources, faster speed, 40% improvement in test efficiency

4. WeTest currently covers the 6 major BGs of Tencent
   

About WeTest

Tencent WeTest is a one-stop quality open platform officially launched by Tencent. More than ten years of experience in quality management, dedicated to quality standard construction and product quality improvement. Tencent WeTest provides mobile developers with excellent R&D tools such as compatibility testing, cloud real machine, performance testing, security protection, etc., and provides solutions for more than 100 industries, covering the testing needs of products at various stages of R&D and operation, and has undergone thousands of products. . The gold medal expert team guarantees the quality of your products 360 degrees through 5 dimensions and 41 indicators.

Follow Tencent WeTest to learn more about popular test products