Dewu Technology Distributed UI Automation Practice

Preface

When it comes to UI automation testing, there will always be many questions:

What value can UI automation bring? Or is it a waste of time?
What role does UI automation testing play in the entire testing process?
When I have time to write UI automation tests, I have already completed business tests. Why should I write a case for automation tests?
For business scenarios where the UI interface often changes, it is simply too difficult to write and maintain an automated case. How can we solve these problems?

......

Today, the editor is here to share with you my understanding of UI automated testing and how the distributed platform DuLab, which is being built by our quality platform, realizes batch running of UI automated test cases.

Why do UI automation?

With the continuous version iteration, more and more new software functions are added, the demand for test resources has also become greater, and the time required to perform manual tests is getting longer and longer. The reliance on manual testing became tricky, so everyone began to look for solutions, and UI automation came into being.

Disadvantages of manual testing

Manual regression testing takes a long time to complete, and a small delay will put the release at risk.
The release cadence is limited by manual regression testing. Manual regression testing for more than two days means that it can be released twice a month in the best case. Moreover, developers need to publish everything at once. Either release all or nothing, because you need to test everything together.

Advantages of UI automation testing

Free up the test team's testing time for temporary and exploratory cases;
Regression testing can be performed at the same time of development to reduce waiting time;
Repeatable use, fast regression test;
Make better use of resources (resource idle periods on weekends/nights).

Features of UI automation

UI is the abbreviation of User Interface (User Interface). What UI automation does is to simulate user behavior and perform operations to complete the test of the user interface. This essentially limits its usage scenarios:

Software requirements change infrequently
Long product update and maintenance cycle
Frequent regression testing
Automated test scripts can be reused

So before you start, it’s best to know which business scenarios can be automated~

expected result

Determine the expected effect according to the current situation of our company's business.

Compatibility test: For the commonly used models and system versions on the market, download and install them to find compatibility failures and then repair them.
Buried point test: Check whether the buried point data is reported normally, whether there are omissions, false reports, or over-reports.
Regression test: In the version iteration, regression test is performed to ensure that code changes will not cause failures in other scenarios.
Performance collection in the test phase: Specify the priority for the automation case in the test phase, and run the automation case according to the priority to provide more performance data.

Ideas

Similar to the interface automation test idea, when we are doing UI automation testing, each Case is a separate TestCase. We put all the cases that need to be executed in the same TestSuit, execute them in batches and generate aggregate reports.

difficulty

Different from interface automation, a single client device only supports running one automation case at a certain time;
Restricted by computer performance, it is impossible to control dozens of mobile devices on a single computer device at the same time;
With version iteration, some cases are only applicable to certain APP versions, and the version range needs to be limited;
The pre-step cannot be omitted. For example, if you want to automate the UI of a certain page, you cannot directly enter the page. You must start the APP and select the pre-path to enter the specified page;
The data may change in the same scenario, making the verification rules unable to be unified;
Different data under different accounts may cause many scenarios to be untestable;
Remotely control various types of mobile devices of various systems;
How to verify the embedded point data in the process of UI automation;
A large number of AB experiments, how to switch the environment for testing;
APP installation package management, how to select the installation package for overwriting installation.
......

Architecture

The following is my own step-by-step thinking process from point to surface, from basic remote management of mobile devices, case writing to case maintenance, and then to the construction of the lab platform:

Mainstream tool survey

There are many mature UI automation tools on the market, such as appium, airtest, Sikuli, etc. They provide very convenient recording services. We only need to drag and drop on the idea, and a simple UI automation script can be generated immediately , And we only need to deploy the tool library packaged by these tools on our computer to run the case we wrote multiple times.

Here we will separately analyze the principles of recorders and actuators used for UI automation:

The principle of case execution

Interpreting the source code, we found that the underlying implementation principles of these tools are basically the same. The core is: remote control of mobile devices, and encapsulation of control instructions into readable and universal grammar.

In fact, when we use mobile phones daily, the mobile phone system traverses down from the top view through the screen coordinates we click until we find the UI controls that can respond to user behavior at that coordinate position, and execute the corresponding response code.

But when we are writing a case, if all the operation behaviors recorded in the case are corresponding to a certain coordinate, it will be particularly obscure and difficult to understand, and it is completely uncommon for devices of different screen sizes, which obviously does not work. of.

The real user behavior is actually nothing more than clicking a certain button, sliding a certain page, etc. The main body is actually the UI control. We are the same when writing the case. What I want is to operate on a certain UI control, then In fact, when executing the case, we can get the position of the UI control on the current screen, and then call the underlying control instructions to operate.

The principle of case recording

In fact, the principle of case recording is the reverse thinking of the above execution principle. We can start the server service on the mobile device, transmit the screen image of the mobile device to the recorder in real time in the form of a binary stream, and cast the mobile device to our recorder. on. At the same time, it is also necessary to obtain the UI Tree on the mobile device in real time. When we move the mouse on the recorder, we use the coordinate information of the mouse on the screen to traverse the corresponding UI element from the UI tree and print all the UI information of the element.

Does it feel complicated? Don't worry, there are already mature frameworks Android Debug Bridge and WebDriverAgent that provide us with these services, which will be highlighted below.

Framework for communicating with mobile devices

Interested students can google by themselves.

Here we directly choose NetEase’s AirtestProject as the core framework of our UI automation, the reason is:

Provides a ready-made script recording tool Airtest IDE;
In addition to locating elements through path, refer to Sikuli support image recognition to locate UI elements;
Compared with Appium, it has removed support for Java, JavaScript, Objective C, Java, Ruby, php, c# and other languages, which is more lightweight;
Provides Airtest and Poco frameworks that can be directly referenced in python projects, and also supports one-click execution on the command line;
Provides a more intuitive report report.

Case management

Solve the case recording problem, let’s think about the case management problem again. During the recording of the case, we found that many cases have highly reusable modules, such as login modules. Perhaps 80% of our cases have logged in. , If the UI of the login page in a certain version is changed, which causes the path of the element to change, at this time, should we find all the cases that are designed to login and modify them one by one?

In fact, when writing a UI automation case, we also need a design pattern, referring to the industry’s excellent patterns, and combining with our company’s actual scenarios, the general idea of sorting out is as follows:
上传![

Base layer

Encapsulation of common operations: extract the repeated code and encapsulate;
Tool packaging: data verification, equipment information acquisition, installation package data query, and operation according to package downloading;
Buried point traversal package: Take a single page as a unit, get all clickable, slidable, and editable UI elements, and then simulate the corresponding user behavior;
Application installation and uninstallation package: Obtain the installation package corresponding to each version according to udid, perform overwrite installation, or uninstall installation;
System setting package: switching environment, ab experiment, network environment;
Version selection package: Specify the version interval in which case runs.

Business Layer

Page encapsulation: Divided by function, such as comment-related operations, login-related operations, etc., extract and encapsulate these common parts for all test cases to call;
Module encapsulation: enter the front roadbed encapsulation of a certain business module, etc.

Use case layer

Test case set: According to the business molecule folder, it contains all the test cases of the business.

Frame layer

suit: Store the Suit class used to organize different use case sets;
report: Generate report reports for a single case and aggregate reports for the entire suit.

APP installation package management

Access the release platform, and obtain the test package and gray-scale package information after the test in regular rotation training and drop the library. The table structure is as follows:

CREATE TABLE `lab_package` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `version` varchar(32) CHARACTER SET utf8mb4 DEFAULT '' COMMENT '版本号',
  `buildVersion` varchar(32) CHARACTER SET utf8mb4 DEFAULT '' COMMENT '构建版本号',
  `bundleId` varchar(255) CHARACTER SET utf8mb4 DEFAULT '' COMMENT '包ID',
  `pkgPath` varchar(255) CHARACTER SET utf8mb4 DEFAULT '' COMMENT '包存储路径，可以用来下载包体',
  `platform` varchar(255) DEFAULT NULL COMMENT '平台 ios/android',
  `timestamp` timestamp NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `branch` varchar(255) DEFAULT NULL COMMENT ' 所属分支',
  PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=579001 DEFAULT CHARSET=utf8;

When specifying the version range of the case, query the database for the latest package address that meets the requirements (there are multiple installation packages with different bundleIds on the iOS side, and you need to obtain the correct bundleId according to the udid of the device), then download the installation package and install it to Running on the device.

Access mock platform

When the UI automation case is running, the biggest problem is the change of the UI interface. But if we connect to the mock platform to ensure that the interface when the case is running is exactly the same as the interface and data when the case is written, then it will be easy for us to verify the verification points when executing the case.

distributed

After the previous preparations are done, we must consider the issue of case operation. Unlike interface automation, UI automation relies on real devices, but device resources are limited. When we run on a Mac, according to the performance of the Mac, we can control several devices at the same time, but when we want to run cases in batches , It may be necessary to run our UI automation case on dozens or even hundreds of devices at the same time. At this point, we need to be distributed. Build a mobile phone lab cluster. One server assigns tasks, multiple workers perform tasks, remotely control the mobile devices connected to the worker, and finally aggregate all report reports.

lab framework

Language: python

Core framework: tornado+celery+redis

Overall structure:

Lab is mainly divided into two modules, server scheduling service, and worker cluster. When the server service receives the user's execution instruction, it will go to the case warehouse to traverse the case scripts that meet the user's requirements. The broker middleware will assign tasks to the worker clusters. After the execution of each worker task is completed, the execution result of each case will be stored In the result set, after all the tasks are executed, the server will generate a total aggregate report and send the report to the user via Feishu.

The overall structure of the worker is as follows:

When the worker receives the task, it will start multiple processes to execute UI automation cases in batches on the connected mobile devices.

It is worth mentioning that the management of local resources in the worker is the management of the installation package and the management of the case warehouse respectively;

The case warehouse is actually a sub-module of the lab project. The testers write and maintain daily cases in the case warehouse project. They don’t need to care about the maintenance of the lab project at all. When a test student pushes the code to the remote warehouse, Jenkins The interface of the server service will be called, and the server service will notify each worker to update the code of the case warehouse to ensure that the code is the latest when the worker runs automated cases in batches;

Installation package management: There are two forms. In addition to the worker service regularly obtaining the latest installation package list and caching it locally every day, when the case is executed, if the local installation package cannot meet the version requirements, the worker will also traverse the database of the app installation package. Appropriate installation packages are cached locally and installed on mobile devices;

Because our app is undergoing a lot of version iterations every day, our case may only be able to run in part of the version range, so we need to limit the version range in the case at this time. When the case is running, the interpretation device Whether the installed app version is within this range, if it does not exist, download and install the version that meets the requirements to the mobile device and then execute the case;

More detailed design will not be introduced here, and interested students will leave a message for discussion~

Wen｜crystal

Pay attention to Dewu Technology and join hands to move towards the cloud of technology