Selection and exploration of Flutter UI automated testing technology solutions

Author: Xiaojiang

Flutter pages cannot directly use Native testing tools to locate elements, which brings a lot of inconvenience to automated testing. Although Google officially launched the Flutter driver and Integration test, there are the following problems in actual use:

• Not applicable to hybrid stack apps. Although there is a related driver in appium, the environment cannot be switched;
• The element positioning ability is relatively weak;
• Depend on VMService, need to build Profile or Debug package.

Based on the above factors, we did not directly use the tools officially launched by Google, but chose to extend the testing capabilities of Flutter pages based on Native testing tools. This article analyzes the principle and implementation of Flutter driver and Integration test, and briefly introduces Xianyu's trial scheme in UI automation testing.

Flutter driver

When we first came into contact with flutter automated testing, we first tried to use the appium framework to drive the APP. When we used the inspect function to dump the page elements, we found that many elements would be merged into one area block, and then when clicked, it could only be located by xpath, and wanted to locate it. Some specific elements are more difficult, and xpath is actually easy to change, and the code maintainability is poor.

Because of the above reasons, we started to investigate flutter driver, the official testing tool provided by Flutter. When I first used the framework, I found that it can only be applied to pure Flutter applications, and it is not suitable for mixed stack applications, but the element positioning capabilities provided by its underlying layer may be useful to us, so we analyzed its source code. The framework The schematic diagram 1 is shown below.

Figure 1 Schematic diagram of flutter driver

The process interaction of the entire framework is relatively simple. When the test script is running, it first uses FlutterDriver.connect() to connect to the VMService to obtain the related isolate, and then uses the websocket to transfer the operation process and data acquisition. Among them, all operations on the test script side are serialized into json strings and passed to ioslate through websocket to be converted into commands for execution on the APP side. For example, if we want to obtain the text content of a certain component, the final json structure generated is as follows :

{
"jsonrpc":"2.0",
"id":5,
"method":"ext.flutter.driver",
"params":{
"finderType":"ByValueKey",
"keyValueString":"counter",
"keyValueType":"String",
"command":"get_text",
"isolateId":"isolates/4374098363448227"
    }
}

After understanding the above principles, you can drive flutter testing in any language and test framework by constructing the protocol format, so we encapsulate this protocol and use Python to drive it, so that you can use uiautomator2 and facebook-wda On the basis of testing flutter pages, to meet the testing requirements of flutter hybrid stack applications. The final implementation code demo is as follows.

from flutter_driver.finder import FlutterFinder
from flutter_driver.flutter_driver import FlutterDriver
import uiautomator2 as u2

if __name__ == "__main__":
    d = u2.connect()
    driver = FlutterDriver(d)
if pageFlutter is True:  # 如果是flutter，则使用flutter driver进行驱动
        driver.connect("com.it592.flutter_app")
        finder = FlutterFinder.by_value_key("input")
        driver.tap(finder)
        time.sleep(1)
        print(driver.getText(FlutterFinder.by_value_key("counter")))
else:
        d(text="increase").click()

We tried to use this set of frameworks and found that the underlying capabilities provided by the flutter driver are relatively weak and cannot fully meet our needs. The main problems are as follows:

• Elements cannot be manipulated in batches. Once the finder locates more than one element, an exception will be thrown;
• In many cases, development students do not write keys, and element positioning is not so convenient;
• Because flutter does not have an inspect tool dump element, it can only use the source code to write scripts, and the code maintenance cost is relatively high;
• The official has given up maintaining the project, so it is estimated that there will be no new feature support in the future.

integration_test

As mentioned earlier, flutter officially gave up maintaining the Flutter driver and launched a new test framework integration_test. Will this framework support mixed stack applications? In fact, after trying it out, things are not as beautiful as we thought. There is a sentence in the official document "This package can perform self-driving tests on Flutter code on devices and simulators".

The operation and positioning of the underlying elements of integration_test is still driven by flutter_test, and its advantages are as follows:

• Test scripts can use various Flutter APIs;
• After packaging ipa and apk, you can run tests on Firebase Test Lab and other device groups without additional drivers;
• There is no correlation between the tests between each page of integration_test, and single-page-level tests can be implemented.

However, because the underlying element positioning is the same as that of the Flutter driver, the problems of the Flutter driver still exist, and there are other limitations:

• The test script is packaged into the APP, and every time the script is modified, it needs to be repackaged;
• It is not friendly enough for end-to-end testing, and additional functions are needed to wait for the data to be loaded;
• Not suitable for page testing at the full link level;
• Scalability is weak.

Based on the above problems, it does not meet our needs, so we just did a simple pre-research without in-depth understanding and application.

Xianyu UI Automation Test Solution

After learning about the relevant testing framework officially launched by Flutter, we began to think about how Xianyu UI automation should go? Is it to build wheels on the official shoulders or reuse existing native automated testing capabilities to expand Flutter's testing capabilities. After comprehensively considering the investment cost and the difficulty of maintaining the test script, we chose to use image processing technology to expand the native automation framework's test capability support for Flutter pages. The entire test solution architecture is shown in Figure 2.

Figure 2 Xianyu UI automation test solution architecture

Flutter elements are not completely unrecognizable by uiautomator2 and facebook-wda, so when writing test scripts, you only need to deal with unrecognized elements. For the positioning of elements with name, label, and xpath that are not easy to change, we give priority to using native positioning capabilities for positioning operations, and other elements directly use image processing technology for positioning operations.

When dealing with elements that cannot be located using native capabilities, we give priority to ocr text matching for positioning, which has high accuracy and is not easily affected by resolution. For pure pictures, we use image search for positioning. For some common element controls such as commodity cards, prices, icons, avatars, etc., we build a training set and use image classification to determine the type of elements, so as to realize the positioning of common controls.

The biggest problem facing UI automation is that with the iteration of the version, the test script also needs to be iterated continuously. Therefore, the robustness and maintainability of the script should be considered in the process of program selection and script writing. In the actual script development, we encapsulate the page elements into separate classes and separate them from the test logic, so as to ensure that only the corresponding page elements need to be modified during the later element iteration, reducing maintenance costs.

Figure 3 Script hierarchy

The related UI operations of Xianyu performance automated testing have already used this solution. When scripting, there is no need to distinguish what type of current page is. Our script has been running stably 500+ times with a success rate of over 98%.

Summarize

Figure 4 Comparison of schemes

It can be seen from Figure 4 that both the flutter driver and the integration test are not mature enough to support the hybrid stack, but the flutter driver can be extended. For pure Flutter applications, this solution can basically meet the test requirements, while the integration test is relatively It is not that mature. For the testing of hybrid stack applications, it may still need to consider the cost of scene switching of hybrid stacks. Using some OCR technology to do some expansion may be cheaper and more profitable.

Thanks to

Thanks to the many underlying capabilities provided by SLM and TMQ, so that we can concentrate on doing business.

• flutter driver(https://flutter.dev/docs/cookbook/testing/integration/introduction)
• integration_test(https://flutter.dev/docs/testing/integration-tests)

, 3 mobile technology practices & dry goods for you to think about every week!