Front-end quality improvement tool-Marco code coverage platform

This article is based on the content of Mr. Song Jiachao's live speech 2021 vivo Developer Conference The account will reply 161dba14097583 [2021VDC] obtain relevant information on the topics of the Internet technology sub-venue.

1. Background

1.1 Project Features

With the rapid development of Internet technology, more and more companies introduce agile development process into project iteration, so more and more projects show three characteristics:

1) The iteration cycle is short: various inversions, scheduling, a small version a week, a large version every two weeks, and even some projects are released every day, which is miserable.

2) Frequent changes in requirements: Our product managers are either overthrowing themselves or on the way to overthrow themselves, so we often joke with product managers and say: Come, come, make a letter, and promise not to change the requirements in the future . At this time, the product manager vowed to say: Okay, if you change this requirement, I promise to give you a document, and I won't change it again next time. But it didn't work, and the demand still changed very frequently. Of course, this cannot be blamed on product managers. Facing the ever-changing market environment, we sometimes need to use this trial-and-error approach to quickly find the right solution.

3) High system complexity: The front-end is not just as simple as drawing pages, you can see it from the current front-end recruitment information. Distributed, micro-services, middle-tier, and various architectures make an already complex business more complicated.

As these characteristics of the project became more and more obvious, the challenges for our development and testing became more and more.

1.2 Annoyance

So we will have the following troubles:

There are two reasons for this trouble in the final analysis:

tester : They don't know where the changes to the development code are and where the impact is. It is difficult to use a precise test plan to improve the test efficiency, so to be on the safe side, they can only test all relevant functions once

Developer : They cannot fully know the execution of each line of code, and it is difficult to analyze which code is useful and which code is useless by executing data, which code has been verified by testing, and which code has not been tested. verify

Therefore, we urgently need a platform that can easily see the changes and execution of the code, and this platform is the integrated code coverage platform.

1.3 Scheme

The so-called integrated code coverage platform refers to a platform that collects code coverage during integration testing and reports it to the system for combined display.

1.4 Difficulties

Since there is a solution, why are there so few such products on the market today? When we were researching this aspect, we found two very big technical difficulties.

The first difficulty is: data merging is difficult. We all know that all front-end tests are carried out at each end, and the data generated by the test is left in each terminal, and it is difficult to collect and combine the data left in each terminal.

The second difficulty is: the data expires quickly. The front-end code is released very frequently. During testing, it may be released many times a day. Although the release means that it is the same file, the content in it may be completely different, which will cause the previously collected coverage to be completely invalid. .

Through continuous attempts, we finally solved these two problems, so the Marco code coverage platform was born.

2. The advantages and highlights of the Marco platform

2.1 Advantages and Highlights

The Marco platform is a complete code coverage platform with the following highlights:

• Support one-click access: no need to choose business, framework, zero intrusion on existing code, one-click access;
• Support incremental report: It enables users to understand the code coverage more accurately;
• Supports multiple languages: The Marco platform is not limited to the front-end, but can also serve the back-end;
• Support a variety of tools: user-friendly, such as code comparison, gitblame, etc.;
• Support multiple users: real-time merging and viewing of multiple reports;
• Support market monitoring: Check the coverage rate trend in real time through the market to understand the trend of project quality;
• Support message notification: regularly push project coverage;
• Support independent deployment: not only supports direct third-party access, but also supports one-click privatization deployment.

2.2 Experience

2.2.1 Details page

• The first point on the : Which files have been changed in the current version, we will display all the changed files in the first area through a tree structure diagram, so that users can know at a glance how many files have been changed in the current version
• The second point on the : Where are the changes in the current file, we intuitively display the change points of each file in the second area through code comparison, so that users can see at a glance the changes in this file. what lines of code
• The third point on the figure below : Whether the current change has been tested and verified, just in the second point we marked which lines of a file have changed, then the third point is to mark the changed code. been verified.

With these three-point markers, we can perfectly combine the change points with the testing process, so that the test results can be seen at a glance.

2.2.2 Trend graph

Through the project trend chart, we can clearly see the trend of project coverage and understand the trend of project quality.

2.2.3 Overview

At present, the Marco platform has been running stably for nearly 3 months, serving 39 projects and generating 133 points of reports.

2.3 Value

Through the Marko platform, we will closely link the four points of concern in the test: modification points, impact points, test points and test results, and at the same time convert the previous evaluation of lack of data into accurate and objective evidence. Quantitative assessment.

3. The technical solution of the Marko platform

The technical solution of the Marko platform consists of four parts: first, the overall architecture of the Marko platform will be introduced, and then the three aspects of the access layer, the service layer and the display layer will be explained in detail.

3.1 Overall Architecture Diagram

The Marko platform is divided into three layers: access layer, service layer and display layer;

• Access layer : Our goal is one-click access, multi-language adaptation, environment distinction and intelligent reporting. At present, the adaptation for the web side has been completed, and other technology stacks are being adapted, such as applet, Node.js, and Typescript. At the same time, we have also planned access methods for other languages, so that the Marko platform can become a full-language code coverage platform.
• service layer : mainly for various processing of reports, such as report management, user management, project management and other functions.
• Display layer : It is mainly an extension of the service layer, which provides more convenient report viewing and management functions through visualization.

3.2 Data flow diagram

Through the data flow diagram, we connect a report from the access layer to the service layer to the presentation layer to form a perfect closed loop.

3.3 Access Layer

The access layer is mainly about access and reporting;

for access to : We provide instrumentation programs, instrumentation specifications, and access guidance for each language.

for reporting : There are mainly collection plans for each language, reporting data standards and reporting plans.

3.3.1 Access

Instrumentation dimension :

Usually we divide instrumentation into 4 dimensions: statement coverage, function coverage, branch coverage, and line coverage.

Instrumentation process

Marco Instrumentation

This is the Marko platform, part of the code for js instrumentation, which mainly does three things;

• first step : Get the relative path of the file

Why is it a relative path? Because Marco code coverage is a platform, we need to get relative paths so that file paths are associated with the project, not with the packaging machine.

• second step : Calling istanbul-lib-instrument for instrumentation We did not completely rewrite the instrumentation code, but used the power of open source to make the Marko platform more lightweight and universal.
• Step 3 : Add custom parameters These parameters are very important and indicate the important information of the current report.

In this way, we can simply complete the instrumentation and adaptation of the js language, and other languages can refer to this idea.

3.3.2 Reporting

Reporting is the process of collecting and sending.

aimed at collecting information, coverage information and project information, both of which are indispensable.

sending : each end will be very different, we provide some advice. For example, the Visibilitychange event can be monitored in the web, and the onShow and onHide events can be monitored in the applet. If it is in other servers such as Node, it can be reported through scheduled tasks.

3.4 Service Layer

Marco's service layer provides a lot of functions, here we pick report management and user management, these two more typical functions will be introduced.

3.4.1 Report Management

report three elements: First, a complete report needs to have three basic elements: coverage information, project information, and document information.

• Coverage information includes four coverages: statement, branch, function, and line.
• Project information includes: project configuration, branch information, file hash, build time.
• File information includes: file address, file content and Git information.

The coverage information and project information collected through the access layer, and then associated with the file information of the Git platform, can make Marco a multi-user, multi-report-supported coverage platform.

3.4.2 Merge Process

In the previous step, we collected the coverage. Now let's talk about how the Marco platform merges reports. Let's consider two cases first:

• Case 1: No file changes;

It is the part on the left side of the figure below. When the file is not changed, there are two tests. Xiaomei and Xiaohua generate two report information respectively. How to merge in this situation?

• The second case is: the file has changed;

In the same file, the content of the file has changed, and the line numbers of the changed file cannot be matched one by one. For example, the 10th line of the file before the change becomes the 6th line in the changed file. These two lines are actually The same, but because of the deletion of other lines, the line number of this line has also changed. How to merge this situation?

With the above two questions in mind, let's sort out the reporting and merging process.

The more complicated thing is to compare the report construction time. There are three results in the construction time comparison, which correspond to different merge processes.

3.4.3 Acquisition of Changed Content

In the mainstream git hosting platforms, API interfaces are provided to return Git information. For example, the comparison information of the two branches can be obtained through the compare interface of Gitlab. By parsing the returned information, we can know Changes to the file.

3.4.4 User Management

In order to make the Marco platform as simple as possible, we use the general Oauth login method to log in. In this way, not only the login problem is solved, but also the problem of code permissions is solved. It really serves multiple purposes. Of course, we are designing When logging in, full consideration is given to mainstream Git platforms, such as Gitlab, Github, Gitee, etc. That is to say, the code hosted in these platforms can be accessed to the Marco code coverage platform with one click.

3.5 Display layer

This is the details displayed by the Marco platform report, which supports real-time brushing reports and code comparison. Through comparison, we can see the content of the file changes in detail, as well as the coverage of the content.

3.5.1 Code comparison

How does the Marko platform compare the code?

We use an open source library called jsdiff, through which we can compare the difference information of two files. The result of the comparison is an object array, which is an object array, and the array contains 4 contents:

• count : number of rows;
• Value : specific code;
• Added : Whether to add;
• Removed : Whether to delete.

By parsing this array, we can restore the comparison results to the page one by one, such as the lower right corner, which is the result of our restoration through this array.

4. Summary planning

So far, we have basically finished talking about the technical solution of the Marco code coverage platform. For the Marco code coverage platform, we have two larger plans next.

1) First, enrich the access of languages at all terminals:

• Front-end includes applet, Node.js, Typescript;
• Other languages include, Go, Java, etc.

2) The second is to do open source:

• We will package the Marco platform as a whole to open source, embrace the open source community, and hope to build the Marco platform together with everyone.

The above are some of our explorations on code coverage. Code coverage can be used as an indicator of test quality, providing relatively reasonable and objective data. However, 100% code coverage does not mean 100% bug-free. There is no clear correspondence between code coverage and whether the use case is correct. It is only a supplement to the testing quality process. Instead of blindly pursuing code coverage, we should find ways to design more and better use cases, so that use cases and code coverage complement each other and improve project quality.

Author: Song Jiachao, front-end team of vivo official website mall