The division and integration of the road of quality system construction

foreword

Jia Pengbo is the test development engineer of Shengwang, responsible for the testing of Shengwang SDK and the establishment of sound related test tools. I have worked in a financial company, read stocks, calculated commissions, and have experience in content distribution, e-commerce, and small games.

This article is based on the reorganization of the content shared by Jia Pengbo in the "Soundnet Developer Entrepreneurship Lecture • Phase III丨How to Ensure Product Quality in the Early Stage of Entrepreneurship ". Follow the official account "Shengwang Developer " and reply to the keyword " JT0611 " to download PPT materials related to the event.

01 Background of Quality System Construction

First of all, as shown in Figure 1, the workflow of requirements is very long from the creation of requirements, to evaluation, establishment, distribution of requirements, to the final test. We need to build and track the workflow to better ensure Business runs. Second, data logging is used for replay, improvement, and efficiency. Because the construction of the quality system covers all aspects, including the implementation of some documents, with the increase of release and test tasks, various versions and test data will be accumulated accordingly. In this process, data analysis is required. Acquire data, analyze, review, and improve to promote the construction of a quality system.

Finally, the construction of the quality system is to obtain high-quality products. From research and development self-test, promotion of the test phase, alignment of requirements, development of test specifications, Jira information visualization and tracking, and follow-up in other aspects. All the services we do are for the launch of the version, providing users with high-quality, highly reliable products and services.

■Figure 1

Figure 2 comprehensively shows the degree of quality, online problems, closed loop of test problems, process improvement, use case increase, and how to coordinate manual and automated cases before and now when the quality system is established. Through the "previous" graph, it is difficult to draw relevant conclusions. Only by establishing a system for continuous analysis can we obtain corresponding conclusions or indicators for improvement. From Figure 2, we can see how it was done "before", what effects can be achieved by establishing the system now, and the measures that need to be done to realize the system. These measures are various, and not only need to be implemented, but some may also need to be continuously aligned and reviewed.

■Figure 2

02 Methodology

1. Run small steps

Small steps are used to quickly go online. Sometimes the needs may be very urgent, just simply understand the purpose, and start working after connecting with the development or product manager, and the test cases may not be maintained and updated in time. And because it is a rapid growth stage, it may also need to pass parallel.

The so-called parallel is shown in Figure 3. Multiple people have N requirements, and these N requirements are tracked by different people, but these requirements are all new, and they only need to be added and launched quickly. The characteristic of this period is that the requirements are newly added, and you can only test the part that you are responsible for. Other requirements have nothing to do with you, and you do not need to return. It should be noted that because we only focus on quick launch, as long as a certain quality is guaranteed, the control of details is not in place.

■Figure 3

2. High quality of "win" while maintaining stability

When a company is in a period of peace, focusing on quality requires many considerations. A requirement may be split into N sub-requirements when initially established, and these sub-requirements are tested by different people. In addition, in the process of testing, continuous interaction is required, and delivery is achieved through different test points and covering different test contents. The entire testing process may be relatively long, because multiple aspects of verification are required, including manual testing, automated testing, performance Tests, laboratory tests, etc., use the comprehensive test results to determine whether the version has rollbacks or quality problems. If the overall is in good shape, it can be released.

This method of seeking "win" while maintaining stability is more suitable for companies with strict quality requirements, such as banks. Because of their strict control over money, its continuous cycle is very long, which is very important for quality problems in the testing process. Attention, the need for different people to cross and complete related test tasks by exchanging test content and test cases are the characteristics of this period.

3. Documentation of the normative process

At this time, the company is still in a period of peace. In the process of running small steps and long-term iteration of testing requirements, it has accumulated various experiences. These experiences need to be transformed into process improvement and function improvement. How to find out? More problems are the test of requirements sorting and data recording capabilities, as shown in Figure 4:

■Figure 4

Design test cases from the requirements. The test cases may include Xmind and Excel, and then develop review-related cases. At this time, it may be necessary to improve the case from the perspective of code design, product use and QA, and modify it to meet the online requirements. Of course, various problems may be encountered in the process of going online. These problems need to cover functions and performance, and require that in the process of continuous iteration (whether it is a test environment, a pre-release environment, an online environment, etc.), the functions are all It's normal, no rollback problem occurs. The main feature of this stage is that it needs to be tracked, summarized and summarized from the beginning to the end of the requirements. At the same time, it is necessary to sort out the documents and pay attention to the specifications, avoid duplicate documents, etc.

■Figure 5

Figure 5 records the number and proportion of bugs in the current version. The number of bugs of different teams can reflect their problem solving time to a certain extent, which is helpful to summarize the test progress and control the highlight task, thus promoting the function launch as scheduled. Figure 6 records the problem solving time and the variance of the problem solving time. In fact, the document data recorded by the document group is to better promote the test progress and ensure timely or real-time launch.

■Figure 6

4. A hundred flowers blooming tool flow

At present, various test tools have been launched, and corresponding test tools can be used according to different businesses to improve efficiency and deliver test tasks in a timely manner. Figure 7 shows the eight categories of test tools, this is just a simple list, there are far more test tools on the market.

■Figure 7

5. The development flow of excellence

At this time, the company may already be in a mature stage, and the current tool flow can no longer support testing tasks. It needs to be developed for the business with the help of tool flow, such as STF (Smartphone Test Farm) and Tencent's Bugly, all to fit their own business. the secondary development. Of course, there are also many tool flows, including sonic, httprunner, etc., which involve all aspects, including interface automation performance, performance testing, etc., which can be used for secondary development according to the business. The main feature of this period is that the requirements for code are relatively high, and it is necessary to fully understand the business and meet the needs of the business through code capabilities.

Figure 8 shows the secondary development of our company's current tools and related practices, which I will briefly explain later.

■Figure 8

03 Landing and practice

1. Real-time and efficient closed-loop processing of SDK crash data

Our company mainly tests the SDK, which may be different from the apps in the major markets. Its implementation practice is mainly the real-time and efficient closed-loop processing of SDK crash data. There are some such tools on the market, such as Bugly, but we found that Bugly still has some deficiencies in the process of dealing with SDK crash reports. For example, the SDK crashed, but Bugly did not monitor it. Therefore, we have made some improvements in this area. The overall process is shown in Figure 9. Our SDK serves customers and is embedded in the customer's App. Its entire closed loop is that the program is executed normally to monitor whether a crash occurs. Then find the daemon-aware capture by monitoring the crash status, and then generate a dmp file and submit it to the crash collector to analyze the relevant system. At this time, some relevant information will be stored, and finally the information will be integrated and sent to the user.

■Figure 9

Figure 10 shows the real-time and efficient closed-loop processing of SDK crash data. First, the key information of the stack is extracted, and then it is judged according to the hash value. If the same version of the hash value exists, the relevant JIRA will be associated. The processing logic of this part is to provide the crash log to the relevant developers for timely processing after obtaining the crash log.

■Figure 10

Figure 11 shows the display and assignment in JIRA after collecting the logs and compiling the crash through the parsing system. First get the crash information, then report it to JIRA with some key information, and notify the relevant developers to fix it. To put it simply, the real-time and efficient closed-loop processing of SDK crash data is to determine whether the crash is caused by our SDK after the SDK is embedded in the App. Because some may be the crash of the client App itself, we will judge this part. The crash of the SDK will be directly reported and then parsed by the compilation system and reported to JIRA. During this process, it will be submitted to the relevant developers for processing to complete the entire link.

■Figure 11

2. Circuit plugging

This part mainly introduces the test of plugging and unplugging the headset, because our SDK may need to plug and unplug the headset continuously during the test process. Figure 12 is a simplification of the circuit diagram. In simple terms, the line in the headphone is welded to the line in the circuit diagram on the motherboard circuit.

The original circuit diagram is very complicated. Students who have studied integrated circuits should know that there are various nodes in it. Test these points. Combined with the principle of the circuit diagram, the function of plugging and unplugging the earphone can be achieved through the program control circuit. We spent a lot of manpower in the process of testing plugging and unplugging headphones in the past. It is conservatively estimated that one person can complete all the cases only after 3 days of being online. But after completing this circuit diagram, it only takes an hour to complete all the related cases. On the circuit diagram, we will integrate the relevant data display (including the analysis of web page data, and the tailoring of performance indicators, etc.), which is equivalent to the productization of headphone plugging and unplugging.

■Figure 12

3. HENGE test platform

Figure 13 is the cloud platform based on the secondary development of STF. Its main link is to dynamically obtain the test package to obtain its information, and then dynamically send it to the cloud real machine for testing, and then submit the relevant test results to the relevant testers. It is convenient for them to obtain the test report in order to draw the test conclusion. The figure mainly shows the app distribution process we tested after packaging and running, and viewing the report after the running is completed, of course, including the crash log and related functions.

■Figure 13

4. Moonlight

Our company's product Moonlight has been open sourced ( https://github.com/AgoraIO-Community/MoonLight ). It is an SDK for automated performance data collection. Interested students can view the relevant code. Some performance tests on it show that it can get very accurate CPU, memory and other system information with low resource consumption.

04 System Diagram

From why the quality system is done, to the different strategies that may be adopted in the five periods, and the various gadgets that may be born during the strategy process, all of which serve the high-quality products we provide. This is the only indicator at the end. . All testing tasks, including the construction of quality systems, are to maintain current business development. Therefore, no matter which strategy is used, whether it is documentation, tool flow, or secondary development, all nodes or task events serve their own business system.

Figure 14 shows a quality system, including requirements phase, development phase, code integration, testing phase, pre-launch, delivery, quality tracking, which is a milestone from 0 to 1, because you can see the specifications in the requirements management development process Code access principles, various test index items for testing, various alignments before going online, report summary and review after delivery, and monitoring of the entire data after the final launch has completed a basic quality system diagram.

■Figure 14

06 Q&A session

1. How to do coverage check?

The coverage check uses the tool bullseye. This tool will generate a Cov file during the CI editing process. The coverage of the code developed by the corresponding module in the Cov file of the relevant case will change. Finally, all modules will be run according to the standard. Match and compare.

2. In which stage is CI/CD mainly placed in startup companies?

For entrepreneurship, it may be put into the testing stage. We are mainly responsible for the CI package, because after the SDK compiles the package, it will directly generate the relevant zip package and trigger the compilation of related apps. These are our testing tools. Because Shengwang mainly does SDK testing, we will embed our SDK into the self-developed test app for testing tasks. Of course, our company divides it into two sets. The development department has related UT, and after submitting the code, there will be related code inspection. If the UT cannot be submitted, this is the development part; the testing part is mainly used for tool outsourcing, Because of the particularity of our company's SDK testing, if most of the workers on the market do not need to embed the SDK, they are apps developed by their own functions, which are generally divided into two parts. After submitting the code for development, some people may conduct a test case, and there may be a backup test environment before the test environment, and there will be related test tasks in this environment. Run through the relevant test case and reach a certain pass rate, and then submit it successfully and proceed to the next test. Of course, it depends on whether you have the energy to maintain two environments, because this environment is before the code is tested, and the related requirements will be higher. If the developed code is UT passed, and then released to the test environment to run the test case, the case may be divided into two types, one is the newly added function, this is impossible, because there is no new function at all. The other is an existing old function. You may need to tag it to determine whether to cover the test case. If it is an old function, you can set the general rate to determine whether to call it back, so it needs to be adjusted according to different business types. .

3. How to measure the end-to-end performance test? Are there tools? What should I do if the data is not buried?

There are many end-to-end performance testing tools. It seems that someone has shared Android tools in the test home. You can take a look. That tool is open source, and it will display relevant data when running the relevant app after installation. The other is the open source tool tidevice, which obtains data through Apple's API called by Python. If it is iOS or Mac, moonlight can be used, and Windows itself has some performance tools that can be directly monitored. If you want to try secondary development, iOS will adjust its own private api, and Android will call adb. Apple also has an official instrument, and both iOS and Mac obtain new data through the call of the instrument.

Regarding the problem of inaccurate burying points, in fact, burying points is more of a development task, because development is the log writing of related tasks in the code. I think it should be developed to bury these points, and then there will be a series of It's some call chain. Our task is to verify whether it's buried correctly by understanding the development requirements and then getting this data.

Upcoming Events

On the afternoon of July 16th, the 4th session of Shengwang Developer Entrepreneurship Lecture will be titled " How can the entrepreneurial team ensure the safety and compliance of product business? " Everyone brings wonderful sharing.

Action is worse than heartbeat, hurry up to scan the QR code to sign up!