"Three-level index system" for quality measurement of material and its application practice

Management guru Peter Drucker once said: There is no management without data. Numbers are an effective way for people to perceive things quickly. Whether in life or work, it is closely related to things and people. When it comes to things or phenomena that are difficult to describe with numbers, there must be no suitable indicators and measurement methods. Especially for quality engineering work, quantitative presentation is far more convincing than qualitative description. The "three-level index system" is a best engineering practice that has gradually become clear and formed in the repeated polishing of the quality platform in this year, and can systematically measure the quality of the project.

Why level three? instead of four or less?

Through the first-level indicators, the level of a certain aspect in the engineering process can be intuitively fed back. The indicators with the nature of results are similar to the role of "rearview mirror", but they also have a large lag and larger granularity; thus, the second-level indicators are introduced. , to disassemble the first-level indicators of large granularity, and divide and conquer to obtain improvement. Due to the uniformity of particle size, it is necessary to further incorporate three-level indicators to be operational, so that improvements can be made to ultimately affect the results. Usually, three-level indicators can play an effective drill-down analysis and fall into the effect of improving the closed loop.

Through practical induction, four aspects of efficiency , quality, stability and resources are selected to jointly build a three-level indicator system. The relationship of the four aspects can be described as quality is the foundation of life, stability is the natural result of effective quality activities; the business side expects fast delivery with quality (constraints), and as little resource investment as possible, which also implies The demand for efficient use of resources.

In short, it is "more, faster, better, and less" to do the work. The three-level indicators refer to:

• The first-level indicator is the result indicator. Play the role of "rearview mirror", there is a certain delay.
• Secondary indicators, namely dismantling indicators/improvement indicators. Component disassembly of the result formation or direct action to improve the result.
• The three-level indicators are improvement indicators. One or a group of improvement actions can be mapped to obtain a partial improvement in the results.

The following is a description of the corresponding three-level indicator composition, internal logic and application description for the above four aspects.

1. Efficiency

Efficiency is a hot topic of communication between engineering teams and business teams, and it is also a frequently criticized aspect of engineering teams. From the quality assurance work for efficient delivery as a cut-off, the application description is carried out.

In terms of efficiency, it is supported and guaranteed from bottom to top through business delivery capabilities, planning assurance capabilities, and process coordination capabilities to achieve the goal of high efficiency, that is, to use data to present "more" and "faster".

Business delivery and response are reflected by the relative value of throughput rate; in terms of absolute value, the total demand number is compared and observed, and the demand online rate and demand distribution reflect the ability to guarantee the plan. In terms of indicators, it reflects how many demands have been raised by the business side, how many demands have been lined up, how many major demands have been lined up, and how many have been completed on time.

Then drill down to the manifestation of collaborative efficiency, which depends on the accuracy of time estimation, and the punctual test rate in the process serves as the contract support, and the automation rate and engineering support availability affect the test execution efficiency. This indicator focuses on the ability description of the implementation process, which belongs to the "muscle" display. Explains why and how efficiency can be guaranteed. Many three-level indicators are hidden here for clear explanation (the same below).

2. Quality

The quality of software is developed, and the program (product) is solidified once it is transferred to the testing process. This refers to the built-in quality, which is the objective program quality. All testing behavior is an excavation of this objective quality. Therefore, the test behavior is to approximate this objective quality activity as closely as possible through scenario coverage.

Under normal circumstances, setting reasonable access standards is conducive to terminating the circulation of substandard products in advance, ensuring the smoothness of the process, independently measuring the quality of the test, and collecting causal data before the test is extended to form a conclusion.

Quality measurement is the caliper for quality access control, and it is a necessary guarantee for the smooth progress of quality activities in the R&D pipeline. In engineering, the input for quality activities is passive input, that is, it is indispensable, but it is expected to reduce the input as much as possible. Therefore, the setting of quality access control can be used in a relatively low-cost way to avoid unnecessary transitional resource consumption. It is like letting the car washer wash the car before waxing the car; otherwise, the waxer has to spend more time Remove the dust covering the car, and then enter the key process. As shown in the figure below, from the defect cost curve, it is an uneconomical investment.

The built-in quality is the natural description result after a series of quality activities. Among them, the defect introduction rate is the most direct description. The absolute number of defects, combined with its composition, the defect distribution, can describe a relatively clear built-in quality.

What is the role of functions? This soul torture is combined with the above figure to make necessary explanations, and expounds its internal orientation and benign promotion closed-loop logic from the perspective of value.

Test execution starts from a value perspective, reflects the value generated by the investment in quality activities as much as possible, starts from both defect value and regression value, and forms a positive closed loop to guide test design. in short:

• Find high-value defects. Combined with PRD and technical implementation, while covering user scenarios, use cases are deliberately designed for technical implementation. Such as idempotent checking, exception handling, data compatibility, etc. And refer to the coverage report to supplement use cases to ensure that the coverage is relatively comprehensive.
• Automate high-value regression use cases. Grasp the two key words of core and stability. With the increasing number of regression use cases, regular review is performed to supplement effective use cases and eliminate invalid use cases to reduce unnecessary maintenance costs.

3. Stability

Stability is generally a natural consequence of effective quality assurance. Often led by the stability or operation and maintenance team, it conducts real-time online monitoring and emergency response to faults. The number of production failures and their distribution are the main approval indicators. Policy compliance: no major problems, small problems are quickly recovered, and the impact of failures is minimized as much as possible, namely: failure impact = failure severity level x failure recovery time.

The specific indicators are that the number of P1P2 production failures is reduced or cleared, and the number of P3P4 production failures has converged and reduced. Usually, the fault grading standard, in addition to the impact surface as a dimension, such as capital loss amount, customer complaint volume, etc., will also define the fault recovery time as a necessary dimension in a step-by-step manner.

In addition, due to the hysteresis of production failures, the to-do items generated from the review of a certain production failure are measures to effectively improve the above-mentioned result indicators. Therefore, the to-do closed-loop rate is an improvement indicator in this regard. Here, the quality platform teamed up with the PMO to periodically collect and review through the "Iterative Quality Review Meeting" mechanism.

It should be noted here that systematic risk accumulation will eventually lead to a production failure. This is the "black swan event" of quality assurance work and the hardest part to explain the effectiveness of a quality assurance strategy. The quality platform follows the two-pronged approach of quality assurance and stability governance, which complement each other. Even if the local optimization reaches the level of perfection, it is difficult to resist a structural damage. The systematic review of production failures can guide the iteration of local optimization strategies, making them more solid and comprehensive. Therefore, this is a cyclic process of global and local binary complementary enhancement.

4. Resources

When it comes to resources, we mainly focus on the optimal allocation of human dimensions, that is: target-oriented, using the right people in the right places, and exerting benefits. Dynamic tuning is performed around the following three aspects.

• How many people are deployed?
• Where is it concentrated?
• How can the benefits be amplified?

This is a problem of configuring strategy, and the strategy itself originates from the goal. What kind of problem to solve in a period of time should be set and carried out in accordance with the "SMART principle".

The time-averaged number of use case executions that respectively introduce the resource development-test ratio, the pre-actual ratio, and the direct feedback function benefit correspond to the above three aspects one-to-one. in:

• The resource open-to-test ratio reflects the result of gain input (development) and passive input (test).
• The pre-actual ratio reflects resource utilization and effective investment.
• The time-averaged number of use case executions reflects the level of efficiency with which the function engages in quality activities. Rely on the comprehensive situation of built-in quality, synergy efficiency, testing strategy and supporting engineering means.

V. Application Scenario 1: Drill-Down Analysis of Iterative "Development-Test Ratio"

For example, the development-test ratio of a certain iteration (estimated) in the trading domain is 4.1:1 , which is a typical result indicator, that is, a first-level indicator. To understand its composition, drill down to the secondary indicator:

• The total estimate is 1659.7 man-days, the development estimate is 1333.7 man-days, and the test estimate is 326 man-days. The scale is extremely distributed when the projections are observed. Drill down to understand the number of throughput requirements per 10 people per day , and understand the demand throughput level of the current business domain; combine the demand dimension to develop a test ratio schedule to understand the specific throughput requirements and the distribution of resources invested.
• On- time detection rate, automation rate, T0T1 environment availability rate . Analyze the smoothness of the development process and the efficiency of test execution. Drill down to the details of the opening and testing ratio of the three-level indicator requirement dimension, and analyze the minimum granularity of each requirement to identify the points to be improved, and fall into the closed loop of to-do item tracking in the "Iterative Quality Review Meeting", as shown in the following figure:

6. Application Scenario 2: Iterative Quality Risk Warning - "Traffic Light Mechanism"

The "traffic light mechanism" is an effective practice of the quality platform in the digitization of the R&D process. According to the trend and fluctuation of indicators, formula calculation is carried out on the index data of continuous iteration as the basis for "lighting on"; it involves quantitative analysis of three aspects: access control quality, built-in quality, and synergy efficiency ; at the same time, combined with qualitative descriptions Supplementary explanation. In this way, iterative quality risk early warning can be quickly given to facilitate the formulation of risk avoidance measures before going online.

Further combine the version conclusions of the quality market to quickly understand the current iterative version quality and risk points of a certain business domain. For example, the conclusion of an iterative version of the transaction domain is shown in the following figure:

Summarize

"No data, no management". Data is a low-cost way to communicate at work. Qualitative descriptions and narrative explanations are not as direct and efficient as quantitative facts.

To sum up, the next three indicators refer to:

• The first-level indicator is the result indicator. Play the role of "rearview mirror", there is a certain delay.
• Secondary indicators, namely dismantling indicators/improvement indicators. Component disassembly of the result formation or direct action to improve the result.
• The three-level indicators are improvement indicators. One or a group of improvement actions can be mapped to obtain a partial improvement in the results.

"Three-level index system" is a best practice that can systematically measure project quality. After baseline establishment, data calibration, especially the fitting with somatosensory, this is also the main work of system polishing, which is extremely time-consuming and labor-intensive. Finally, systematic collection and visualization are carried out, so that data can serve daily work in real time.

The relationship between efficiency, quality, stability and resources can be described as the following three sentences:

1. Quality is the foundation of the functional line, and stability is the natural result of effective quality activities;
2. The constraint of cumulative efficiency to achieve rapid delivery with quality;
3. At the same time, invest as little resources as possible.

The three-level indicator system is a systematic, somatosensory fitting, and sustainable accumulation of R&D digital assets. With continuous accumulation and application, engineering problems at specific stages can be solved through local combinations, such as combined smoke pass rate, smoke defect rate, and delayed detection rate to reflect the quality of the test, that is, the quality of access control. The improvement behavior for its combined indicators is an effective measure that has been verified. Through summarization, it falls into a set of "expert opinion base", which becomes the team's experience and replicable ability, and even the intangible wealth of the enterprise.

In the same way, combined with the quality review mechanism before going online, through the three stages before, during, and after the test, selecting combinations and calculating through algorithms to form a "traffic light" prompt of the quality risk early warning effect is an effective attempt to apply data visualization.

To sum up, designing an effective indicator system and accumulating data continuously can help us choose a reasonable and scientific path to achieve each project goal.

---

Text/Bruce pays attention to Dewu Technology and be the most fashionable technical person!