Online traffic comparison application practice

1. Background of traffic comparison

When we carry out code refactoring and requirement iteration, we need to conduct one round, two rounds and regression testing before going online. If the business scenario is more complicated, there will be the following problems:

(1) The test cycle will be correspondingly longer;

(2) With the continuous iteration of business functions, test cases may not be able to cover all, some unpopular hidden problems may not be discovered in time, and the difficulty of regression testing is gradually increasing;

(3) Affect the frequency of publication;

(4) Development and testing personnel spend a long time in the testing phase, which is not conducive to freeing up time for the next step of business iteration, technical transformation, etc.;

(5) Code quality often relies on artificial code review, with many lines of iterative code, and code review takes a lot of time;

To this end, whether we can solve these pain points, improve our work efficiency to a certain extent, and ensure the quality of our code is a question we need to think about. The industry's practice is to introduce online traffic into the comparison machine, compare the production results with the comparison machine in real time, and expose problems in time. Development and testing can be discovered and modified in time.

2. Comparison with the traffic playback platform

At present, the traffic playback platforms in the industry are basically based on traffic recording. The traffic is directed to the test environment for playback, and the problems in the system are analyzed after playback. To a certain extent, it can only release some resources for regression testing, which does not allow us to develop Existing problems can be exposed in time before testing, allowing developers to solve problems in advance and improve the quality of code testing. Our goal is to ensure the quality of our code even without testing participation. The impact of the business can get timely feedback to form a closed-loop processing, and a real-time comparison plan is necessary.

3. Traffic comparison scheme

For general business systems, the dimensions we compare generally include database landing data, mq messages, interface call request parameters and output. Generally, when the comparison traffic is 50% of the entire production traffic, it can basically cover production cases. On this basis, the consistency rate of the comparison results for these three dimensions is close to 100%, so the impact of our online iterative version on the stock business can be guaranteed. The main idea is to construct the request message, execute the request, and then analyze the comparison result.

(1) Overall structure

When the production application finishes processing the corresponding request, all the invocation of the interface, the sending of mq messages and other buried information will be recorded to the ES (since ES is a distributed, high-expansion, high-real-time search and data analysis engine, it provides interface, you can easily obtain the buried point information), at this time, an mq message will be sent, the comparison tool will monitor the mq message for consumption, query the production order data and mock according to the corresponding order information, and then call the comparison application, the comparison tool After a certain delay, compare the content to be compared, and send the comparison result to ES. You can check the comparison consistency rate and the specific reasons for the inconsistency according to the ES comparison result.

Note : Production Application: Actual Production Application Cluster

Comparing Applications: Our Iterative Version Deployment Cluster

Comparison tool: a separate application dedicated to running comparisons

(2) Data burying point

Buying data points is the premise of comparison, including the buried points of interface requests, the buried points of q messages, the buried points of input requests, and the subsequent comparison tool reads the data for comparison.

(3) Data mock

The comparison application cannot actually operate the db and interact with the upstream and downstream systems. All data needs to be mocked. The data of the mock is mainly the response data of calling the downstream interface. The data of the mock can be written to redis. The comparison application can directly process the business logic. Read the corresponding mock data of redis as return.

(4) Comparison configuration + data comparison

The comparison configuration is generally configured in the configuration center, which can be configured and managed according to the actual comparison dimension. The comparison of the comparison table fields, such as some time, can be configured to ignore the fields, and some parameters in the interface and mq can also be configured to ignore the comparison. Such as order number, delivery time, etc.

(5) Analysis of the Consistency Rate of Results

The comparison tool can send the comparison results to the ES, and we can use the ES to filter the corresponding comparison results, the consistency rate and the reason for the inconsistency according to the specific buried point actionType.

4. Summary

The online traffic comparison scheme is generally based on basic business points. For the specific interfaces involved in a business process, mq messages need to be comprehensively sorted out. It should be pointed out that if you want to really use the real-time comparison scheme to know in advance When iterating on the premise of risk, we can release comparative applications at any time, and there should be no restrictions on this. Our vision is that development can ensure the security of live code even without testing participation, and development itself can form a closed loop. There are some advantages and disadvantages of this comparison scheme as follows:

advantage

(1) Problems existing in code refactoring and version iterative changes can be discovered in time, and repaired in time. There is no need to wait until after the test to find out, and potential problems can be exposed in advance. Not having to rely too much on testing to find problems;

(2) To a certain extent, the resources for regression testing are released;

(3) Ensure the quality of the entire R&D process;

(4) The frequency of publication can be increased;

(5) The comparison application does not need to build a separate comparison database for comparison, which reduces the comparison cost;

(6) As a non-core application, the comparison application and comparison tool can be changed and released with certain flexibility;

(7) Flexible configuration of comparison content;

shortcoming

(1) There is a certain degree of intrusion into the code, and a comparative mq message needs to be issued

(2) For incremental business, because there is no comparison reference object, there is no effective way to ensure code quality, only through ut unit testing, code review and other means to assist.

Text / HUZHIMIN

Pay attention to Dewu Technology and be the most fashionable technical person!