Grayscale release architecture design! Worth tasting

Definition of grayscale release

Internet products need to be developed quickly and iteratively and online, but also to ensure quality, to ensure that the newly grayscale publishing system .

The role of the grayscale release system , according to the configuration, can direct the user's traffic to the newly online system to quickly verify the new function, and once a problem occurs, it can be repaired immediately, in short, it is a set A/B Test system.

Gray release is allowed to go online with bugs, as long as the bug is not fatal, of course, if the bug is unknown, if you know it, you will quickly change it.

Design of Simple Grayscale Publishing System

The gray scale simple architecture is shown in the figure above, and the necessary components are as follows:

• 1. The strategy configuration platform, storing the gray strategy
• 2. Execution program of gray scale function
• 3. The registration center, the registered service carries ip/Port/name/version

With the above three components, it can be regarded as a complete grayscale platform

Grayscale strategy

The gray level must have a gray level strategy. The common methods of gray level strategy are as follows:

• 1. Traffic segmentation based on Request Header
• 2. Traffic segmentation based on cookies
• 3. Traffic segmentation based on request parameters

For example: take the modulus according to the user uid carried in the request, the gray scale is one percent, then the uid modulus range is 100, the modulus is 0 to access the new version of the service, and the modulus is 1 to 99 to access the old version of the service.

Gray release strategies are divided into two categories, single strategy and combined strategy

Single strategy: For example, take the modulus according to the user's uid, token, and ip

Combination strategy: multiple services are grayed out at the same time. For example, I have three services A/B/C, and I need to gray level A and C at the same time, but B does not need gray level. At this time, a tag field is needed. The specific implementation is Detailed below

Gray release specific execution control

In the above simple grayscale publishing system architecture, we learned that grayscale publishing services are divided into upstream and downstream services. Upstream services are specific programs that execute grayscale strategies. This service can be nginx or microservice architecture. Gateway layer/business logic layer, let’s analyze how different upstream services are implemented

Nginx

If the upstream service is nginx, then nginx needs to extend nginx through Lua to implement the configuration and forwarding of the gray-scale strategy, because nginx itself does not have the implementation of the gray-scale strategy

The implementation of the gray-scale strategy is realized through the lua extension, but the problem is coming again. Nginx itself does not have the gray-scale strategy for receiving the configuration management platform.

Solution: Deploy Agent locally (you need to develop it yourself), receive the gray policy issued by the service configuration management platform, update the nginx configuration, and gracefully restart the Nginx service

Gateway layer/business logic layer/data access layer

It only needs to integrate the client SDK of the configuration management platform, receive the gray policy issued by the service configuration management platform, and execute the gray policy through the integrated SDK.

Publish complex scenes in grayscale

Let’s take an example of two slightly more complicated grayscale publishing scenarios. The grayscale strategy assumes that the uid modulates one percent of the grayscale users.

Scenario 1: Invoke multiple services on the chain at the same time grayscale

The function upgrade involves multiple service changes. The gateway layer and data access layer are gray-scale, and the business logic layer remains unchanged. How should the gray-scale be performed at this time?

solution:

After the request of the new version of the gateway layer, all are tagged with tag T, and forwarded according to tag T in the business logic layer, all requests for tag T are forwarded to the new version of the data access layer service, and requests without tag T are all forwarded to the old version of data access Layer up.

Scenario 2: Grayscale service involving data

When it comes to the gray-scale service of data, you will definitely use the database. When you use the database, the table fields before and after using the database are inconsistent. My old version has three fields A/B/C, and the new version is A/B/ C/D four fields. At this time, the new version of Grayscale cannot be modified to the old version of the database. At this time, you need to copy the data to do this.

The database does not actually have the concept of grayscale. At this time, we can only copy the data for reading and writing, because at this time your writing must be full (double writing), and it cannot be said that 90% of the data is written to In the old version, 10% of the data is written to the new version, because at this time you will find that the data in the two databases is not full.

There will be data loss during the offline full copy of data. At this time, the business logic layer needs to write a copy of data to MQ. After the data synchronization is completed, the new version of the data access layer will write the MQ data to the new version. In DB, to achieve data consistency, this is also the main purpose of introducing MQ.

During the gray-scale process, it is necessary to compare the data of the two databases to observe whether the data is consistent. In this way, whether it is gray failure, abandoning the new version of DB, or gray successfully switching to the new version of DB, data will not be lost.

Address: http://www.fblinux.com/?p=1672 Author: Ximen Feibing