RocketMQ grayscale of full link grayscale

Author:

In the previous series of articles, we have introduced the full-link canary release function to introduce the full-link traffic control scenario of MSE, and we have learned about the full-link grayscale of RPC calls such as Spring Cloud and Dubbo. How should it be implemented, but it does not involve flow control in asynchronous scenarios such as messages. Today we will introduce the "How can obtain the same enterprise-level full-link grayscale capability in 20 minutes?" " to further introduce the full-link grayscale of the message scene.

Although the grayscale requirements for messages in most business scenarios are not as strict as those for RPC, in the following two scenarios, there are still certain requirements for the full link of messages.

1. The first scenario is that when a message is consumed, a new RPC call may be generated. If the previously set full-link flow control rules are not followed in the message loop, this part of the message will be generated. Traffic "escapes", which leads to the destruction of the full-link grayscale rules, resulting in unexpected situations.

In order to prevent this from happening, we need to restore the original traffic mark in the message when consuming, and follow the original rules when making RPC calls. Let's describe in detail through the architecture diagram. After this logic is satisfied, what is the call link? From the figure below, we can see that the messages produced by the grayscale and baseline environments are random when the messages are pushed. However, in the process of consumption, new RPC calls generated can still return to the environment to which the traffic originally belonged.

2. The second scenario requires stricter grayscale isolation of messages. For example, when the message consumption logic is modified, it is hoped to verify the correctness of the new message consumption logic by means of small traffic. It is strictly required that grayscale messages can only be pushed to grayscale message consumers.

Today, let's practice the full-link grayscale of the second scenario message. Currently, MSE only supports the grayscale of RocketMQ messages. If you are using the open source version of RocketMQ, the version needs to be 4.5.0 or above. If you are using the Alibaba Cloud commercial version of RocketMQ, you need to use the platinum version, and the Ons Client version must be 1.8.0.Final and above. If you only want to use the first scenario, you only need to enable the full-link grayscale function for application B, and you do not need to do additional configuration related to message grayscale.

In this best practice operation, we deploy the application in the Kubernetes version of Alibaba Cloud Container Service, that is, the ACK cluster to demonstrate, but in fact, message grayscale has no restrictive requirements for the deployment mode of the application. You can Refer to the MSE help documentation to find the access method corresponding to the deployment mode you are using. You can also use the message full-link grayscale.

Preconditions

1. To activate MSE Professional Edition, please refer to Activating MSE Microservice Governance Professional Edition [1].

2. Create an ACK cluster, see Creating a Kubernetes Cluster [2].

Steps

Step 1: Access MSE Microservice Governance

1. Install mse-ack-pilot

a. Log in to the container service console [3].

b. Click Marketplace > Application Catalog in the left navigation bar.

c. On the application catalog page, click Cloud application , select , and click ack-mse-pilot .

d. Select a cluster in the cluster list on the right side of the ack-mse-pilot page and click Create.

It takes about 2 minutes to install the MSE microservice governance component, please be patient.

After the creation is successful, it will automatically jump to the Helm page of the target cluster to check the installation result. If the following page appears, showing relevant resources, the installation is successful.

2. Enable MSE microservice governance for applications in the ACK namespace

a. Log in to the MSE Governance Center console [4]. If you have not yet activated MSE Microservice Governance, please activate it according to the prompts.

b. Select Governance Center > Kubernetes Cluster List in the left navigation bar.

c. Select the cluster name or cluster ID in the search box list on the Kubernetes cluster list page, enter the corresponding keyword, and click the search icon.

d. Click in the Target Cluster Actions column.

e. In the namespace list area of the cluster details page, click under the target namespace operation column to enable microservice governance .

f. Click OK in the Open Microservice Governance dialog box.

Step 2: Restore the online scene

First, we will deploy the four business applications spring-cloud-zuul, spring-cloud-a, spring-cloud-b, spring-cloud-c, as well as the registry Nacos Server and the message service RocketMQ Server, to simulate a real call link.

The structure diagram of the demo application is shown in the figure below. The calls between applications include both Spring Cloud calls and Dubbo calls, covering the two most commonly used microservice frameworks on the market. The C application will produce RocketMQ messages, which will be consumed by the A application. When A consumes the message, it will also initiate a new call. These applications are the simplest standard usage of Spring Cloud, Dubbo and RocketMQ. You can also directly use the https://github.com/aliyun/alibabacloud-microservice-demo/tree/master/mse-simple-demo project See the source code above.

Before deployment, briefly introduce this call link

When the spring-cloud-zuul application receives a request from "/A/dubbo", it will forward the request to spring-cloud-a, and then spring-cloud-a accesses spring-cloud-b and spring-cloud through the dubbo protocol -b also accesses spring-cloud-c through the dubbo protocol. After spring-cloud-c receives the request, it will generate a message and return its own environment label and ip. These produced messages will be consumed by the spring-cloud-a application. When the spring-cloud-a application consumes messages, it will call B through spring cloud, and B will then call C through spring cloud, and output the result to itself in the log.

当我们调用 /A/dubbo 的时候
返回值是这样 A[10.25.0.32] -> B[10.25.0.152] -> C[10.25.0.30]


同时，A 应用在接收到消息之后，输出的日志如下

2021-12-28 10:58:50.301  INFO 1 --- [essageThread_15] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:C[10.25.0.30],invoke result:A[10.25.0.32] -> B[10.25.0.152] -> C[10.25.0.30]

After getting familiar with the call chain, we continue to deploy the application, you can use kubectl or directly use the ACK console to deploy the application. The yaml file used for deployment is as follows. You can also obtain the corresponding source code directly at https://github.com/aliyun/alibabacloud-microservice-demo/tree/master/mse-simple-demo .

# 部署 Nacos Server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nacos-server
spec:
  selector:
    matchLabels:
      app: nacos-server
  template:
    metadata:
      annotations:
      labels:
        app: nacos-server
    spec:
      containers:
        - env:
            - name: MODE
              value: "standalone"
          image: registry.cn-shanghai.aliyuncs.com/yizhan/nacos-server:latest
          imagePullPolicy: IfNotPresent
          name: nacos-server
          ports:
            - containerPort: 8848

---
apiVersion: v1
kind: Service
metadata:
  name: nacos-server
spec:
  type: ClusterIP
  selector:
    app: nacos-server
  ports:
    - name: http
      port: 8848
      targetPort: 8848

# 部署业务应用
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-zuul
spec:
  selector:
    matchLabels:
      app: spring-cloud-zuul
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-zuul
      labels:
        app: spring-cloud-zuul
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            - name: enable.mq.invoke
              value: 'true'
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-zuul:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-zuul
          ports:
            - containerPort: 20000

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: slb.s1.small
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: internet
  name: zuul-slb
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 20000
  selector:
    app: spring-cloud-zuul
  type: LoadBalancer
status:
  loadBalancer: {}

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a
spec:
  selector:
    matchLabels:
      app: spring-cloud-a
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-a
          ports:
            - containerPort: 20001
          livenessProbe:
            tcpSocket:
              port: 20001
            initialDelaySeconds: 10
            periodSeconds: 30


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b
spec:
  selector:
    matchLabels:
      app: spring-cloud-b
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-b
      labels:
        app: spring-cloud-b
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-b
          ports:
            - containerPort: 20002
          livenessProbe:
            tcpSocket:
              port: 20002
            initialDelaySeconds: 10
            periodSeconds: 30

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c
spec:
  selector:
    matchLabels:
      app: spring-cloud-c
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-c
      labels:
        app: spring-cloud-c
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-c
          ports:
            - containerPort: 20003
          livenessProbe:
            tcpSocket:
              port: 20003
            initialDelaySeconds: 10
            periodSeconds: 30
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rockectmq-broker
spec:
  selector:
    matchLabels:
      app: rockectmq-broker
  template:
    metadata:
      labels:
        app: rockectmq-broker
    spec:
      containers:
        - command:
            - sh
            - mqbroker
            - '-n'
            - 'mqnamesrv:9876'
            - '-c /home/rocketmq/rocketmq-4.5.0/conf/broker.conf'
          env:
            - name: ROCKETMQ_HOME
              value: /home/rocketmq/rocketmq-4.5.0
          image: registry.cn-shanghai.aliyuncs.com/yizhan/rocketmq:4.5.0
          imagePullPolicy: Always
          name: rockectmq-broker
          ports:
            - containerPort: 9876
              protocol: TCP
            - containerPort: 10911
              protocol: TCP
            - containerPort: 10912
              protocol: TCP
            - containerPort: 10909

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocketmq-name-server
spec:
  selector:
    matchLabels:
      app: rocketmq-name-server
  template:
    metadata:
      labels:
        app: rocketmq-name-server
    spec:
      containers:
        - command:
            - sh
            - mqnamesrv
          env:
            - name: ROCKETMQ_HOME
              value: /home/rocketmq/rocketmq-4.5.0
          image: registry.cn-shanghai.aliyuncs.com/yizhan/rocketmq:4.5.0
          imagePullPolicy: Always
          name: rocketmq-name-server
          ports:
            - containerPort: 9876
              protocol: TCP
            - containerPort: 10911
              protocol: TCP
            - containerPort: 10912
              protocol: TCP
            - containerPort: 10909
              protocol: TCP

---  

apiVersion: v1
kind: Service
metadata:
  name: mqnamesrv
spec:
  type: ClusterIP
  selector:
    app: rocketmq-name-server
  ports:
    - name: mqnamesrv-9876-9876
      port: 9876
      targetPort: 9876

After successful installation, the example is as follows:

➜  ~ kubectl get svc,deploy
NAME                   TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)        AGE
service/kubernetes     ClusterIP      192.168.0.1       <none>         443/TCP        7d
service/mqnamesrv      ClusterIP      192.168.213.38    <none>         9876/TCP       47h
service/nacos-server   ClusterIP      192.168.24.189    <none>         8848/TCP       47h
service/zuul-slb       LoadBalancer   192.168.189.111   123.56.253.4   80:30260/TCP   47h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nacos-server           1/1     1            1           4m
deployment.apps/rockectmq-broker       1/1     1            1           4m
deployment.apps/rocketmq-name-server   1/1     1            1           5m
deployment.apps/spring-cloud-a         1/1     1            1           5m
deployment.apps/spring-cloud-b         1/1     1            1           5m
deployment.apps/spring-cloud-c         1/1     1            1           5m
deployment.apps/spring-cloud-zuul      1/1     1            1           5m

At the same time, we can use zuul-slb to verify the call link just mentioned

➜  ~ curl http://123.56.253.4/A/dubbo
A[10.25.0.32] -> B[10.25.0.152] -> C[10.25.0.30]

Step 3: Turn on the message grayscale function

Now, according to the prompt of the console, both the message producer spring-cloud-c and the message consumer spring-cloud-a turn on the grayscale of the message. We open it directly through the MSE console, click to enter the application details page, and select the "Message Grayscale" tab.

It can be seen that in the label ignored by the unmarked environment, we entered gray, which means that messages with the gray environment label can only be consumed by spring-cloud-a-gray, not by spring-cloud-a. Consumption.

*1. An additional explanation is needed here, because considering the actual scenario, the owners of the spring-cloud-c application and the spring-cloud-a application may not be the same person, and it may not be possible to synchronize the grayscale release of the two at the same time. operation, so in the grayscale of the message, the default behavior of the unmarked environment is to consume all messages. In this way, when spring-cloud-c is publishing in grayscale, there is no need to force the application of spring-cloud-a to be released in grayscale at the same time.
2. We give the owner of spring-cloud-a the right to choose the consumption behavior of the unmarked environment. If you need to realize that the unmarked environment does not consume the messages produced by c-gray, you only need to configure it in the console. , the configuration takes effect in real time. *

• You don't need to modify your app's code and configuration to use this feature.
• The message producer and message consumer need to turn on the message grayscale at the same time, and the grayscale function of the message can take effect.

• The message type currently only supports RocketMQ, including the open source version and Alibaba Cloud commercial version.

  • If you are using open source RocketMQ, both RocketMQ Server and RocketMQ Client need to use version 4.5.0 and above.
  • If you use Alibaba Cloud RocketMQ, you need to use the platinum version and the Ons Client version 1.8.0.Final and above.
• After enabling message grayscale, MSE will modify the Consumer Group of the message. For example, the original Consumer Group is group1, and the environment label is gray. After enabling message grayscale, the group will be modified to group1_gray. If you are using Alibaba Cloud RocketMQ, please create the group in advance.
• By default, the filtering method of SQL92 is used. If you use the open source RocketMQ, you need to enable this function on the server (ie, configure enablePropertyFilter=true in broker.conf).
• By default, unmarked nodes will consume messages from all environments. If you need to specify that unmarked nodes do not consume messages produced by a certain labeled environment, please configure "Ignored tags for unmarked environments". After modifying this configuration It takes effect dynamically without restarting the application.

Step 4: Restart the node, deploy the new version of the application, and introduce traffic for verification

First of all, because the message grayscale function of the application needs to be restarted to take effect, first we need to restart the spring-cloud-a and spring-cloud-c applications. The restart method can be selected on the console to redeploy , or delete existing pods directly using the kubectl command.

Then, continue to deploy new versions of spring-cloud-a-gray, spring-cloud-b-gray, and spring-cloud-c-gray in the Kubernetes cluster using yaml files

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a-gray
spec:
  selector:
    matchLabels:
      app: spring-cloud-a-gray
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a-gray
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-a-gray
          ports:
            - containerPort: 20001
          livenessProbe:
            tcpSocket:
              port: 20001
            initialDelaySeconds: 10
            periodSeconds: 30
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b-gray
spec:
  selector:
    matchLabels:
      app: spring-cloud-b-gray
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-b
      labels:
        app: spring-cloud-b-gray
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-b-gray
          ports:
            - containerPort: 20002
          livenessProbe:
            tcpSocket:
              port: 20002
            initialDelaySeconds: 10
            periodSeconds: 30

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c-gray
spec:
  selector:
    matchLabels:
      app: spring-cloud-c-gray
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-c
      labels:
        app: spring-cloud-c-gray
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-c-gray
          ports:
            - containerPort: 20003
          livenessProbe:
            tcpSocket:
              port: 20003
            initialDelaySeconds: 10
            periodSeconds: 30

1. After the deployment is complete, we introduce traffic and verify it

2. Log in to the MSE Governance Center console [4] and select the application list.

Click the application details menu of the application spring-cloud-a, and you can see that all traffic requests go to the unmarked version of the spring-cloud-a application, that is, the stable version.

Click the Add button in the label routing at the bottom of the page to set grayscale rules for the gray version of the spring-cloud-a application.

To initiate traffic calls, we use zuul-slb to initiate traffic calls respectively, and check the grayscale situation.

We check the message consumption through the logs of spring-cloud-a and spring-cloud-a-gray. It can be seen that the grayscale function of the message has taken effect. The spring-cloud-a-gray environment will only consume messages with gray labels, and the spring-cloud-a environment will only consume unlabeled traffic produced. information.

In the screenshot, we can see that the log output by the spring-cloud-a-gray environment topic: TEST_MQ, producer: Cgray [10.25.0.102] , invoke result: Agray[10.25.0.101] -> Bgray[10.25.0.25] -> Cgray[10.25.0.102], spring-cloud-a-gray will only consume messages produced by Cgray, and the Spring Cloud call initiated in the process of consuming messages results in Agray[10.25.0.101] -> Bgray[10.25.0.25] -> Cgray[10.25.0.102], that is, closed loop in grayscale environment.

In the spring-cloud-a environment, the output log is topic:TEST_MQ,producer:C[10.25.0.157],invoke result:A[10.25.0.100] -> B[10.25.0.152] -> C[10.25.0.157 ], only the messages produced by the baseline environment of C will be consumed, and the Spring Cloud calls initiated in this process are also closed-loop in the baseline environment.

Step 5: Adjust the label filtering rules of the message and verify

Considering the actual scenario, the owners of the spring-cloud-c application and the spring-cloud-a application may not be the same person, and they may not be able to perform grayscale publishing and synchronization operations at the same time. Therefore, in the grayscale of the message , the default behavior for unmarked environments is to consume all messages. In this way, when spring-cloud-c is publishing in grayscale, there is no need to force the spring-cloud-a application to be released in grayscale at the same time, and use the same environment label.

When spring-cloud-a consumes, the choice of the behavior of the unmarked environment is given to the owner of spring-cloud-a. If you need to realize the unmarked environment without consuming the messages produced by c-gray, you only need to You can configure it on the console, and it will take effect in real time after the configuration.

1. Adjust the filtering rules of the unmarked environment of spring-cloud-a. For example, here we want to select the unmarked environment to no longer consume the messages produced by the gray environment, just select gray in the "tags ignored by the unmarked environment", and then click OK.

2. After adjusting the rules, the rules can take effect dynamically without restarting. We directly check the logs of spring-cloud-a to verify that the rule adjustment takes effect.

From this log, we can see that the baseline environment can consume messages produced by gray and baseline environments at the same time, and Spring Cloud calls generated when consuming messages from the corresponding environment are routed to gray and baseline environments respectively.

Operation summary

1. The whole process of full-link message grayscale does not need to modify any code and configuration.

2. Currently only RocketMQ is supported, and the client version needs to be a version after 4.5.0. The RocketMQ server side needs to support SQL92 rule filtering, that is, open-source RocketMQ needs to be configured with enablePropertyFilter=true, and Alibaba Cloud RocketMQ needs to use the platinum version.

3. After enabling message grayscale, MSE Agent will modify the group of message consumers. For example, if the original consumer group is group1 and the environment label is gray, the group will be modified to group1_gray. If Alibaba Cloud RocketMQ is used, it needs to be created in advance Good modified group.

4. After the message grayscale is turned on and off, the application needs to be restarted to take effect; the function of modifying the label ignored by the unmarked environment can take effect dynamically without restarting.

Related Links

[1] MSE Microservice Governance Professional Edition:

https://help.aliyun.com/document_detail/333529.html

[2] Kubernetes cluster:

https://help.aliyun.com/document_detail/86488.html

[3] Container Service Console:
https://cs.console.aliyun.com/

[4] MSE Governance Center Console
https://mse.console.aliyun.com/#/msc/home

Click here to go to the MSE official website for more details!