This is done in full-link grayscale, and new requirements can be iteratively launched, and you can rest assured

Author: Pan Shengwei (Ten Mian)

summary

​Full-link grayscale is one of the core functions of microservices, and it has always been a must-have for cloud customers in the process of deepening microservices. Because of the many technologies and scenarios involved in the full-link grayscale, if enterprises on the cloud implement them one by one, they will need to spend a lot of labor costs to expand and maintain them.

Technical fields involved

• ​RPC: Routing between microservices​
• • Java system involves Spring Cloud, Apache Dubbo mainstream microservice framework, and multi-language involves Service Mesh​
• • Device-cloud interconnection scenarios, online traffic DEBUG. The local service is registered to the registration center through the springboard. It is hoped that the online traffic will be routed to the local service instance after meeting the routing rules, and then DEBUG will be performed. The traffic that does not meet the routing rules will be routed to the online instance​
• ​MQ​
• • ​In the full-link stress test scenario, the stress test traffic sends messages to the shadow topic, and the stress test traffic only subscribes to the shadow topic​
• • ​In traffic isolation/full-link grayscale scenarios, using the same topic, online traffic subscribes to online messages, and isolated traffic only subscribes to grayscale messages​
• ​Database​
• • ​In the full-link stress test scenario, the stress test traffic data is stored in the shadow table​
• • ​In the high-availability switching scenario, database operations are prohibited; in unitized traffic, there is no unit mark, and database operations are prohibited​
• ​Redis​
• • ​In the full-link stress test scenario, the stress test traffic cache is stored on the shadow KEY​
• • ​In the scenario of high-availability switching, the cache operation is prohibited; in the case of unitization, the traffic has no unit mark, and the cache operation is prohibited​
• ​Distributed Task Scheduling​
• • ​For task scheduling, tasks submitted by the grayscale environment are scheduled to be executed on the machine in the grayscale environment​
• ​front end​
• • ​Inconsistent page information seen by different customers​
• ​Observability​
• • ​Monitor traffic flow through observability and view traffic escaping​

MSE full link grayscale solution

​Currently, MSE Service Governance Professional Edition provides a complete productized full-link grayscale solution, covering most scenarios such as RPC, MQ, and observability. As long as your architecture is based on the Spring Cloud or Dubbo framework, your application does not need to be upgraded or a single line of code changes, you can play the enterprise-level full-link grayscale function.​

• ​Full-link isolation traffic lanes​

​1) By setting traffic rules to 'dye' the required traffic, the 'dye' traffic will be routed to the grayscale machine.​

​2) The grayscale traffic carries the grayscale mark and is passed downstream to form a grayscale-exclusive environment traffic lane. Applications without a grayscale environment will select the unmarked baseline environment by default.​

• ​End-to-end stable baseline environment​

​Unmarked applications belong to the baseline stable version of the application, that is, a stable online environment. When we will release the corresponding grayscale version code, we can then configure rules to direct specific online traffic to control the risk of grayscale code.​

• ​One-click dynamic flow cutoff​

​After the traffic rules are customized, you can stop and start them with one click, add, delete, modify and check according to your needs, and take effect in real time. Grayscale drainage is more convenient.​

• ​Observability​

​Single-application observability at the swimlane level

​At the same time, it has the observability of full-link applications, and you can observe whether there is traffic escaping from a global perspective. Gray is not gray, it is clear at a glance.

• ​Low-cost access, no need to modify a line of business code based on Java Agent technology​

​MSE's microservice governance capability is based on the enhanced technology of Java Agent bytecode, which seamlessly supports all Spring Cloud and Dubbo versions on the market in the past 5 years. Users can use it without changing a line of code, and do not need to change the existing business. Architecture, up and down at any time, no binding. Just open the MSE Microservice Governance Professional Edition, configure it online, and take effect in real time.​

• ​With lossless online and offline capabilities, making the release more silky​

​After MSE microservice governance is enabled, the application has the ability to go online and offline without loss. In scenarios such as release, rollback, capacity expansion, and capacity reduction under large traffic, the traffic can be guaranteed to be lossless.​

How to use MSE full link grayscale

​Next, we will demonstrate the ability of full-link grayscale. The application architecture we use consists of Ingress-nginx and the back-end microservice architecture (Spring Cloud). The back-end call link has 3 hops, shopping cart (a), Trading center (b), inventory center (c), clients access back-end services through the client or H5 page, and they do service discovery through the Nacos registry.​

Ready to work

Open MSE Microservice Governance Professional Edition

​Log in to the MSE Governance Center console. If you have not yet activated MSE Microservice Governance, follow the prompts to activate the Professional Edition. If you have already activated the basic version of MSE Microservice Governance, please upgrade to the professional version according to the prompts on the right side of the overview page.​

Install the Ingress-nginx component

1. ​Log in to the Container Service console [1] ​​​.​​
2. ​Select > App Catalog in the left navigation bar.
3. ​Enter ​ack-ingress-nginx​ in the application catalog ​ page search box, click ​ icon, and then click Components.​
4. ​On the details ​ page select the component's namespace ​ for -system ​ and click ​ to create. After the installation is complete, the application ack-ingress-nginx-default-controller ​ appears in the namespace kube-system ​, indicating that the installation is successful.​

Deploy the demo application

​Save the following file to ingress-gray-demo-deployment-set.yaml and execute kubectl apply -f ingress-gray-demo-deployment-set.yaml to deploy the application, here we will deploy A, B, C Three applications, each of which deploys a baseline version and a grayscale version.​

# A 应用 base 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-a
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-a
        ports:
        - containerPort: 20001
        livenessProbe:
          tcpSocket:
            port: 20001
          initialDelaySeconds: 10
          periodSeconds: 30

# A 应用 gray 版本
---            
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a-new
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-a-new
  strategy:
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a-new
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-a-new
        ports:
        - containerPort: 20001
        livenessProbe:
          tcpSocket:
            port: 20001
          initialDelaySeconds: 10
          periodSeconds: 30

# B 应用 base 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-b
  strategy:
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-b
      labels:
        app: spring-cloud-b
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-b
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20002
          initialDelaySeconds: 10
          periodSeconds: 30

# B 应用 gray 版本  
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b-new
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-b-new
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-b
      labels:
        app: spring-cloud-b-new
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-b-new
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20002
          initialDelaySeconds: 10
          periodSeconds: 30

# C 应用 base 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-c
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-c
      labels:
        app: spring-cloud-c
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-c
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20003
          initialDelaySeconds: 10
          periodSeconds: 30

# C 应用 gray 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c-new
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-c-new
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-c
      labels:
        app: spring-cloud-c-new
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
        imagePullPolicy: IfNotPresent
        name: spring-cloud-c-new
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20003
          initialDelaySeconds: 10
          periodSeconds: 30

# Nacos Server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nacos-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nacos-server
  template:
    metadata:
      labels:
        app: nacos-server
    spec:
      containers:
      - env:
        - name: MODE
          value: standalone
        image: nacos/nacos-server:latest
        imagePullPolicy: Always
        name: nacos-server
      dnsPolicy: ClusterFirst
      restartPolicy: Always

# Nacos Server Service 配置
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-server
spec:
  ports:
  - port: 8848
    protocol: TCP
    targetPort: 8848
  selector:
    app: nacos-server
  type: ClusterIP

​For the entry application A, configure two k8s services, spring-cloud-a-base corresponds to the base version of A, and spring-cloud-a-gray corresponds to the gray version of A.​

apiVersion: v1
kind: Service
metadata:
  name: spring-cloud-a-base
spec:
  ports:
    - name: http
      port: 20001
      protocol: TCP
      targetPort: 20001
  selector:
    app: spring-cloud-a

---
apiVersion: v1
kind: Service
metadata:
  name: spring-cloud-a-gray
spec:
  ports:
    - name: http
      port: 20001
      protocol: TCP
      targetPort: 20001
  selector:
    app: spring-cloud-a-new

Quickly build full-link grayscale capabilities

• ​A swimlane defines a set of isolated environments for applications of the same version. Only the request traffic that satisfies the flow control routing rules will be routed to the marking application in the corresponding swimlane. An application can belong to multiple swimlanes, and a swimlane can contain multiple applications. There is a many-to-many relationship between applications and swimlanes.​
• ​Swimlane group: A collection of lanes. The role of swimlane groups is mainly to distinguish between different teams or different scenarios.​

​Log in to the MSE Governance Center console [2] ​​​, find Governance Center> Full Link Grayscale

​You can see that we need to implement the capabilities described above, we only need two steps, creating a swimlane group and creating a swimlane​

Create a swimlane group

​Click the Create Swimlane Group button and select the back-end microservice applications involved in our swimlane group. According to the above demo, there are three applications A, B, and C​

​​

Create swimlanes

​On the full-link grayscale ​page, select the same microservice space when creating the swimlane group, and then click at the bottom to create the first split lane ​. It should be noted that applications that add full-link flow control will no longer support functions such as canary publishing and label routing.​

​​

​According to the steps of the product, we need to create a swimlane name, configure the application label, select the label associated with the swimlane, and go to the ACK console to configure the routing rules of the Ingress.​

Create a completed swimlane

​View swimming lanes, there are two modes​

• ​View Mode​

• ​editable mode​

Ingress rules

​Configure the ingress rules of the entry, visit www.base.com ​ to route to the base version of a application, visit www.gray.com ​ to route to the gray version of a application.​

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: spring-cloud-a-base
spec:
  rules:
  - host: www.base.com
    http:
      paths:
      - backend:
          serviceName: spring-cloud-a-base
          servicePort: 20001
        path: /

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: spring-cloud-a-gray
spec:
  rules:
  - host: www.gray.com
    http:
      paths:
      - backend:
          serviceName: spring-cloud-a-gray
          servicePort: 20001
        path: /

Verify that characteristic traffic is routed to the target application

1. ​Result verification​

​Visit www.base.com ​ Route to Baseline Environment

curl -H"Host:www.base.com" http://{ingress-ip}/a
A[172.18.144.155] -> B[172.18.144.120] -> C[172.18.144.79]%

​At this point, visit www.gray.com ​ Route to grayscale environment

curl -H"Host:www.gray.com" http://{ingress-ip}/a
Agray[172.18.144.160] -> Bgray[172.18.144.57] -> Cgray[172.18.144.157]%

2. ​View the traffic monitoring graph of the marking application.​

​Select the target swimlane group on the full link grayscale ​ page. Select the corresponding application in the involved applications, and the corresponding traffic view will appear​

3. ​View all application monitoring graphs.​

​In addition to viewing the monitoring graphs of a single application, we can also view the monitoring graphs of all applications in the swimlane group. By comparing and analyzing the monitoring graphs of all applications, more useful information can be analyzed.​

• ​You can see which applications are called at the same time.​
• ​Analyze the traffic escape problem and determine the escape object.​

Summarize

​The full-link grayscale productization capability of MSE service governance is still evolving. At present, we support MQ, RPC, observable, etc., and will support more scenarios such as XXL-JOB in the future. Realize multilingual microservice governance [3] ​​, use Cloud Toolkit to realize end-cloud interconnection of [4] ​​, realize full-link grayscale based on Ingress-nginx gateway [5] ​​, full link grayscale based on MSE cloud native gateway [6] ​​, full link grayscale based on self-built Spring Cloud Gateway or Zuul gateway [7] ​​, Realize full-link grayscale based on message queue RocketMQ version [8] ​​, Build CI/CD through Jenkins to achieve canary release [9] ​​, the most agile development of microservices Best practices [10] ​​and other complete solutions related to full-link grayscale, with the increase of user scenarios and practices, our solutions will continue to iterate and enrich.​

Typical Case

Caller Technology

​MSE service governance helps our system to quickly realize the full-link grayscale capability in a low-cost and non-intrusive way, which further improves the stability of our system and makes our iterative launch of new requirements more secure.​

​-- Tang Changzheng, Architect of Inline Technology​

​Caller Technology has entered the field of shared charging since 2014, defined and created the industry, and is the earliest shared charging company in the industry. The main business covers self-service rental of power bank, customized shopping mall navigation machine development, advertising display equipment and advertising communication services. Caller Technology has a three-dimensional product line in the industry, including large, medium and small cabinets and desktop models. At present, more than 90% of the cities in the country have achieved business services, with more than 200 million registered users, meeting the needs of users in all scenarios.​

Full link grayscale landing

​The business structure of the incoming call is as follows. The top layer is the user interface such as the mobile terminal. The self-built Nginx gateway is used as the access layer, and the service layer is various services. Spring Cloud and Dubbo are used as the service framework.​

​​

​The architecture of the full-link grayscale landing of Caller Technology is as follows:​

​Configure the configuration of traffic diversion at the Nginx layer, 10% of the traffic enters the grayscale environment, 90% of the traffic enters the unmarked or online formal environment, and then the traffic passing through the grayscale environment will be automatically dyed by the MSE with the color of the corresponding environment , so as to perform full-link grayscale routing to ensure that the traffic is closed in a grayscale environment. If there is no grayscale environment machine, such as the payment center only has online machines, then the traffic will go to the online environment. When our data center has If there are machines in a grayscale environment, then the grayscale traffic will return to the grayscale environment of the data center.​

tail

​MSE's full-link grayscale capabilities continue to expand and iterate with the deepening of customer scenarios. Only products that have been polished by customers will become more lasting and new, and you are welcome to try them out.​

Related Links

[1] Container Service Console

​​https://cs.console.aliyun.com/#/authorize​​

[2] MSE Governance Center Console

​​https://mse.console.aliyun.com/?spm=a2c4g.11186623.2.13.f90a6a60WiEx0N#/auth​​

[3] How to implement multilingual microservice governance on MSE

​​https://help.aliyun.com/document_detail/184289.html​​

[4] Use Cloud Toolkit to realize end-cloud interconnection of microservices

​​https://help.aliyun.com/document_detail/196920.html​​

[5] Realize full link grayscale based on Ingress-nginx gateway

​​https://help.aliyun.com/document_detail/347790.html​​

[6] Realize full link grayscale based on MSE cloud native gateway

​​https://help.aliyun.com/document_detail/359851.html​​

[7] Realize full link grayscale based on self-built Spring Cloud Gateway or Zuul gateway

​​https://help.aliyun.com/document_detail/359858.html​​

[8] Realize full link grayscale based on message queue RocketMQ version

​​https://help.aliyun.com/document_detail/397318.html​​

[9] Build CI/CD through Jenkins to achieve canary release

​​https://help.aliyun.com/document_detail/384436.html​​

[10] Best practices for agile development of microservices​

​​https://help.aliyun.com/document_detail/397319.html​​

​Click “ ​here ​” to learn more