Realize full-link grayscale based on Apache APISIX through MSE

Author: ten sleep

What is full link grayscale?

In the microservice architecture, the dependencies between services are intricate, and sometimes a function release depends on multiple services being upgraded and launched at the same time. We hope that the new versions of these services can be verified with small traffic at the same time. This is the unique full-link grayscale scene in the microservice architecture. By building an environment isolation from the gateway to the entire backend service, multiple different versions can be verified. service for grayscale verification.

During the release process, we only need to deploy the grayscale version of the service. When the traffic flows on the call link, the grayscale traffic is identified by the gateways, middleware and microservices it flows through, and dynamically forwarded to the corresponding service. Grayscale version. As shown below:

The above figure can well show the effect of this scheme. We use different colors to represent the gray-scale traffic of different versions. It can be seen that both the microservice gateway and the microservice itself need to identify the traffic and make dynamic decisions according to the governance rules. . When the service version changes, the forwarding of this call link will also change in real time. Compared with the grayscale environment built by machines, this solution can not only save a lot of machine cost and O&M manpower, but also help developers to perform refined full-link control of online traffic in real time and quickly.

So how is the full-link grayscale implemented? Through the above discussion, we need to solve the following problems:

1. Each component and service on the link can be dynamically routed according to the request traffic characteristics

2. All nodes under the service need to be grouped to distinguish versions

3. It is necessary to carry out grayscale identification and version identification for the traffic

4. Need to identify different versions of grayscale traffic

The following will tell you the technical details required to achieve full-link grayscale by introducing the v1alpha1 standard defined by OpenSergo for traffic routing.

Q: What is OpenSergo?

A: OpenSergo is a set of open, general-purpose service governance standards oriented to distributed service architecture and covering the whole-link heterogeneous ecology. It forms a general service governance standard based on industry service governance scenarios and practices. The biggest feature of OpenSergo is that it defines service governance rules with a unified set of configuration/DSL/protocol, oriented to multi-language heterogeneous architecture, and achieves full-link ecological coverage. Whether the language of the microservice is Java, Go, Node.js or other languages, whether it is standard microservice or mesh access, from gateway to microservice, from database to cache, from service registration discovery to configuration, developers can The same set of OpenSergo CRD standard configuration performs unified governance and control for each layer, without paying attention to the differences between frameworks and languages, reducing the complexity of heterogeneous and full-link service governance and control

Q: Why did you introduce OpenSergo to me before learning about full-link grayscale?

A: OpenSergo defines a unified YAML configuration method to implement full-link service governance specifications for distributed architectures. While introducing the specifications and standards, we can understand the implementation of the technical details, and we can also apply new The components are implemented with the OpenSergo standard.

OpenSergo traffic routing v1alpha1 standard

Traffic routing, as the name suggests, is to route traffic with certain attribute characteristics to a specified destination. Traffic routing is an important part of traffic governance. Developers can implement various scenarios based on traffic routing standards, such as grayscale publishing, canary publishing, disaster recovery routing, and label routing.

Full link grayscale example:

Traffic routing rules (v1alpha1) are mainly divided into three parts:

• Workload Label Rule (WorkloadLabelRule): Label a group of workloads with corresponding labels, which can be understood as labeling each upstream of APISIX with corresponding labels
• Traffic Label Rule (TrafficLabelRule): Label traffic with certain attribute characteristics with corresponding labels
• Make matching routes according to the workload label and traffic label, and route the traffic with the specified label to the matching workload

We can give labels different semantics to achieve routing capabilities in various scenarios.

To mark traffic:

Traffic with certain attribute characteristics needs to be marked with corresponding labels.

Suppose now that the internal test user needs to be grayed out to the new home page, the test user uid=12345, and the UID is located in the X-User-Id header:

 apiVersion: traffic.opensergo.io/v1alpha1
kind: TrafficLabelRule
metadata:
  name: my-traffic-label-rule
  labels:
    app: my-app
spec:
  selector:
    app: my-app
  trafficLabel: gray
  match:
  - condition: "=="    # 匹配表达式
    type: header       # 匹配属性类型
    key: 'X-User-Id'   # 参数名
    value: 12345       # 参数值
  - condition: "=="
    value: "/index"
    type: path

Through the above configuration, we can mark the HTTP traffic with the path as /index and the uid header as 12345 and mark it as gray, indicating that the traffic is grayscale traffic.

Label the Workload:

So how to add different labels to service nodes? Driven by today's hot cloud-native technologies, most businesses are actively embarking on a containerization transformation journey. Here, I take the containerized application as an example to introduce how to mark the node of the service Workload in the two scenarios of using Kubernetes Service as service discovery and using the more popular Nacos registry.

In a business system that uses Kubernetes Service as service discovery, the service provider completes service exposure by submitting Service resources to ApiServer, and the service consumer monitors the Endpoint resources associated with the Service resources, and obtains the associated business Pod resources from the Endpoint resources. Read the above Labels data and use it as the metadata information of the node. Therefore, we only need to add labels to the nodes in the Pod template in the business application description resource Deployment.

In a business system that uses Nacos as a service discovery, it is generally necessary for the business to decide the marking method according to the microservice framework it uses. If the Java application uses the Spring Cloud microservice development framework, we can add the corresponding environment variables to the business container to complete the operation of adding tags. For example, if we want to add a version grayscale label to the node, then add traffic.opensergo.io/label: gray to the business container, so that the framework will add a gray label to the node when it registers with Nacos.

For some complex workload marking scenarios (such as database instances, cache instance labels), we can use WorkloadLabelRule CRD for marking. Example:

 apiVersion: traffic.opensergo.io/v1alpha1
kind: WorkloadLabelRule
metadata:
  name: gray-sts-label-rule
spec:
  workloadLabels: ['gray']
  selector:
    app: my-app-gray

Flow Color:

How do each component on the request link identify different grayscale traffic? The answer is traffic coloring, adding different grayscale marks to the request traffic to facilitate differentiation. We can color the traffic at the source of the request, and the front-end will mark the traffic according to the user information or platform information when initiating the request. If the front end cannot do this, we can also dynamically add traffic identifiers on the microservice gateway to requests that match specific routing rules. In addition, when the traffic flows through the grayscale nodes in the link, if the request information does not contain the grayscale logo, it needs to be automatically colored, and then the traffic can preferentially access the grayscale version of the service in the subsequent flow process.

At present, OpenSergo v1alphal1 does not define the standard for traffic coloring in detail. The traffic coloring standard can be designed by subsequent community discussions. Apache APISIX will also be adapted to implement the OpenSergo standard. Developers can use the same set of OpenSergo CRD standard configuration to carry out unified governance and control of the traffic gateway layer, which can release the new value of the Apache APSIX-based microservice architecture.

Full-link grayscale is one of the core functions of microservices, and it is also a function that users on the cloud must have in the process of deepening microservices. Because the full-link grayscale involves many technologies and scenarios, if the enterprise implements itself one by one, it needs to spend a lot of labor costs to expand and operate it.

Product practice based on Apache APISIX full-link grayscale solution

After introducing the technology, let's introduce the product practice of full-link grayscale based on Apache APISIX on Alibaba Cloud.

Preconditions

Step 1: Install the Ingress-APISIX component

The APISIX architecture is shown in the figure below, and we need to install APISIX.

1. Install components such as apisix, apisix-ingress-controller, etcd

 helm repo add apisix https://charts.apiseven.com
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
kubectl create ns ingress-apisix
helm install apisix apisix/apisix \
  --set gateway.type=LoadBalancer \
  --set ingress-controller.enabled=true \
  --set etcd.persistence.storageClass="alicloud-disk-ssd" \
  --set etcd.persistence.size="20Gi" \
  --namespace ingress-apisix \
  --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix
kubectl get service --namespace ingress-apisix

See the stateless apisix, apisix-ingress-controller applications, and stateful etcd applications under the ingress-apisix namespace.

2. Install APISIX Admin

 helm repo add apisix https://charts.apiseven.com
helm repo update
helm install apisix-dashboard apisix/apisix-dashboard --namespace ingress-apisix

After the installation is complete, you can bind an SLB

Access APISIX console via {slb-ip}:9000 (default password admin/admin)

Step 2: Start Microservice Governance

In this step, you need to enable MSE microservice governance, install the MSE service governance component (ack-onepilot), and enable microservice governance for the application. For specific operation information, please refer to the official Alibaba Cloud tutorial:

https://help.aliyun.com/product/123350.html

Step 3: Deploy the Demo Application

Deploy three applications A, B, and C in Alibaba Cloud Container Service, each of which deploys a base version and a gray version; and deploys a Nacos Server application for service discovery. For details, please refer to this tutorial to complete the application deployment: Deploy the Demo application. After the deployment is complete, you can configure the application for upstream configuration through APISIX Dashboard.

Application scenario: Routing according to the specified request parameters to achieve full-link grayscale

Some clients cannot rewrite the domain name, and hope to access www.demo.com to route to the grayscale environment by passing in different parameters. For example, in the figure below, the grayscale environment is accessed through the request parameter env=gray.

Call the link Ingress-APISIX -> A -> B -> C , where A can be a spring-boot application.

Configure APISIX routing rules

Select the route on the APISIX Dashboard and click Create. In the matching conditions, create a new advanced matching rule, select the request path /*, and select the corresponding upstream. Configure the following routes respectively:

• When the host is www.demo.com and the request parameter env=gray, the route preferentially matches the upstream corresponding to the id 401163331936715388, namely spring-cloud-a-gray-svc;
• When the host is www.demo.com, the route will match the upstream corresponding to the id 401152455435354748, namely spring-cloud-a-svc.

Then perform the routing configuration corresponding to the base:

 {
  "uri": "/*",
  "name": "spring-cloud-a",
  "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
  ],
  "host": "www.demo.com",
  "upstream_id": "401152455435354748",
  "labels": {
    "API_VERSION": "0.0.1"
  },
  "status": 1
}

Perform the routing configuration corresponding to the gray, as shown in the following figure:

 {
  "uri": "/*",
  "name": "spring-cloud-a-gray",
  "priority": 1,
  "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
  ],
  "host": "www.demo.com",
  "vars": [
    [
      "arg_env",
      "==",
      "gray"
    ]
  ],
  "upstream_id": "401163331936715388",
  "labels": {
    "API_VERSION": "0.0.1"
  },
  "status": 1
}

Configuring MSE full-link grayscale

You need to configure the full-link release of MSE. For details, please refer to this tutorial: Configure full-link grayscale.

Result verification

At this point, visit www.demo.com to route to the baseline environment

 curl -H"Host:www.demo.com" http://47.97.253.177/a
A[172.18.144.15] -> B[172.18.144.125] -> C[172.18.144.90]%

At this point, when visiting www.demo.com and env=gray, it is routed to the grayscale environment

 curl -H"Host:www.demo.com" http://47.97.253.177/a?env=gray
Agray[172.18.144.16] -> Bgray[172.18.144.57] -> Cgray[172.18.144.157]%

Note: 47.97.253.177 is the public IP of APISIX

Summarize

At present, the full-link grayscale capability of MSE service governance has supported cloud native gateways, ALB, APISIX, Apache Dubbo, Spring Cloud, RocketMQ and databases.

Based on the flexible routing capabilities of Apache APISIX and the MSE full-link grayscale capability, enterprise-level full-link grayscale capabilities can be quickly realized. APSIX supports routing according to Header, Cookie, Params, domain name and other methods. It only needs to route traffic to different "swim lane" environments on the gateway side according to requirements, and the traffic will automatically close the loop in the "swim lane" of the corresponding label. When there are no other services that depend on the call chain in the call chain, the traffic needs to fall back to the baseline environment, and further route back to the swimlane of the corresponding label when necessary.

Service governance is the only way to go through the transformation of microservices to a certain stage. During this process, we continue to have new problems.

• In addition to full-link grayscale, is there any other capability for service governance?
• Is there a standard definition of service governance capability, and what does service governance capability include?
• In multilingual scenarios, are there any best practices or standards for full links?
• How can heterogeneous microservices be managed uniformly?

In the process of exploring service governance, when we are connecting with other microservices, we find that the trouble caused by different governance systems is huge, and the cost of connecting two or even multiple governance systems is also huge. For this we propose the OpenSergo project. What OpenSergo wants to solve is the fragmentation and inability to communicate with different frameworks and languages in the concept of microservice governance.

The OpenSergo community is also working with the Apache APISIX community for further cooperation. The community will discuss and define a unified service governance standard. The current community is also working with bilibili, ByteDance and other companies to jointly build standards. Interested developers, communities and companies are also welcome to join in the joint construction of OpenSergo service governance standards. Welcome to join the OpenSergo community exchange group (Dingding group) for discussion: 34826335

Related Links

• OpenSergo: https://opensergo.io/en-us
• MSE Microservice Engine: https://www.aliyun.com/product/aliware/mse

10% discount for the first purchase of MSE Registration and Configuration Center Professional Edition, 10% discount for MSE Cloud Native Gateway Prepaid Full Specifications. Click here to enjoy the discount~