The practice of Alibaba Cloud MSE's full-link grayscale solution based on Apache APISIX

Apache APISIX is an open source cloud-native API gateway. As an API gateway, it has the characteristics of dynamic, real-time, and high performance. It provides load balancing, dynamic upstream, grayscale publishing, service fuse, identity authentication, and observability. traffic management function. You can use Apache APISIX to handle traditional north-south traffic and east-west traffic between services, or it can be used as a K8s Ingress controller. Thanks to the fully dynamic design of APISIX, configuration changes can be made at any time without restarting the service.

Alibaba Cloud's microservice engine MSE provides a very easy-to-use traffic lane capability. It is implemented based on Java Agent bytecode-enhanced technology and seamlessly supports all Spring Cloud and Dubbo versions on the market for nearly 5 years. A code-free way to achieve full-link grayscale and release the new value of APISIX-based microservice architecture.

Introduction to the full-link grayscale solution

Related concepts

• Swimlane: A set of isolated environments defined for applications of the same version. Only the request traffic that satisfies the flow control routing rules will be routed to the marking application in the corresponding swimlane. An application can belong to multiple swimlanes, and a swimlane can contain multiple applications. There is a many-to-many relationship between applications and swimlanes.
• Baseline environment: The unmarked application belongs to the baseline stable version of the application, that is, the stable online environment.
• Traffic fallback: The number of services deployed in the swimlane is not required to be exactly the same as the baseline environment. When there are no other services that depend on the call chain in the swimlane, the traffic needs to be rolled back to the baseline environment, and further routed back when necessary. The swimlane corresponding to the label.
• Lane Group: A collection of lanes. The role of swimlane groups is mainly to distinguish between different teams or different scenarios.

Business scene

The full-link grayscale capability based on traffic lanes is suitable for the following business scenarios:

• Daily development/project/test environment isolation;
• Full link grayscale release;
• High availability with priority routing in the same computer room;
• Full link stress test.

Technical principle

How to quickly implement full-link grayscale in actual business scenarios? Currently, there are two main solutions, one based on physical environment isolation and one based on logical environment isolation.

Physical environment isolation

Physical environment isolation is actually to build real traffic isolation by adding machines.

This solution needs to build a network-isolated and resource-independent environment for the grayscale service, and deploy the grayscale version of the service in it. Due to isolation from the formal environment, other services in the formal environment cannot access the services that require grayscale. Therefore, these online services need to be redundantly deployed in the grayscale environment so that the entire call link can forward traffic normally. In addition, some other dependent middleware components such as the registry also need to be redundantly deployed in the grayscale environment to ensure visibility between microservices and ensure that the obtained node IP addresses only belong to the current network environment.

This solution is generally used for enterprise testing and the construction of pre-development environments. It is not flexible enough for online grayscale publishing and drainage scenarios. In addition, the existence of multiple versions of microservices is very common in the microservice architecture, and it is necessary to maintain multiple sets of grayscale environments by adopting heap machines for these business scenarios. If the number of applications is small, this method can be accepted; if you have too many applications, it will cause excessive operation and maintenance and machine costs, and the costs and costs far exceed the benefits.

Logical environment isolation

Another solution is to build logical environment isolation. We only need to deploy the grayscale version of the service. When the traffic flows on the call link, the grayscale traffic is identified by the passing gateway, each middleware and each microservice, and Dynamically forward to the grayscale version of the corresponding service. As shown below:

The above figure can well show the effect of this scheme. We use different colors to represent the gray-scale traffic of different versions. It can be seen that both the microservice gateway and the microservice itself need to identify the traffic and make dynamic decisions according to the governance rules. . When the service version changes, the forwarding of this call link will also change in real time. Compared with the grayscale environment built by machines, this solution can not only save a lot of machine cost and O&M manpower, but also help developers to perform refined full-link control of online traffic in real time and quickly.

Full-link grayscale product practice based on Apache APISIX

Full-link grayscale is one of the core functions of microservices, and it is also a function that users on the cloud must have in the process of deepening microservices. Because the full-link grayscale involves many technologies and scenarios, if the enterprise implements itself one by one, it needs to spend a lot of labor costs to expand and operate it.

MSE service governance provides a complete productized full-link grayscale solution, covering most scenarios such as RPC, MQ, and observability. As long as the architecture is based on the Spring Cloud or Dubbo framework, the application can realize the enterprise-level full-link grayscale function without upgrading or code changes.

Prerequisites for use

Step 1: Install APISIX related components

1. Install components such as APISIX, apisix-ingress-controller, etcd.

 helm repo add apisix https://charts.apiseven.com
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
kubectl create ns ingress-apisix
helm install apisix apisix/apisix \
  --set gateway.type=LoadBalancer \
  --set ingress-controller.enabled=true \
  --set etcd.persistence.storageClass="alicloud-disk-ssd" \
  --set etcd.persistence.size="20Gi" \
  --namespace ingress-apisix \
  --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix
kubectl get service --namespace ingress-apisix

Stateless APISIX and apisix-ingress-controller applications, and stateful etcd applications can be seen under the ingress-apisix namespace.

2. Install APISIX Dashboard using Helm.

    helm repo add apisix https://charts.apiseven.com
   helm repo update
   helm install apisix-dashboard apisix/apisix-dashboard --namespace
   ingress-apisix

3. After the installation is complete, you can bind an SLB.

4. Access APISIX Dashboard via {slb-ip}:9000 .
   

   Step 2: Start Microservice Governance

   In this step, you need to enable MSE microservice governance, install the MSE service governance component (ack-onepilot), and enable microservice governance for the application. For specific operation information, please refer to the official Alibaba Cloud tutorial .

   Step 3: Deploy the Demo Application

   Deploy three applications A, B, and C in Alibaba Cloud Container Service, each application deploys a base version and a gray version; and deploys a Nacos Server application , which is used to implement service discovery. For details, please refer to this tutorial to complete the application deployment: Deploy the Demo application . After the deployment is complete, you can configure the application through APISIX Dashboard Service for upstream configuration.

   Application Scenario 1: Routing by Domain Name

   In some scenarios, the online baseline environment and the grayscale environment can be distinguished by different domain names. The grayscale environment has a separate domain name that can be configured. Suppose we request the grayscale environment by visiting www.gray.com , and visit www.base.com to take the baseline environment.
   
   Call link Ingress-nginx -> A -> B -> C , where A can be an application of spring-boot .

   Configure APISIX routing rules

   Select the route on the APISIX Dashboard and click Create. In the matching conditions, select /* for the domain name and request path, and select the corresponding upstream. Configure the following routes respectively:

5. When host is www.base.com , the route to the upstream corresponding to the id is 401152455435354748 , namely spring-cloud-a-svc ;

6. When host is www.gray.com , the route to the upstream corresponding to the id 401163331936715388 is spring-cloud-a-gray-svc .
   Then perform base the corresponding routing configuration:

    {
     "uri": "/*",
     "name": "spring-cloud-a",
     "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
     ],
     "host": "www.base.com",
     "upstream_id": "401152455435354748",
     "labels": {
     "API_VERSION": "0.0.1"
     },
     "status": 1
   }

   Perform gray the corresponding routing configuration:

    {
     "uri": "/*",
     "name": "spring-cloud-a-gray",
     "priority": 1,
     "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
     ],
     "host": "www.gray.com",
     "upstream_id": "401163331936715388",
     "labels": {
    "API_VERSION": "0.0.1"
     },
     "status": 1
   }

   Configuring MSE full-link grayscale

   You need to configure the full-link release of MSE. For details, please refer to this tutorial: Configure full-link grayscale .

   Result verification

   Access www.base.com routed to A application's base version:

    curl -H"Host:www.base.com" http://47.97.253.177/a
   A[172.18.144.15] -> B[172.18.144.125] -> C[172.18.144.90]%

   Access www.gray.com routed to A application's gray version:

    curl -H"Host:www.gray.com" http://47.97.253.177/a
   Agray[172.18.144.16] -> Bgray[172.18.144.57] -> Cgray[172.18.144.157]%

   Application Scenario 2: Routing according to the specified request parameters

   Some clients cannot rewrite the domain name, and hope to be able to access www.demo.com to route to the grayscale environment by passing in different parameters. For example, in the figure below, the grayscale environment is accessed through the request parameter env=gray .
   
   Call link Ingress-APISIX -> A -> B -> C , where A can be an application of spring-boot .

   Configure APISIX routing rules

   Select the route on the APISIX Dashboard and click Create. In the matching conditions, create a new advanced matching rule, select the request path /*, and select the corresponding upstream. Configure the following routes respectively:

7. 当host www.demo.com ，请求参数env=gray时， id 401163331936715388 , namely spring-cloud-a-gray-svc ;

8. When host is www.demo.com, the route will match id is the upstream corresponding to 401152455435354748 , namely spring-cloud-a-svc .
   Then perform base the corresponding routing configuration:

    {
     "uri": "/*",
     "name": "spring-cloud-a",
     "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
     ],
     "host": "www.demo.com",
     "upstream_id": "401152455435354748",
     "labels": {
    "API_VERSION": "0.0.1"
     },
     "status": 1
   }

   Perform the routing configuration corresponding to gray , as shown in the following figure:
   

    {
     "uri": "/*",
     "name": "spring-cloud-a-gray",
     "priority": 1,
     "methods": [
    "GET",
    "POST",
    "PUT",
    "DELETE",
    "PATCH",
    "HEAD",
    "OPTIONS",
    "CONNECT",
    "TRACE"
     ],
     "host": "www.demo.com",
     "vars": [
    [
      "arg_env",
      "==",
      "gray"
    ]
     ],
     "upstream_id": "401163331936715388",
     "labels": {
     "API_VERSION": "0.0.1"
     },
     "status": 1
   }

   Configuring MSE full-link grayscale

   The steps for configuring MSE full-link grayscale are the same as those in Scenario 1 .

   Result verification

   At this point, access www.demo.com is routed to the baseline environment:

    curl -H"Host:www.demo.com" http://47.97.253.177/a
   A[172.18.144.15] -> B[172.18.144.125] -> C[172.18.144.90]%

   At this point, when accessing www.demo.com and env=gray , it is routed to the grayscale environment:

    curl -H"Host:www.demo.com" http://47.97.253.177/a?env=gray
   Agray[172.18.144.16] -> Bgray[172.18.144.57] -> Cgray[172.18.144.157]%

   Note: where 47.97.253.177 is the public IP of APISIX.

   Summarize

   Based on the flexible routing capabilities of Apache APISIX and the MSE full-link grayscale capability, enterprise-level full-link grayscale capabilities can be quickly realized.
   APSIX supports routing according to Header, Cookie, Params, domain name and other methods. It only needs to route traffic to different "swim lane" environments on the gateway side according to requirements, and the traffic will automatically close the loop in the "swim lane" of the corresponding label. When there are no other services that depend on the call chain in the call chain, the traffic needs to fall back to the baseline environment, and further route back to the swimlane of the corresponding label when necessary.