Cainiao CPaaS Platform Microservice Governance Practice

Author: Zhuda Zhao Yu

background

CPaaS (cainiao platform as a service) is an enterprise-level DevOps PaaS platform based on public cloud and combined with advanced cloud-native physiology. CPaaS mainly supports the scenarios currently: Cainiao ecological cloud R&D operation and maintenance, Cainiao public The ability of cloud SaaS is revealed, and the commercial output of Cainiao is supported, and it is deployed to the customer's public cloud and private cloud environment.

In the process of serving many ecological companies in Cainiao and some commercialized products, he has accumulated some valuable experience in the process of in-depth customer business scenarios and solving business development and deployment pain points. Here, we mainly introduce the implementation process of standardizing the R&D process on the cloud, improving the R&D efficiency for the construction of the environmental governance (the project environment on the cloud) and reducing the risk of online version release and the construction of the grayscale platform.

Target

1. Through the project environment, provide traffic isolation and rapid joint debugging capabilities for multi-branch parallel development scenarios.

2. The production environment realizes the grayscale release of services (canary release) to reduce the risk of change.

3. Microservice applications have the ability to go online and offline gracefully to avoid service call errors caused by the start and stop process.

research stage

Microservice Traffic Control

We first investigated the open source self-built solution. During our research, we found that the cost of developing and maintaining open source SDK solutions is very high. It is necessary to have a good understanding of the microservice frameworks such as Spring Cloud and Dubbo, as well as the message middleware such as RockeMQ, in order to accurately find the enhancement points of each framework for customized development.

In addition, the version of the microservice framework used by the business side is also very wide, and maintaining the adaptation of these different versions of the microservice framework also requires a lot of effort.

The most important point is that, using the open source SDK self-built solution, colleagues in business development need to perceive the existence of the SDK in the process of application development, deployment and operation and maintenance, which is very intrusive to development, construction, and operation and maintenance. It is difficult to promote.

Later, we also found colleagues in charge of middleware in Alibaba Group for support. We learned that the middleware team has launched a public cloud-oriented microservice governance product MSE, so we conducted research.

As a microservice governance product of the public cloud, MSE has the capabilities of cloud service management and control, microservice testing, label routing, outlier removal, and graceful online and offline. Non-intrusive, more suitable for PaaS platform to increase the expansion of business applications.

Figure 1.1

After several simultaneous phone calls and meetings with the MSE team, I gradually gained some functional understanding of the MSE product. Among them, in microservice governance, we have implemented some of the following MSE capabilities based on actual business needs.

Figure 1.2

landing scene

Project environment

In multi-branch development scenarios, we usually need to deploy multiple branches at the same time. However, after multiple branches are deployed at the same time, how to separate the development self-test traffic from the test traffic in the daily environment, and how to allow each branch to have its own independent traffic, are all problems that need to be solved.

• Traffic isolation

After investigating and verifying the label routing capability of the MSE, the realization idea is to isolate the traffic through the label routing capability.

Different branches of the same application use different deployments to manage versions and container labels. In Figure 2.1, the core application project environment c1 and the project environment c2 are deployed separately using containers independent of the daily environment, and their route labels are joint1 and joint2. By carrying traffic marks to traffic, the control of project environmental traffic is completed.

The access layer application implements traffic routing through K8s-Ingress, and only needs to carry the label of x-mse-tag in the request header of the request traffic, and then the traffic can be routed to the ingress layer application of the corresponding label. The ingress application sets up the development of label delivery, which can label the traffic flowing through this container and deliver it to the on-premises service. Repeatedly, the traffic is closed in the whole calling link.

Figure 2.1

• service test

In the R&D process, in addition to the need to isolate the mutual interference between branches, it is also necessary to solve the efficiency problem of service testing from the perspective of business-side R&D. The MSE platform provides a microservice testing platform, which can quickly help developers implement service self-testing, and we integrate it into our own PaaS platform through integration, saving the pain of rebuilding ourselves.

The test platform supports service testing according to the dimension of service provider IP, which just fits with traffic isolation. It can initiate service testing on the application container of the project environment of its own concern to complete the service self-test.

Figure 2.2

Grayscale environment

The purpose of the grayscale environment is to reduce the risk of online application version release and reduce the explosion radius of problems. The same can be done based on the label routing function provided by the MSE to complete the realization of the traffic grayscale.

Figure 2.3

If you want to achieve grayscale traffic, you need to implement traffic routing for all ingress of the traffic. Usually, the traffic entrances we perceive include: HTTP access layer, RPC downstream calls, MQ service consumption, and Task task scheduling. Currently, within the range supported by MSE's grayscale capabilities, microservice cloud gateway, canary release, and MQ grayscale can all be implemented in combination.

Of course, here we only implement the grayscale capability of HTTP access layer + RPC. For historical reasons, another access layer (MSFE of MSHA, which is essentially a tengine) is used in the access layer to realize the access layer. grayscale. However, this does not affect the serial connection of traffic with the grayscale capability of MSE, which is due to the good compatibility of MSE products with other products. We only need to set the label when the traffic passes through this container on the application of the entry layer to complete the transfer of grayscale traffic.

Figure 2.4

In the actual business grayscale scenarios, we summarize four common grayscale scenarios, which are all completed and implemented through MSE.

Figure 2.5

future plan

After using MSE's cloud products, for the PaaS platform layer, the construction of many repetitive functions is avoided. The actual implementation of our business side is far more than the scenarios listed above, such as: graceful service shutdown, registry and other capabilities, all of which solve the difficult problem of microservice governance on the business side.

After realizing the capability development of the project environment and grayscale release, we will focus on the capabilities of service outlier removal, application service list disclosure, service authentication, local joint debugging and deployment, etc., in order to reduce the service operation on the business side. Strengthen cooperation with the MSE team in terms of dimension cost, microservice observability, and service availability to help the business side solve the pain points in microservice governance.

10% discount for the first purchase of MSE Registration and Configuration Center Professional Edition, 10% discount for MSE Cloud Native Gateway Prepaid Full Specifications. Click here to take advantage of the discount!