Introduction to MSE microservice engine will launch the service governance professional version, providing out-of-the-box and complete professional microservice governance solutions to help companies better realize their microservice governance capabilities. If your system can also quickly have complete full-link grayscale capabilities as described in this article, and further microservice governance practices based on this capability, not only can you save objective manpower and costs, but also enable your company The exploration in the field of microservices is more confident.
Author: Shi Mian
review & proofreading__ to:
Editing & Typesetting: Wen Yan
On Double 11 this year, cloud-native middleware completed the trinity of open source, self-research, and commercialization, and was fully upgraded to middleware cloud products. MSE microservice governance has supported the traffic peak of Alibaba Group’s core business on Double 11 through Dubbo3.0. Up to now, 50% of the users in the group are accustomed to using MSE microservices to manage HSF and Dubbo3.0 applications. Today, let’s talk about MSE services in detail. Manage the full-link gray-scale capability in the professional version, and some scenarios of its large-scale practice in production.
background
Under the microservice architecture, there are some demand developments that involve changes to multiple microservices on the microservice invocation link at the same time. It is necessary to better control the risk and explosion radius of the new version of the service through the gray release method. Usually each microservice will have a grayscale environment or group to receive grayscale traffic. We hope that the traffic entering the upstream grayscale environment can also enter the downstream grayscale environment to ensure that 1 request is always delivered in the grayscale environment , Even if there are some microservices on this call link that do not have a grayscale environment, these applications can still return to the grayscale environment when requesting downstream. With the full-link grayscale capability provided by MSE, the above capabilities can be easily achieved without modifying any of your business codes.
MSE microservice management full-link grayscale features
As a fist function in the professional version of MSE service management, full-link grayscale has the following six characteristics
- can introduce refined traffic through customized rules
In addition to simply introducing traffic in proportion, we also support the introduction of Spring Cloud and Dubbo traffic according to rules. Spring Cloud traffic can be introduced according to the requested cookie, header, param parameters or random percentages, and Dubbo traffic can be introduced according to services, methods, and parameters. To introduce.
- Full link isolation traffic lane
1) "Dyeing" the required traffic by setting the traffic rules, and the "dyeing" traffic will be routed to the gray machine.
2) The gray-scale traffic carries the gray-scale mark to be transmitted downstream, forming a gray-scale exclusive environment flow lane. The application in the non-gray-scale environment will default to the unmarked baseline environment.
- End-to-end stable baseline environment
Unmarked applications belong to the stable version of the baseline, that is, a stable online environment. When we will release the corresponding gray version code, then we can configure rules to introduce specific online traffic to control the risk of gray code.
- One-key dynamic flow cut
After the flow rules are customized, one-click stop and start, add, delete, modify, and check can be performed according to requirements, and they will take effect in real time. Gray-scale drainage is more convenient.
- Low-cost access, based on Java Agent technology, no need to modify a line of business code
MSE's microservice governance capabilities are implemented based on the enhanced technology of Java Agent bytecode, which seamlessly supports all Spring Cloud and Dubbo versions on the market for the past 5 years. Users can use it without changing a line of code and without changing the existing business architecture. , Can go up and down at any time, without binding. Just open the professional version of MSE microservice management, online configuration, and real-time effect.
- lossless online and offline capabilities, making the release more silky
After the application opens the MSE microservice management, it has the ability to go online and offline without loss. The release, rollback, expansion, shrinkage and other scenarios under high traffic can ensure that the traffic is lossless.
mass production practice
This article mainly introduces the production and implementation scenarios of several commonly used full-link gray-scale solutions summarized and abstracted in the process of supporting large customers by MSE microservice governance.
Scenario 1: Automatically dye the traffic passing through the machine to achieve full-link grayscale
- After entering the node with tag, subsequent calls will give priority to the node with the same tag, that is, "coloring" the traffic passing through the tag node.
- If no node with the same tag is found on the call link with tag, it will fallback to the node without tag.
- The call link with tag passes through the node without tag. If the link calls the node with tag subsequently, the mode of tag call will be restored.
Scenario 2: Realize full link grayscale by bringing a specific header to the traffic
The client adds the identification of the specified environment to the request, and the access layer forwards it to the gateway representing the corresponding environment according to the representation. The gateway of the corresponding environment calls the identified project isolation environment through the isolation plug-in, requesting to close the loop in the business project isolation environment.
Scenario 3: Perform full link grayscale through custom routing rules
By adding the specified header to the grayscale request, and the entire call link will transmit the header transparently, you only need to configure the header-related routing rules in the corresponding application, and the grayscale request with the specified header enters the grayscale machine , You can achieve full-link traffic grayscale on demand.
Full Link Grayscale Practice
How can we quickly obtain the above-mentioned full-link grayscale capability of the same model? Below I will take you from 0 to 1 to quickly build our full-link grayscale capability.
We assume that the architecture of the application consists of Ingress-nginx and the back-end microservice architecture (Spring Cloud). The back-end call link has 3 hops, shopping cart (a), transaction center (b), inventory center (c), and they Service discovery is done through the Nacos registry, and the client accesses the back-end service through the client or H5 page.
Prerequisites
Install the Ingress-nginx component
Access to the container service console, open the application directory, search ack-ingress-nginx
, select the namespace kube-system
, click Create, the installation is complete, the kube-system
namespace'll see a Deployment ack-ingress-nginx-default-controller
, the installation was successful.
$ kubectl get deployment -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
ack-ingress-nginx-default-controller 2/2 2 2 18h
Open MSE Microservice Governance Professional Edition
- Click to activate the MSE Microservice Management Professional Edition to use the full-link grayscale capability.
- Visit the Container Service console, open the application directory, search for ack-mse-pilot, and click Create.
- In the MSE service management console, open the K8s cluster list, select the corresponding cluster, the corresponding namespace, and open the microservice management.
Deploy the demo application
Save the following to the ingress-gray.yaml file and executes kubectl apply -f ingress-gray.yaml
to deploy applications, where we want to deploy A, B, C three applications, each application are deployed and a gray scale version of a baseline version.
# A 应用 base 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-a
name: spring-cloud-a
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-a
template:
metadata:
annotations:
msePilotCreateAppName: spring-cloud-a
labels:
app: spring-cloud-a
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-a
ports:
- containerPort: 20001
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20001
initialDelaySeconds: 10
periodSeconds: 30
# A 应用 gray 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-a-new
name: spring-cloud-a-new
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-a-new
strategy:
template:
metadata:
annotations:
alicloud.service.tag: gray
msePilotCreateAppName: spring-cloud-a
labels:
app: spring-cloud-a-new
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
- name: profiler.micro.service.tag.trace.enable
value: "true"
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-a-new
ports:
- containerPort: 20001
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20001
initialDelaySeconds: 10
periodSeconds: 30
# B 应用 base 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-b
name: spring-cloud-b
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-b
strategy:
template:
metadata:
annotations:
msePilotCreateAppName: spring-cloud-b
labels:
app: spring-cloud-b
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-b
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20002
initialDelaySeconds: 10
periodSeconds: 30
# B 应用 gray 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-b-new
name: spring-cloud-b-new
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-b-new
template:
metadata:
annotations:
alicloud.service.tag: gray
msePilotCreateAppName: spring-cloud-b
labels:
app: spring-cloud-b-new
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-b-new
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20002
initialDelaySeconds: 10
periodSeconds: 30
# C 应用 base 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-c
name: spring-cloud-c
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-c
template:
metadata:
annotations:
msePilotCreateAppName: spring-cloud-c
labels:
app: spring-cloud-c
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
imagePullPolicy: Always
name: spring-cloud-c
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20003
initialDelaySeconds: 10
periodSeconds: 30
# C 应用 gray 版本
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: spring-cloud-c-new
name: spring-cloud-c-new
spec:
replicas: 2
selector:
matchLabels:
app: spring-cloud-c-new
template:
metadata:
annotations:
alicloud.service.tag: gray
msePilotCreateAppName: spring-cloud-c
labels:
app: spring-cloud-c-new
spec:
containers:
- env:
- name: LANG
value: C.UTF-8
- name: JAVA_HOME
value: /usr/lib/jvm/java-1.8-openjdk/jre
image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
imagePullPolicy: IfNotPresent
name: spring-cloud-c-new
ports:
- containerPort: 8080
protocol: TCP
resources:
requests:
cpu: 250m
memory: 512Mi
livenessProbe:
tcpSocket:
port: 20003
initialDelaySeconds: 10
periodSeconds: 30
# Nacos Server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nacos-server
name: nacos-server
spec:
replicas: 1
selector:
matchLabels:
app: nacos-server
template:
metadata:
labels:
app: nacos-server
spec:
containers:
- env:
- name: MODE
value: standalone
image: nacos/nacos-server:latest
imagePullPolicy: Always
name: nacos-server
resources:
requests:
cpu: 250m
memory: 512Mi
dnsPolicy: ClusterFirst
restartPolicy: Always
# Nacos Server Service 配置
---
apiVersion: v1
kind: Service
metadata:
name: nacos-server
spec:
ports:
- port: 8848
protocol: TCP
targetPort: 8848
selector:
app: nacos-server
type: ClusterIP
Hands
Scenario 1: Automatically dye the traffic passing through the machine to achieve full-link grayscale
Sometimes, we can distinguish between the online baseline environment and the grayscale environment through different domain names. The grayscale environment has a separate domain name that can be configured. Assuming that we request the grayscale environment by visiting www.gray.com, visit www.base. com takes the baseline environment.
Call the link Ingress-nginx -> A -> B -> C, where A can be a spring-boot application.
Note: In the portal application A's gray and A's base environment, you need to turn on the A application's transparent transmission switch according to the traffic ratio in the MSE service management console, which means that the function of backward transparent transmission of the label of the current environment is enabled. In this way, after Ingress-nginx routes the gray of A, even if the request does not carry any header, because this switch is turned on, the x-mse-tag:gray header will be automatically added when it is called in the future, and the value of the header is gray comes from the label information of the A application configuration. If the original request contains x-mse-tag:gray, the tag in the original request will take precedence.
For entry application A, configure two k8s services, spring-cloud-a-base corresponds to the base version of A, and spring-cloud-a-gray corresponds to the gray version of A.
apiVersion: v1
kind: Service
metadata:
name: spring-cloud-a-base
spec:
ports:
- name: http
port: 20001
protocol: TCP
targetPort: 20001
selector:
app: spring-cloud-a
---
apiVersion: v1
kind: Service
metadata:
name: spring-cloud-a-gray
spec:
ports:
- name: http
port: 20001
protocol: TCP
targetPort: 20001
selector:
app: spring-cloud-a-new
Configure the Ingress rule of the entrance, visit www.base.com to route to the base version of A application, and visit www.gray.com to route to the gray version of A application.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: spring-cloud-a-base
spec:
rules:
- host: www.base.com
http:
paths:
- backend:
serviceName: spring-cloud-a-base
servicePort: 20001
path: /
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: spring-cloud-a-gray
spec:
rules:
- host: www.gray.com
http:
paths:
- backend:
serviceName: spring-cloud-a-gray
servicePort: 20001
path: /
Result verification
At this point, visit www.base.com to be routed to the baseline environment
curl -H"Host:www.base.com" http://106.14.155.223/a
A[172.18.144.155] -> B[172.18.144.120] -> C[172.18.144.79]
At this point, visit www.gray.com and be routed to the gray environment
curl -H"Host:www.gray.com" http://106.14.155.223/a
Agray[172.18.144.160] -> Bgray[172.18.144.57] -> Cgray[172.18.144.157]
Further, if no entry application A gray environment, access to the base A of the environment, but also need to A -> B enter when the gradation environment, by adding a special header x-mse-tag
achieved, the value of the header Yes Preferred environmental labels, such as gray
.
curl -H"Host:www.base.com" -H"x-mse-tag:gray" http://106.14.155.223/a
A[172.18.144.155] -> Bgray[172.18.144.139] -> Cgray[172.18.144.8]
You can see that in the first jump, you entered the base environment of A, but when A->B, it returned to the grayscale environment.
The advantage of this method of use is that the configuration is simple. You only need to configure the rules at Ingress. When an application needs to be released in grayscale, you only need to deploy the application in the grayscale environment, and the grayscale traffic will naturally enter the grayscale. In the machine, if the verification is okay, the gray-scale image is released to the baseline environment; if there are multiple applications that need to be released in the gray-scale environment for one change, then they can all be added to the gray-scale environment.
Best Practices
- Mark the gray mark for all applications in the gray scale environment, and not mark the application in the baseline environment by default.
- Online normalization of 2% of the traffic into the gray environment
Scenario 2: Realize full link grayscale by bringing a specific header to the traffic
Some clients cannot rewrite the domain name, and hope to access www.demo.com to route to the gray environment by passing in different headers. For example, in the figure below, by adding the x-mse-tag:gray header, you can access the grayscale environment.
This time demo of Ingress rules are as follows, where increased attention nginx.ingress.kubernetes.io/canary
multiple rules related
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: spring-cloud-a-base
spec:
rules:
- host: www.demo.com
http:
paths:
- backend:
serviceName: spring-cloud-a-base
servicePort: 20001
path: /
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: spring-cloud-a-gray
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-mse-tag"
nginx.ingress.kubernetes.io/canary-by-header-value: "gray"
nginx.ingress.kubernetes.io/canary-weight: "0"
spec:
rules:
- host: www.base.com
http:
paths:
- backend:
serviceName: spring-cloud-a-gray
servicePort: 20001
path: /
Result verification
At this point, visit www.demo.com to be routed to the baseline environment
curl -H"Host:www.demo.com" http://106.14.155.223/a
A[172.18.144.155] -> B[172.18.144.56] -> C[172.18.144.156]
How to access the grayscale environment? Only need to add a header in the request x-mse-tag:gray
can.
curl -H"Host:www.demo.com" -H"x-mse-tag:gray" http://106.14.155.223/a
Agray[172.18.144.82] -> Bgray[172.18.144.57] -> Cgray[172.18.144.8]
You can see that Ingress is directly routed to A's gray environment based on this header.
Go further
Ingress can also use more complex route, such as the client has put on a header, want to use existing routing header to achieve, rather than add a header, for example, as shown in the following figure, suppose we want x-user-id
is a request of 100 to enter the grayscale environment.
Only need to add the following 4 rules:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: spring-cloud-a-base
spec:
rules:
- host: www.demo.com
http:
paths:
- backend:
serviceName: spring-cloud-a-base
servicePort: 20001
path: /
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: spring-cloud-a-base-gray
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
nginx.ingress.kubernetes.io/canary-by-header-value: "100"
nginx.ingress.kubernetes.io/canary-weight: "0"
spec:
rules:
- host: www.demo.com
http:
paths:
- backend:
serviceName: spring-cloud-a-gray
servicePort: 20001
path: /
Bring a special header when visiting, and enter the grayscale environment if the conditions are met
curl -H"Host:www.demo.com" -H"x-user-id:100" http://106.14.155.223/a
Agray[172.18.144.93] -> Bgray[172.18.144.24] -> Cgray[172.18.144.25]
Requests that do not meet the conditions enter the baseline environment:
curl -H"Host:www.demo.com" -H"x-user-id:101" http://106.14.155.223/a
A[172.18.144.91] -> B[172.18.144.22] -> C[172.18.144.95]
Compared with scenario one, the advantage is that the domain name of the client remains unchanged and only needs to be distinguished by request.
Scenario 3: Perform full link grayscale through custom routing rules
Sometimes we don’t want automatic transparent transmission and automatic routing, but hope that each application on the upstream and downstream of the microservice call chain can customize the grayscale rules. For example, application B wants to control that only requests that meet the custom rules will be routed to B Apply here, and the C application may want to define a grayscale rule different from that of B. How to configure it at this time, see the following picture for the scene:
Note that it is best to clear the parameters configured in scene one and two.
First step, Application A at the inlet (inlet preferably all applications have increased the environmental variables, including gray and base) to increase an environment variable: alicloud.service.header=x-user-id
, x-user-id
is transparently transmitted header, it The function is to identify the header and do automatic transparent transmission.
Note that x-mse-tag is not used here, it is a default header of the system and has special logic.
The second step is to configure label routing rules in the MSE console at the application B in the middle
The third step is to configure routing rules at Ingress. For this step, refer to scenario two and use the following configuration:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: spring-cloud-a-base
spec:
rules:
- host: www.base.com
http:
paths:
- backend:
serviceName: spring-cloud-a-base
servicePort: 20001
path: /
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: 'true'
nginx.ingress.kubernetes.io/canary-by-header: x-user-id
nginx.ingress.kubernetes.io/canary-by-header-value: '100'
nginx.ingress.kubernetes.io/canary-weight: '0'
name: spring-cloud-a-gray
spec:
rules:
- host: www.base.com
http:
paths:
- backend:
serviceName: spring-cloud-a-gray
servicePort: 20001
path: /
Result verification
Test verification, visit the gray-scale environment, bring the header that meets the conditions, and route to the gray-scale environment of B.
curl 120.77.215.62/a -H "Host: www.base.com" -H "x-user-id: 100"
Agray[192.168.86.42] -> Bgray[192.168.74.4] -> C[192.168.86.33]
Visit the grayscale environment, bring the header that does not meet the conditions, and route to the base environment of B.
curl 120.77.215.62/a -H "Host: www.base.com" -H "x-user-id: 101"
A[192.168.86.35] -> B[192.168.73.249] -> C[192.168.86.33]
Remove the Ingress Canary configuration, access the base A service (the entry application of the baseline environment needs to add the alicloud.service.header environment variable), bring the header that meets the conditions, and route it to the gray environment of B.
curl 120.77.215.62/a -H "Host: www.base.com" -H "x-user-id: 100"
A[192.168.86.35] -> Bgray[192.168.74.4] -> C[192.168.86.33]
Visit the base environment, bring the header that does not meet the conditions, and route to B's base environment.
curl 120.77.215.62/a -H "Host: www.base.com" -H "x-user-id: 101"
A[192.168.86.35] -> B[192.168.73.249] -> C[192.168.86.33]
summary
The full-link gray-scale capability, which is very technically difficult, can be quickly practiced in 20 minutes. The full-link grayscale is actually not that difficult!
Based on the full-link gray-scale capability of MSE service governance, we can quickly implement enterprise-level full-link gray-scale capabilities. The above three scenarios are standard scenarios for large-scale implementation in our production practice. Of course, we can based on MSE service governance. The ability to customize and adapt according to its own business; even in the context of multiple traffic sources, it can achieve precise drainage according to business customization.
At the same time, the observability capability of the professional version of MSE Service Management makes the gray-level effectiveness measurable.
- gray-scale flow rate second-level monitoring
specification release process
In daily releases, we often have the following wrong ideas:
- The content of this change is relatively small, and the online requirements are relatively urgent, so you don't need to test it and release it directly.
- The release does not need to go through the grayscale process, just quickly release and go online.
- Gray-scale publishing is useless, it is just a process. After publishing, you can publish directly online without waiting for observation.
- Although grayscale publishing is very important, it is difficult to build a grayscale environment, and the priority of time-consuming and labor-intensive is not high.
These ideas may make us make a wrong release. Many failures are directly or indirectly caused by the release. Therefore, improving the quality of release and reducing the occurrence of errors is a key link in effectively reducing online failures. To achieve safe release, we need to standardize the release process.
tail
With the popularity of microservices, more and more companies are using microservice frameworks. Microservices provide better fault tolerance due to their high cohesion, low coupling and other characteristics. They are also more adaptable to rapid business iterations. Brings a lot of convenience. However, with the development of business, the splitting of microservices has become more and more complicated, and the governance of microservices has become a more troublesome issue.
Taking the full link gray level only, in order to ensure the verification of the functional correctness of the new version of the application before it is launched, it is necessary to take into account the efficiency of application release. If the scale of our application is small, we can directly ensure the release by maintaining multiple sets of environments. The correctness. But when our business develops to a large and complex level, suppose our system is composed of 100 microservices. Even if each service in the test/gray-scale environment occupies 1 to 2 pods, we need to face huge challenges in so many environments. Cost and efficiency challenges brought by the operation and maintenance environment.
Is there a simpler and more efficient way to solve the problem of microservice governance?
The MSE microservice engine will launch the service governance professional version, providing out-of-the-box and complete professional microservice governance solutions to help companies better realize their microservice governance capabilities. If your system can also quickly have complete full-link grayscale capabilities as described in this article, and further microservice governance practices based on this capability, not only can you save objective manpower and costs, but also enable your company The exploration in the field of microservices is more confident.
Copyright Notice: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own the copyright, and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用
。你还可以使用@
来通知其他用户。