Getting Started and Beyond｜Guide to Selection of Load Balancing for Cloud Native Applications

Author

Ran Xin, TCM product manager of Tencent Cloud Service Grid, is now responsible for the planning and design of product features such as cloud native traffic access gateway and application communication observability.

Liu Xu, senior engineer of Tencent Cloud, focuses on the native field of container cloud. He has many years of experience in large-scale Kubernetes cluster management and microservice governance. He is currently responsible for the design and research and development of Tencent Cloud service grid TCM data plane product architecture.

introduction

Application ingress traffic management has always been one of the focuses of development, operation and maintenance. With the development and change of computing resources, network environment, and application architecture deployed by the business, the development of access layer traffic management solutions can be roughly divided into traditional architectures and cloud-native containers. Two stages of transformation. In order to meet the efficiency and demands of application delivery, different access layer solutions have emerged at each stage, from the initial simple load balancing, to the later reverse proxy such as HAProxy and Nginx, and then to the current containerized environment. Class Kubernetes Ingress Controller. What are the characteristics of each stage of development? What are the challenges? What are the solutions?

stage	Application deployment resource granularity	Application architecture	Application access addressing
Traditional architecture	Physical/virtual machine (low resource utilization)	Single or simple split module	Based on relatively fixed IP address management
Cloud native containerization	Container (high resource utilization)	Servicing	Container IP changes dynamically and is updated through dynamic service registration

In the traditional architecture stage, the business is a single application with a three-tier architecture; deployed on physical machines/virtual machines; the network environment is based on IP address management, which is relatively fixed and basically unchanged; the speed of business update iterations is slow, and the access layer is mainly The requirement is to have 4-layer and 7-layer load balancing capabilities, which can be supported by a traditional load balancer. With the evolution of application architecture (applications have been split into certain modules) and the improvement of iteration efficiency, some more complex access layer requirements have emerged: routing according to traffic content characteristics, gray-scale publishing, current limiting, authentication, etc., generally through A layer of network proxy (eg Nginx) support is added after the load balancer. The network proxy Nginx has more 7-layer traffic processing capabilities, and can be extended through the Lua of the OpenResty community to extend the above-mentioned content routing, grayscale publishing, authorization flow and other advanced Features.

The ideal state of the cloud-native containerization stage is that business developers only need to focus on implementing business logic, without worrying about resource scheduling and operation and maintenance management, and can truly achieve on-demand use and billing by volume. Virtual machine/physical machine resources have coarse granularity and low utilization efficiency. Computing, storage, and network resources need to be planned in advance, which is far from ideal. In the cloud native stage, the granularity of container resources is finer, the utilization rate is high, the start/destroy speed reaches the second level, and it can be flexibly and elastically scaled (Kubernetes has become the industry standard for container orchestration and scheduling, and the following container environments refer to Kubernetes clusters); network management The environment has also changed. The concept of Service appears. A microservice is often carried by a group of elastically scalable and dynamically scheduled containers (Pods). The IP address of the Pod changes dynamically. This group of Pods generally provides external access as a Service. Management is based on Service. It is easier to split business modules into services to build applications. Coupled with the good elastic scalability of the container environment, the DevOps concept can be implemented well, the iteration of microservices is accelerated, and rolling updates are often required. At this time, the ingress traffic management faces the following new challenges:

1. It needs to be integrated with Kubernetes to support forwarding traffic to the specified Pod.
2. The update iteration speed is accelerated, and the demand for the gray release of the new version of the service is even stronger.
3. The cluster concept appears, the service discovery between clusters is isolated, and the access layer needs to support cross-cluster service discovery (that is, the access layer can choose backend to be a Pod of multiple clusters); this is different from the traditional physical machine/virtual machine phase , There is no cluster isolation, only need to ensure network connectivity, you can configure the back end of the access layer for any corresponding service IP address.
4. During the migration process from the traditional stage to the cloud-native stage, the VM and container environments were mixed.

Based on the above challenges, the following access layer traffic management solutions for container environments have emerged:

1. Ingress API officially defined by Kubernetes: veteran network proxies (eg Nginx, HAProxy) or cloud vendors' load balancing products (eg AWS Elastic Load Balancer, Tencent Cloud CLB) have implemented their respective Ingress Controllers, which are used as ingress traffic management solutions for a single cluster Program. Depending on the capabilities of the Ingress Controller, the capabilities of grayscale publishing and authorization flow authentication can be extended through Annotation. Some Ingress Controllers have also designed their own traffic management models and grammars.
2. Service Mesh Ingress: The service discovery and management boundary of the service mesh is larger than the cluster latitude. Taking Istio Ingress Gateway as an example, based on Istio's cross-cluster service discovery capability, the backend can come from services of different clusters, and it also supports registration to run in the grid. Services on virtual machines. Istio also designed its own management model and grammar, declaratively supporting the consistent configuration of north-south + east-west traffic management.
3. The network proxy deployed on the original VM is used to forward traffic to the VM service or the service of the Kubernetes cluster.

The following article will start with the usage scenarios of ingress traffic management in the cloud-native containerized environment, and take you to understand the various solutions and comparison of advantages and disadvantages of cloud-native access layer traffic management.

Cloud native access layer traffic management scenarios and solutions

Scenario 1: Basic traffic management

The first use scenario of ingress traffic management is to expose the service to the outside for the client to call. The common way is to expose the service by URL. For example, an e-commerce website needs to route the request of /login to the login service, and route the request of /product to the commodity service, etc. This scenario requires the access layer to have traffic content routing based ability.

Solution: Load Balancer + NodePort

In the early stage of containerization, applications are deployed on virtual machines and Kubernetes clusters at the same time. Many users will use the original load balancing (eg Nginx, Tencent Cloud CLB) to forward requests to the virtual machines and containers respectively, while being limited by the container network In the solution, the original load balancer cannot directly access the Pod IP, so the services in the cluster need to be exposed through the NodePort.

However, the program has the following problems:

1. The number of NodePort ports is limited (default 30000-32767)
2. With the expansion of the cluster size, the Nginx configuration file becomes more and more complex and difficult to manage
3. After the user publishes the application to the Kubernetes cluster, he needs to modify the Nginx configuration separately, and the experience is not cloud-native enough

Solution: Kubernetes Ingress

Kubernetes provides the Ingress API [1] to expose HTTP services in the cluster. Ingress supports routing requests to different services based on Host and Path. In order for Ingress to work, the cluster must have a running Ingress controller (eg Nginx Ingress Controller). The native Ingress syntax provides simple Host, Path routing, and the ability to configure TLS.

1. Host-based routing

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-services
  namespace: default
spec:
  rules:
  - host: service1.tencent.com
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80
        path: /
  - host: service2.tencent.com
    http:
      paths:
      - backend:
          serviceName: service2
          servicePort: 80
        path: /

2. Route based on Path

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-services
  namespace: default
spec:
  rules:
  - host: services.tencent.com
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80
        path: /service1
      - backend:
          serviceName: service2
          servicePort: 80
        path: /service2

3. TLS configuration

Ingress also provides TLS support, which can expose the HTTP service in the cluster as HTTPS. We need to save the SSL certificate in the cluster in the form of a Secret, and then use Ingress to reference the Secret just created.

apiVersion: v1
kind: Secret
metadata:
  name: public-services-tls
  namespace: default
data:
  tls.crt: base64 encoded cert
  tls.key: base64 encoded key
type: kubernetes.io/tls
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: services-with-tls
  namespace: default
spec:
  tls:
  - hosts:
      - services.tencent.com
    secretName: public-services-tls
  rules:
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80
        path: /service1
      - backend:
          serviceName: service2
          servicePort: 80
        path: /service2

Kubernetes Ingress Summary : For simple HTTP traffic routing, it is very easy to use Ingress configuration, which is the reason why Ingress is currently popular (according to CNCF 2020 Cloud Native Survey Report [2], 50% of users are or will use a third party The proxy does application traffic forwarding, among which Nginx and Envoy [3] are the most popular Ingress providers).

But on the other hand, native Ingress has very limited functions and cannot meet the needs of many complex scenarios. Many third-party Ingress Controllers [4] extend the functionality of native Ingress through annotations or new configuration models and syntax, but are still limited by the problem of service discovery and isolation between clusters and can only be used as a single-cluster ingress traffic management solution.

Scene 2: Grayscale release

After the service can be exposed to external access, it is also necessary to consider how to release the version and iterate smoothly and without risk. Two common methods are to cut part of the traffic to the new version to verify the stability according to the weight or traffic content, and gradually transition to the new version when there is no problem, that is, we are familiar with the gray release and AB test.

The Kubernetes Ingress API does not have a gray-scale publishing function natively. The Nginx ingress controller extends the functions of the native Ingress API through annotations to achieve gray-scale publishing, but this method does not support the publishing strategy of controlling application traffic well. In contrast, Istio CRD configuration is more flexible and easy to use. The following describes how to use Istio VirtualService to configure grayscale publishing routing rules.

1. Based on the weight

Istio can configure weight-based grayscale publishing through Virtual Service. The following is an example of configuring 95% of the ingress traffic from {namespace}/{gateway} to be routed to the current version of {service} and 5% to the canary version:

apiVersion: ...
kind: VirtualService
metadata:
  name: canary-weight
spec:
  hosts:
    - '*'
  gateways:
    - {namespace}/{gateway}
  http:
    - route:
        - destination:
            host: {service}
            subset: current
          weight: 95
        - destination:
            host: {service}
            subset: canary
          weight: 5

2. Based on the requested content

VirtualService also supports the configuration of content-based grayscale publishing routing rules. The following is the configuration of ingress traffic from {namespace}/{gateway} when the header cookie "version=stable" is routed to the current version of {service}, and when "version=canary" Example of the canary version routed to {service}:

apiVersion: ...
kind: VirtualService
metadata:
  name: canary-content
spec:
  hosts:
    - '*'
  gateways:
    - {namespace}/{gateway}
  http:
    - match:
        - headers:
            cookie:
              exact: version=stable
      route:
        - destination:
            host: {service}
            subset: current
    - match:
        - headers:
            cookie:
              exact: version=canary
      route:
        - destination:
            host: {service}
            subset: canary

Scenario 3: Application traffic authentication and current limiting

Authentication and current limiting are two important capabilities to ensure the security and robustness of north-south traffic.

The access layer is a unified entrance to access back-end services. Ensuring the security of the access layer is an important scenario for access layer traffic management. Generally, authentication and authorization rules need to be configured at the entrance. The authentication and authorization functions under the traditional architecture generally pass code logic To achieve, Istio has provided AuthorizationPolicy and RequestAuthentication CRD [5] since 1.5, which can flexibly configure the authentication and authorization rules of the entry layer.

1. Request for identity authentication (JWT)

The Json Web Token carried in the authentication request at the entrance allows requests with legal tokens to pass, and rejects requests with illegal tokens.

The following is a configuration example of using Istio RequestAuthentication to configure the Ingress Gateway to allow legitimate JWT requests to be carried:

apiVersion: ..
kind: RequestAuthentication
metadata:
  name: jwt-example
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: {issuer that issued the JWT}
    jwksUri: {URL of the provider’s public key set to validate signature of the JWT}

2. Authorization

Configure authorization policies at the entrance and allow/deny traffic access based on the characteristics of the traffic content. For example, configure IP black/white lists at the entrance; or there is an external authentication service, and it is hoped that the entrance component can connect to the external authentication service and follow the return The authentication result allows/rejects traffic.

The following is an example of using Istio AuthorizationPolicy to configure the IP block whitelist for the Ingress Gateway:

apiVersion: ...
kind: AuthorizationPolicy
metadata:
  name: white-list
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        ipBlocks: {single IP or CIDR}

Istio 1.9 enhanced AuthorizationPolicy's support for docking with external authentication systems. Ingress Gateway can be configured to allow or deny traffic according to the results returned by the external authentication system.

apiVersion: ...
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: CUSTOM
  provider:
    name: "my-ext-authz-service"
  rules: ...

3. Current limit
The business scale is large. When the back-end service is provided to many tenants, the request rate needs to be controlled at the entrance. For example, each User ID can only request the "/product" interface 100 times per minute.
In order to use the current limiting function of Istio Ingress Gateway, you first need to install the Ratelimit service, you can implement it yourself or directly use the community ratelimit [6], and then use Envoyfilter to configure the current limiting rules. For the specific configuration method, please refer to the official document [7].

Scenario 4: Ingress traffic management in multi-cluster heterogeneous scenarios

With the increase of business scale, or the improvement of the requirements for disaster tolerance, data compliance, and isolation between businesses, the business will consider and implement multiple Kubernetes clusters, and even heterogeneous mixing of containerized and non-containerized environments will occur The situation has brought a series of new challenges to ingress traffic management.

Multiple Kubernetes clusters are generally based on two considerations: disaster tolerance and business isolation:

(1) Disaster tolerance. Kubernetes clusters have geographic attributes. According to the access timeliness and disaster tolerance requirements of the services provided by application delivery, the same application may be distributed in multiple different geographic regions. Disaster recovery of multi (public) cloud and hybrid cloud (IDC + public cloud) architecture also requires the deployment of multiple clusters. Cross-regional multi-cluster disaster recovery and nearby access can be provided through DNS resolution, but DNS has cache, and the actual effective time of failover may be longer, and it is not possible to cut part of the traffic to the backup region depending on the health of the service, and can only switch all.

Istio is based on the following capabilities: 1. Multi-cluster service discovery capabilities; 2. Regional awareness, fault awareness, and disaster tolerance traffic capacity planning, which can be achieved: 1. When all cluster services are healthy, route the request to the nearest service according to the source of the request ; 2. When a service of a certain cluster fails, a certain percentage of traffic will be transferred to the backup service of other clusters depending on the health of the service.

(2) Business isolation. According to the CNCF 2020 Cloud Native Survey Report [2], the use of multiple clusters for application isolation is second only to the use of namespace isolation, and the utilization rate has increased from 47% in 2019 to 50% in 2020. When multiple services still share the same traffic entry, the access layer needs to have the capability of multi-cluster service discovery, and route traffic to the services of the specified cluster according to the specified strategy.

Scheme: Service Mesh Ingress

One of the challenges encountered by the Kubernetes Ingress Controller is that the Kubernetes cluster isolates service discovery between clusters, and the Ingress Controller can only be used as a cluster-level traffic entrance. With the help of the control plane service discovery capability, Service Mesh technology can discover or register services of multiple clusters or even heterogeneous services, open up the service discovery barriers between clusters, and is not restricted by the application deployment platform, and naturally provides consistent access traffic forwarding Management ability.

Istio is the most popular Service Mesh open source project. Its access layer Istio Ingress Gateway also provides support for the Ingress API, but it is not recommended to use Ingress to configure the Ingress Gateway, which greatly weakens the capabilities of Istio. Istio provides a higher level of abstraction to the traffic management model. You can directly use the Istio API to achieve more flexible traffic management capabilities, achieve gray-scale publishing, cross-cluster routing, and regional awareness and other advanced features.

Istio Ingress Gateway is based on Envoy [3]. Envoy was originally created by Lyft as a high-performance service proxy software designed for cloud-native scenarios. It was later donated by Lyft to the CNCF community and has graduated from CNCF.

1. Multi-Kubernetes cluster service management

Istiod can obtain endpoints information through the API Servers of all clusters in the grid. After aggregating the information of multiple clusters, it pushes the final configuration to the Ingress Gateway. The Ingress Gateway can forward the request to all Pods in the grid on demand.

2. Region-aware load balancing

In the service grid, the geographic information of a Pod includes the following three parts [8]:

• Region : Usually represents a larger geographic area (eg Beijing and Shanghai). In Kubernetes, the region of the node is determined topology.kubernetes.io/region
• Zone (Availability Zone) : A region usually contains multiple Availability Zones (eg Beijing District 1 and Beijing District 2). In Kubernetes, the node's availability zone is determined topology.kubernetes.io/zone
• Sub-zone : Allows further division of available zones to achieve more fine-grained control. For example, it can be divided according to rack (rack). There is no concept of sub-zone in Kubernetes. Istio uses the topology.istio.io/subzone label of the node to define sub -zone

If you use the Kubernetes service hosted by a cloud vendor, the Region and Zone labels of the node have been configured by the cloud vendor. For example, in a TKE cluster, the nodes in the second district of Shanghai will have the following labels:

• topology.kubernetes.io/region: sh
• topology.kubernetes.io/zone: "200002"

Clusters in the grid may be distributed in different regions and different availability zones. In most cases, we want to minimize cross-region/cross-availability zone request calls, because this will increase request latency. Therefore, the access layer needs to have the ability to sense the geographic information of endpoints, and support the configuration of load balancing and failover strategies based on geographic information.

(1) Regional failover

When geographical load balancing is turned on, Istio will tell the Ingress Gateway to forward the request to the nearest location. When all instances are normal, the request will remain in the same location. When the instance is abnormal, the traffic will be distributed to the instance in the next priority region.

For example, the priority of the Ingress Gateway at bj.bj-01 for forwarding requests is as follows:

priority	Geographical Location
1	bj.bj-01	Region Zone exact match
2	bj.bj-02	Region matches Zone does not match
3	sh.sh-01/sh-02	Region Zone does not match

(2) Geographically weighted load balancing

Geographically weighted load balancing can distribute a certain percentage of user-defined traffic to certain regions. For example, we can use the following configuration to distribute traffic:

global:
  localityLbSetting:
    enabled: true
    distribute:
    - from: bj/bj-01/*
      to:
        "bj/bj-01/*": 70
        "bj/bj-02/*": 20
        "sh/sh-01/*": 10

3. Heterogeneous service ingress traffic management

In addition to multi-clusters, in the process of cloud-native transformation, users often face the situation that some services have been containerized and run in a Kubernetes cluster, and some inconvenient services are still in virtual machines, and some even use the cloud. Vendor serverless cloud function service (eg AWS lambda). The access layer needs to have the ability to register/discover heterogeneous services to manage the north-south traffic of heterogeneous deployment services.

The service on the virtual machine can be registered in the grid through the WorkloadGroup and WorkloadEntry provided by Istio, and the same service can run on the Kubernetes cluster and the virtual machine at the same time.

Istio Ingress Gateway Summary : Istio Ingress Gateway provides multi-cluster service discovery, geographic awareness, traffic capacity planning, and more powerful and flexible traffic management API support in scenarios such as portal gray release, security, and multi-cluster heterogeneous traffic management. , But at the same time, users have to face the complexity of Istio. It is necessary to invest resources and labor costs to operate and maintain Istiod and Istio Ingress Gateway, integrate metric, trace, log and other observability and certificate management peripheral systems with high costs, and also need to properly configure various CRDs (Gateway VirtualService Destination Rule, etc.)

Function comparison of access layer solutions

The following is a comparison of common access layer solutions in the Tencent Cloud container environment.

Multi-cluster grayscale release/Cross-cluster disaster recovery Demo

The following will use Tencent Cloud Service Mesh TCM console to demonstrate Service Mesh Ingress to do grayscale publishing and disaster recovery in a multi-Kubernetes cluster environment.

1. Create a service grid, add two service discovery clusters for deploying services (basic monitoring indicators are automatically connected to cloud monitoring and can be viewed in the console, and cloud native monitoring can be turned on according to the situation to meet custom monitoring requirements), and check Enable Ingress Gateway
   
2. Use Destination Rule to define the version of the frontend service (frontend service has the same deployment in both clusters)
   
3. Use Gateway to configure ingress gateway monitoring rules, enable https access on port 443, and use Tencent Cloud SSL platform server certificate
   
4. Use VirtualService to configure routing rules, 50% of the traffic is routed to the v1 version, and 50% to the v2 version
   
5. After there is an access request, check the workload (frontend, frontend-canary) monitoring, both versions have traffic, the ratio is roughly 1:1
   
6. The gray scale is over, the weight is changed, 100% of the traffic is routed to the v2 version, and the monitoring data of the workload is checked again, and it is found that all the traffic has been requested to the frontend-canary
   
   
7. Next, we will simulate the frontend service failure of one of the clusters by adjusting the number of frontend service workload Pods of one of the clusters to 0. After discovering that the frontend service of one of the clusters fails, the service can still be accessed normally and check the frontend service of the other cluster. Workload monitoring will find that the incoming bandwidth has doubled, indicating that after the service failure of one of the clusters, the traffic disaster recovery is switched to the other cluster.
   
   
   
8. If you need to expand east-west traffic management, you can inject the envoy sidecar into your business, and you can use the same set of Istio APIs to achieve consistent management of north-south east-west traffic, and use network topology, call tracking and other observability functions out of the box.
   
   

Tencent Cloud Service Mesh TCM is a service mesh product of Tencent Cloud that is fully compatible with Istio. It has realized the hosting of control plane components. To use TCM Ingress Gateway, you only need to deploy a set of data plane envoy pods in the business cluster, which can be used out of the box. All ingress traffic management capabilities of the aforementioned Istio Ingress Gateway. At the same time, TCM integrates Tencent Cloud monitoring and certificate peripheral products to provide out-of-the-box observability and certificate configuration functions.

Conclusion

This article is introduced by the two stages of business deployment development and introduces:

1. A typical scenario of access layer traffic management in a cloud-native containerized environment.
2. Ingress traffic management solutions and comparison of advantages and disadvantages.
3. Taking Tencent Cloud Service Grid TCM as an example, it demonstrates the ability of service mesh ingress grayscale publishing and cross-cluster disaster recovery in a multi-cluster environment.

The main conclusions are:

1. For simple HTTP traffic routing, it is very easy to use the Kubernetes native Ingress configuration. Some Ingress Controllers (eg Nginx, Traefik) also extend the functionality of native Ingress through annotation or CRD, but they are still cluster-level traffic entrances.
2. The Service Mesh level access layer, with the help of the control plane service discovery capability, can be used as a unified traffic entry in a multi-cluster/heterogeneous environment. It can have advanced features such as cross-cluster routing and geographic awareness; subsequent smooth expansion of consistent syntax management East-west flow.

This article is the first in a series of articles on cloud-native access layer traffic management. In the future, we will launch a series of articles detailing best practices in scenarios such as ingress traffic management, security, observability, and multi-cluster heterogeneous ingress traffic management. .

Reference

[1] https://kubernetes.io/docs/concepts/services-networking/ingress/

[2] https://www.cncf.io/wp-content/uploads/2020/12/CNCF_Survey_Report_2020.pdf

[3] https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy

[4] https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/

[5] https://istio.io/latest/docs/reference/config/security/

[6] https://github.com/envoyproxy/ratelimit

[7] https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/

[8] https://istio.io/latest/docs/tasks/traffic-management/locality-load-balancing/

[Tencent Cloud Native] Yunshuo new products, Yunyan new technology, Yunyou Xinhuo, Yunxiang information, scan the QR code to follow the public account of the same name, and get more dry goods in time! !