Design and Implementation of KubeSphere Gateway (Interpretation)

Author: Hong Zhouzi, KubeSphere back-end R&D engineer, cloud native enthusiast, now focusing on cloud native microservices.

Why is a gateway needed in KubeSphere?

What are the ways to expose the services in the K8s cluster to external access? It can be exposed by setting the Service as NodePort or by Ingress. In addition, using the Ingress method can achieve the advantages of distributing requests to one or more services, and exposing multiple services under the same IP address.

However, for the Ingress method, only the Ingress CRD (which can create Ingress resources) is built-in in K8s, and there is no built-in Ingress Controller. The Ingress Controller must be deployed to provide the Ingress resource with the ability to externally access the internal services of the cluster. The gateway in KubeSphere is the Ingress Controller.

Gateway Design

KubeSphere v3.2 reconstructs the gateway, and adds the following new functions on the basis of retaining the original gateway functions:

1. Enable cluster and project-level gateways: Gateways with different granularities can be flexibly selected according to business needs.
2. Increase or decrease the number of gateway replicas: flexibly adjust the number of replicas to achieve higher availability.
3. Flexible configuration of Ingress Controller configuration options.
4. You can specify the location where the gateway application load is installed: You can choose to specify a fixed namespace for the installation location of the gateway application load, or let it be located under its own project namespace. Combined with the permission management in KubeSphere, if the resources are located in each project namespace, users with the project permission can also view the gateway resources.
5. Gateway logs: Centrally query the gateway logs, and query the gateway logs distributed in each replica in a centralized manner.
6. Gateway monitoring indicators: Monitor some indicators in the gateway, including indicators such as the total number of requests/success rate/delay.

Implementation of the gateway

Currently, K8s supports and maintains AWS , GCE and Nginx Ingress controllers. KubeSphere uses Ingress Nginx Controller as the default gateway implementation without any code modification.

The realization idea of each function point

• Cluster and project-level gateways: This is implemented by passing in parameters to override the default Helm Chart Values and controlled in the code logic. If the cluster gateway is enabled, the project gateway cannot be enabled; if the project gateway is enabled and the cluster gateway is enabled, Then it can be accessed through both gateway entrances, but there will be two Ingress Controllers watching the same Ingress object at the same time.
• Increase or decrease the number of gateway replicas & configure Ingress Controller configuration options: This is achieved by overriding the default Helm Chart Values by passing in parameters. The Helm Operator used in the implementation process will be introduced later.
• You can specify the location where the gateway application load is installed: You can choose to specify a fixed namespace for the installation location of the gateway application load, or let it be located under its own project namespace. This is controlled in the code logic and made into a configuration item. By default, all resources are installed under kubesphere-controls-system.
• Gateway log: The log component in KubeSphere is used. The log component collects log data and stores it in Elasticsearch. The gateway queries logs in Elasticsearch according to the parameters during the log query process.
• Gateway monitoring indicators: The monitoring component in KubeSphere is used, and KubeSphere is internally configured with Prometheus-related parameters to collect Ingress-related indicators. The process of querying monitoring information will query related data according to the API in the monitoring component.

The following focuses on the CRD abstracted from the design and implementation process and how to integrate it skillfully with Helm Operator.

Abstract the Gateway CRD for adaptation

In design, a Gateway CRD is abstracted to adapt to different Ingress Controllers. The Gateway CRD contains the public properties required to set the Ingress Controller. KubeSphere API and UI only interact with Gateway CRD.

 # Gateway sample
apiVersion: gateway.kubesphere.io/v1alpha1
kind: Gateway
metadata:
  name: kubesphere-router-proj1
  namespace: kubesphere-controls-system # all Gateway workload will be created in the kubesphere-controls-system namespace by default. However, it's configurable in kubesphere-config when calling KubeSphere API. 
spec:
  controller:
  # controlpanel replicas. For ingress Controler that has controlpanel and workers. *Reserved field. Changing on UI isn't supported yet. 
  replicas: 1
  # annotations of the controlpanel deployment. *Reserved field. Changing on UI isn't supported yet. 
  annotations: {}
  
  # Watching scope,
  # enabled =true, watching for the project only. The user needs to specify the watching namespace.
  # enabled =false, Global gateway, watching for all namespaces.
  scope:
    enabled: false
    namespace: ""  # defaults to .Release.Namespace

  # gateway configurations. only key-value pair supported currently.
  config:
    max-bucket: 1m

  # worker workload deployment configuration
  deployment:
    annotations: 
    "servicemesh.kubesphere.io/enabled": "false"
    replicas: 1

  # 
  service:
    # Cloud LoadBalancer configurations for service
    annotations: 
      "service.beta.kubernetes.io/qingcloud-load-balancer-eip-ids": "test-ip-id"
    # Service Type, only LoadBalancer and NodePort are supported
    type: LoadBalancer

Integrate Nginx Ingress Controller

KubeSphere uses Nginx Ingress Controller as the default gateway implementation. To simplify deployment steps, we integrated Helm-operator-plugins as Helm Operator .

There are the following key points in Helm Operator:

Create or update Chart resources according to the CR under the specified CRD configured in watch.yaml. The value in the default Helm Chart can be overwritten according to the value in the CR spec, which is determined by the mechanism in the Helm Operator, see the official description for details .

The meaning of the following is that the Nginx CR of Watch gateway.kubesphere.io/v1alpha1 is required. If there is a change, the Reconcile will be triggered, and the corresponding resource will be created or updated according to the address configured in the chart.

 - group: gateway.kubesphere.io
  version: v1alpha1
  kind: Nginx
  chart: /var/helm-charts/ingress-nginx

Usage in KubeSphere:

The following configuration is made in watches.yaml:

 - group: gateway.kubesphere.io
  version: v1alpha1
  kind: Nginx
  chart: /var/helm-charts/ingress-nginx
- group: gateway.kubesphere.io
  version: v1alpha1
  kind: Gateway
  chart: /var/helm-charts/gateway

Among them for chart:

• Nginx uses the official Helm Chart, which is downloaded when ks-controller-manager is packaged. See: https://github.com/kubesphere/kubesphere/blob/v3.2.0/build/ks-controller-manager/Dockerfile#L34
• Gateway is a Helm Chart customized in KubeSphere, which mainly operates Nginx CR resources. See: https://github.com/kubesphere/kubesphere/blob/v3.2.0/config/gateway/templates/nginx-ingress.yaml

Overall:

The Helm Operator Watch watches the resources of the Gateway and Nginx CRDs. When the front-end initiates the creation or update of the gateway, it initiates the creation or update of the Gateway CR:

1. Initiate a request to create or update a Gateway CR;
2. According to the Gateway configured in watches.yaml, Helm Operator will create or update Nginx CR when it detects that there is a Gateway CR resource change;
3. According to the Nginx configured in watches.yaml, Helm Operator creates or updates the Nginx Ingress Contoller by overwriting the value in the default Helm Chart according to the value in the spec in the Nginx CR after monitoring the change of the Nginx CR resource.

Configuration item design

In order to facilitate changing some parameters of the gateway, the following configuration items are designed:

 gateway:
  watchesPath: /var/helm-charts/watches.yaml
  repository: kubesphere/nginx-ingress-controller
  tag: v1.1.0
  namespace: kubesphere-controls-system

• watchesPath: Specifies the configuration file of the Helm Operator Watch. If you need to disable the Helm Operator, you can delete this configuration item.
• repository: Specify the repository of nginx-ingress-controller.
• tag: Specifies the tag of nginx-ingress-controller.
• namespace: The location where the specified gateway application load is installed is located in the specified namespace. If this configuration item is deleted, it will be installed in each project namespace.

Precautions during use

1. If servicemesh is enabled, additional annotations need to be added to the original Ingress nginx.ingress.kubernetes.io/upstream-vhost: [service-name].[service-namespace].svc.cluster.local traffic topology/link tracking can work normally, otherwise there will be exceptions at the ingress traffic.
2. Modifying gateway-related properties, such as the number of replicas, Nginx configuration items, etc., cannot be directly modified in the related deployment/configmap and other application loads, but needs to be modified in the gateway settings (the gateway CR is modified). Because the Helm Operator is used to manage and control the status of the gateway-related resources, all values will be based on the configuration in the Gateway CR, and the changed values in the gateway-related application load will eventually be restored by the Helm Operator.

refer to:

1. https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
2. https://github.com/kubesphere/community/blob/master/sig-microservice/concepts-and-designs/KubeSphere-gateway-operator-design.md
3. https://github.com/kubesphere/kubesphere
4. https://sdk.operatorframework.io/docs/building-operators/helm/

This article is published by OpenWrite , a multi-post blog platform!