Want to perform custom monitoring in KubeSphere? Come see this

In early November, KubeSphere released version 3.2.0 . The new version adds a complete set of monitoring and management pages to the project gateway, and at the same time introduces a cluster gateway to provide the global Ingress gateway capability at the cluster level.

In order to let users better understand how to deploy and use third-party Ingress Controller in the new version of KubeSphere, this article will take Apache APISIX Ingress Controller as an example to show you how to quickly use different types of gateways for Kubernetes clusters and perform status monitoring through KubeSphere.

Ready to work

Install KubeSphere

There are two ways to install KubeSphere:

• Install directly on Linux
• Install in the existing Kubernetes [2]

The minimal installation version of KubeSphere already includes the monitoring module, so no additional activation is required. You can confirm the installation status through the "Monitoring" tab in the "System Components" page.

Deploy httpbin demo application

Due to the need to demonstrate the access control capabilities of the gateway, we must first have an accessible application as the backend service of the gateway. Here we use the kennethreitz/httpbin container application provided by httpbin.org as the demo application.

In KubeSphere, we can first create a new project or use an existing project. After entering the project page, select "Service" under "Application Load" to directly create a stateless workload and generate supporting services.

Use kennethreitz / httpbin container default 80 port as a service port, after the completion of creation to ensure that under the "workload" and "Service" page you can see httpbin corresponding entry, as shown below.

### Project gateway details supplement

The project gateway is a function launched after KubeSphere 3.0: the gateway in the KubeSphere project is an NGINX Ingress controller. KubeSphere's built-in mechanism for HTTP load balancing is called application routing, which defines the connection rules from the outside to the cluster service. To allow external access to services, users can create routing resources to define URI paths, back-end service names and other information.

httpbin service project that has been deployed above, open the "Gateway Settings" page in the "Project Settings", and then perform the "Open Gateway" operation. For convenience, just select NodePort as the "Access Method".

After confirming, return to the gateway page, wait a while and refresh the page, you can get the deployment completion status as shown in the figure below, where you can see that NodePort is assigned two node ports by default. To accept the recruitment, we use the "Manage" button in the upper right corner to "View Details".

At this time, what we see is the new monitoring page of the 3.2.0 version about the project/cluster gateway. Next we need to create an application route for the httpbin service.

From the "application load", enter the "application routing" page, and start to "create" a route. After naming the route httpbin , we specify a domain name that is convenient for testing, and set the "path" to / , select "service" httpbin and "port" 80 .

In the next step, skip the advanced settings and complete the route creation, and you can get the httpbin application route item as shown in the figure below.

Next, we can access the httpbin application service through the NodePort address of the project gateway and the specified domain name (for example, http://httpbin.ui:32516 here), refresh or operate the request generation function of the page at will, and then enter the gateway's details page, you can see it in " Some of the built-in monitoring indicators of the gateway have appeared on the "Monitoring" panel.

Specify NodePort node port

For public cloud environments, if you use NodePort to expose access capabilities, the open ports are usually limited and controlled. Therefore, we need to modify the NodePort used by the gateway.

Since the gateway is managed uniformly by KubeSphere, to modify the NodePort of the gateway service, you need to have the project permission to kubesphere-controls-system kubesphere-router-<project-namespace> and open NodePort for external access through the "Service" page of "Application Load". The NodePort service port needs to be directly modified through "Edit YAML".

Start using the cluster gateway

KubeSphere 3.2.0 began to support cluster-level global gateways. All projects can share the same gateway, and project gateways that have been created before will not be affected by the cluster gateway. The gateways of all projects can also be managed uniformly, and they can be centrally managed and configured. Administrator users no longer need to switch to different corporate spaces to configure gateways.

If you are using KubeSphere 3.2.0 version, we recommend that you use the cluster gateway function to unify the application routing of the entire cluster. To enable the cluster gateway is also very simple: use an account with cluster management permissions to enter a cluster that it can manage (for example, we take the default cluster as an example), and you can "open" in the "Gateway Settings" of "Cluster Settings" Gateway”, and check the “Project Gateway” at the same time.

The opening method of the cluster gateway and the modification of aligning the NodePort access port are basically the same as the operation of the previous project gateway, so I won't go into details here.

However, there is one point that needs special attention : After the cluster gateway is turned on, the opened project gateway will still be retained; but the project that has not yet created a gateway can no longer create a separate gateway, and the cluster gateway will be used directly.

The figure below shows an overview of all gateways displayed on the "Gateway Settings" page after having a project and a cluster gateway at the same time for a project with a gateway that has been created.

Quickly use Apache APISIX Ingress Controller

Apache APISIX is an open source, high-performance, dynamic cloud native gateway, donated by Shenzhen Tributary Technology Co., Ltd. to the Apache Foundation in 2019. It has become the top open source project of the Apache Foundation and the most active gateway project on GitHub. Apache APISIX currently covers multiple scenarios such as API Gateway, LB, Kubernetes Ingress, and Service Mesh.

How to deploy

First add the Apache APISIX Helm Chart repository. Then select an enterprise space and add the following Apache APISIX warehouse (Warehouse URL: https://charts.apiseven.com ) through the "Application Repository" under "Application Management".

Next create a project apisix-system After entering the project page, choose to create an "application" in the "application load" to deploy Apache APISIX, and select the apisix application template to start deployment.

Why is the Helm Chart of the Apache APISIX application directly deployed instead of the Apache APISIX Ingress Controller? This is because the Apache APISIX Ingress Controller is currently strongly associated with the Apache APISIX gateway (as shown in the figure below), and it is currently through the Apache APISIX Helm Charts It is the most convenient to deploy Apache APISIX Gateway + Dashboard + Ingress Controller at the same time, so this article recommends directly using the Helm Chart of Apache APISIX to deploy the whole set of components.

Name the application apisix to avoid mismatches in the workload and service names of multiple components (Gateway, Dashboard, Ingress Controller); in the "application settings" section edited in the installation step, please refer to the following configuration to fill in ( , please pay special attention to the description in the comment part marked with [Note], and the rest can be edited and modified by . 161af22a105d8b).

global:
  imagePullSecrets: []
  
apisix:
  enabled: true
  customLuaSharedDicts: []
  image:
    repository: apache/apisix
    pullPolicy: IfNotPresent
    tag: 2.10.1-alpine
  replicaCount: 1
  podAnnotations: {}
  podSecurityContext: {}
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAntiAffinity:
    enabled: false
    
nameOverride: ''
fullnameOverride: ''

gateway:
  type: NodePort
  externalTrafficPolicy: Cluster
  http:
    enabled: true
    servicePort: 80
    containerPort: 9080
  tls:
    enabled: false
    servicePort: 443
    containerPort: 9443
    existingCASecret: ''
    certCAFilename: ''
    http2:
      enabled: true
  stream:
    enabled: false
    only: false
    tcp: []
    udp: []
  ingress:
    enabled: false
    annotations: {}
    hosts:
      - host: apisix.local
        paths: []
    tls: []
    
admin:
  enabled: true
  type: ClusterIP
  externalIPs: []
  port: 9180
  servicePort: 9180
  cors: true
  credentials:
    admin: edd1c9f034335f136f87ad84b625c8f1
    viewer: 4054f7cf07e344346cd3f287985e76a2
  allow:
    ipList:
      - 0.0.0.0/0
      
plugins:
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  - echo
  - fault-injection
  - grpc-transcode
  - hmac-auth
  - http-logger
  - ip-restriction
  - ua-restriction
  - jwt-auth
  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  - node-status
  - openid-connect
  - authz-casbin
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - traffic-split
  - gzip
  - real-ip
  #【注意】添加此插件以配合 Dashboard 展示服务信息
  - server-info

stream_plugins:
  - mqtt-proxy
  - ip-restriction
  - limit-conn

customPlugins:
  enabled: true
  luaPath: /opts/custom_plugins/?.lua
  #【注意】如下配置保障 Prometheus 插件可对外暴露指标
  plugins:
   - name: prometheus
     attrs:
       export_addr:
         ip: 0.0.0.0
          port: 9091
      configMap:
       name: prometheus
        mounts: []

dns:
  resolvers:
    - 127.0.0.1
    - 172.20.0.10
    - 114.114.114.114
    - 223.5.5.5
    - 1.1.1.1
    - 8.8.8.8
  validity: 30
  timeout: 5

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

configurationSnippet:
  main: ''
  httpStart: ''
  httpEnd: ''
  httpSrv: ''
  httpAdmin: ''
  stream: ''

etcd:
  enabled: true
  host:
    - 'http://etcd.host:2379'
  prefix: /apisix
  timeout: 30
  auth:
    rbac:
      enabled: false
      user: ''
      password: ''
    tls:
      enabled: false
      existingSecret: ''
      certFilename: ''
      certKeyFilename: ''
      verify: true
  service:
    port: 2379
  replicaCount: 3

dashboard:
  enabled: true
  #【注意】为 Dashboard 开启 NodePort 方便后续使用
  service:
   type: NodePort

ingress-controller:
  enabled: true
  config:
    apisix:
     #【注意】一定要设置 gateway 所在的 namespace
      serviceNamespace: apisix-system
  serviceMonitor:
    enabled: true
    namespace: 'apisix-system'
    interval: 15s

After the deployment is successful, click the application name to enter the details page, and you can see the following service deployment and working status display under the "Resource Status" tab.

💡 The default configuration parameters of the two other Helm Charts of the Apache APISIX project can be referred to respectively: Dashboard and Ingress Controller of values.yaml .

Dashboard magical use

After the Apache APISIX application is deployed, you can check the current status of the Apache APISIX gateway through the Apache APISIX Dashboard.

apisix-dashboard service from the application load-service page. Since we have enabled the NodePort for the Dashboard in the application configuration, we can directly access the Dashboard through the NodePort port here.

Use the default user name and password admin log in to the Apache APISIX Dashboard, you can enter the "System Information" page to view the current connection management "Apache APISIX Node" information.

how to use

Next, let's go back to the "application routing" page, and create a new route (such as apisix-httpbin ), set the path to /* httpbin 80 and add the key value of kubernetes.io/ingress.class : apisix

Verify that the application route is valid

Go back to the Apache APISIX Dashboard and enter the "Routing" page. You can see that the newly created application route has been recognized by the Apache APISIX Ingress Controller and automatically added to the Apache APISIX gateway. You can also see an automatically created upstream entry on the "Upstream" page.

Next, go back to the "Services" page of the apisix-system project, find the port corresponding to the apisix-gateway <domain name specified by the apisix-httpbin application route>:<apisix-gateway external access port> (for example, httpbin.ui:30408 here) Access to the background service associated with the apisix-httpbin

Custom monitor Apache APISIX gateway

When using the Apache APISIX gateway, the monitoring capabilities can be supported by the Prometheus plug-in and the custom monitoring capabilities that come with KubeSphere.

Expose related Prometheus monitoring indicators

Since we have already enabled the Prometheus plug-in when deploying the Apache APISIX application, we only need to expose the interface of the Prometheus monitoring indicator.

Enter the apisix-system apisix on the "workload" page and enter the deployment details page, and then select "Edit Settings" in the "More Operations" on the left operation panel.

In the pop-up panel, enter the apisix container editing interface, find "Port Settings", add a new prom to the container's 9091 port, after apisix workload will restart.

Create ServiceMonitor for monitoring indicators

Next, we need to connect the exposed indicator interface to KubeSphere's own Prometheus to make it accessible (captured indicator data).

Since KubeSphere Prometheus Operator , the quickest way is to directly create ServiceMonitor resources to implement indicator interface access.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: apisix
  namespace: apisix-system
spec:
  endpoints:
    - scheme: http
     #【注意】使用上一步中工作负载暴露的容器端口名称
     targetPort: prom
     #【注意】需要正确绑定 apisix 对应的指标接口路径
     path: /apisix/prometheus/metrics
      interval: 15s
  namespaceSelector:
    matchNames:
      - apisix-system
  selector:
    matchLabels:
      app.kubernetes.io/name: apisix
      app.kubernetes.io/version: 2.10.0
      helm.sh/chart: apisix-0.7.2

Use kubectl apply -f your_service_monitor.yaml create a ServiceMonitor resource. After the creation is successful, if you have cluster management authority, you can also search and view the ServiceMonitor resource in the CRD management page of the cluster and find apisix . You can also make subsequent YAML modifications here.

Indicator access to custom monitoring panel

In the menu list on the left side of the project, find "Custom Monitoring" in "Monitoring Alarms", and start "Create" a custom monitoring panel.

Fill in the "Name" in the pop-up window, select the "Custom" monitoring template, and enter the "Next" monitoring panel creation.

After entering the edit page, click on the + area on the left, and configure the Prometheus monitoring indicators in the "Data" area on the right. For example, here we can use sum(apisix_nginx_http_current_connections) to count the total real-time connections of the Apache APISIX gateway.

After saving, find "+ Add Monitoring Item" in the lower right corner of the page, and select "Line Chart" to create the Nginx connection state indicator: use sum(apisix_nginx_http_current_connections) by (state) as the indicator, {{state}} as the legend name, and "Legend Type" as the stacked graph, you can get something similar to the figure below result. After saving the template, you will get the first custom monitoring panel!

Prometheus indicators Apache APISIX Gateway currently offers can be found official documents can be some indicators part .

Due to the troublesome indicator configuration process, it is recommended to directly import the Apache APISIX Grafana template in the "Custom Monitoring" at the cluster level (download the JSON and import it through the "local upload").

After the creation is complete, a very rich Apache APISIX gateway monitoring panel can be directly displayed. KubeSphere is also actively promoting the introduction of the Grafana template import function into the project's custom monitoring capabilities, so stay tuned!

Summarize

Through the very detailed step-by-step introduction in this article, everyone can fully understand and follow the hands-on experience how to "connect the Apache APISIX Ingress gateway to KubeSphere and perform custom monitoring". I hope that by reading this article, you can deepen your understanding of Apache APISIX Ingress Controller and Apache APISIX applications.

About the author

Author Zhang Haili, R&D Director of Uisee Technology Cloud Platform. Open source enthusiast, PMC member of the cloud native community Shanghai station, KubeSphere Ambassador.

About Apache APISIX

Apache APISIX is a dynamic, real-time, high-performance open source API gateway that provides rich traffic management functions such as load balancing, dynamic upstream, grayscale publishing, service fuse, identity authentication, and observability. Apache APISIX can help companies quickly and securely process API and microservice traffic, including gateways, Kubernetes Ingress, and service meshes.

Apache APISIX landing users (only some)

• Apache APISIX GitHub：https://github.com/apache/apisix
• Apache APISIX official website: https://apisix.apache.org/
• Apache APISIX documentation: https://apisix.apache.org/zh/docs/apisix/getting-started