Health check of spring boot application in k8s (1)

I. Overview

Container probes were introduced in spring boot 2.3, that is, /actuator/health/liveness and /actuator/health/readiness , were added. For applications deployed in k8s, spring-boot-actuator will automatically perform health checks through these two paths. This article mainly describes and records the use process based on the description of official documents, and introduces from the following aspects:

1. Health check in
2. k8s probe in spring-boot-actuator
3. Spring boot health check practice in

2. K8s container health check

1. Probes in k8s

kubernetes provides three probes (support exec, tcp and http methods) to detect the status of the container:

• LivenessProbe : Container survivability check, used to determine whether the container is healthy, and tell kubelet a container is in an unhealthy state. If the LivenessProbe probe detects that the container is unhealthy, kubelet will delete the container, and deal with it according to the container's restart strategy. If a container does not contain the LivenessProbe probe, then kubelet thinks that the value returned by LivenessProbe Success ;
• ReadinessProbe : Container readiness check, used to determine whether the container is started and ready to receive requests. If the ReadinessProbe probe fails, Endpoint Controller will Endpoint entry containing the IP address of the Pod where the container is located Service of Endpoint If the container does not provide a ready state probe, the default state is Success .
• startupProbe : Container startup check, indicating whether the application in the container has been started. If a startup probe is provided, all other probes will be disabled until this probe succeeds. If the detection fails, kubelet will kill the container, and the container will restart according to its restart strategy. If the container does not provide startup detection, the default status is Success .

startupProbe was added to the alpha version in k8s v1.16. The official explanation for its role is:
Indicates whether the application within the Container is started. All other probes are disabled if a startup probe is provided, until it succeeds. If the startup probe fails, the kubelet kills the Container, and the Container is subjected to its restart policy. If a Container does not provide a startup probe, the default state is Success

2. The probe processing program of k8s

Probe probe inspection is a periodic diagnosis performed by the kubelet on the container. The kubelet calls the Handler implemented by the container. Each probe can be configured with the following three types of handlers:

• ExecAction : Execute the specified command in the container. If the return code is 0 when the command exits, the diagnosis is considered successful.

Check whether the service is normal by executing commands. For example, use the cat command to check whether an important configuration file in the pod exists. If it exists, it means the pod is healthy. Otherwise, it is abnormal. The syntax of the yaml file in Exec detection mode is as follows:

spec:  
  containers:  
  - name: liveness  
    image: k8s.gcr.io/busybox  
    args:  
    - /bin/sh  
    - -c  
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600  
    livenessProbe:               #选择livenessProbe的探测机制  
      exec:                      #执行以下命令，如果文件存在则探测成功
        command:  
        - cat  
        - /tmp/healthy  
      initialDelaySeconds: 5          #在容器运行五秒后开始探测  
      periodSeconds: 5                #检查时间间隔为 5s 检查一次

• TCPSocketAction : Perform TCP check on the specified port on the IP address of the container. If the port is open, the diagnosis is considered successful.
  This method is similar to the HTTPget detection mechanism. The tcpsocket health check is suitable for TCP services. The yaml file syntax of the tcpSocket detection method is as follows:

  spec:  
  containers:  
  - name: goproxy  
    image: k8s.gcr.io/goproxy:0.1  
    ports:  
      - containerPort: 8080  
    #这里两种探测机制都用上了，都是为了和容器的8080端口建立TCP连接  
    readinessProbe:  
      tcpSocket:  
        port: 8080  
      initialDelaySeconds: 5  # 容器启动 5s 后开始第一次就绪性探测
      periodSeconds: 10       # 每 10s 进行一次探测
    livenessProbe:  
      tcpSocket:  
        port: 8080  
      initialDelaySeconds: 15  # 容器启动 15s 后开始第一次就存活性探测
      periodSeconds: 20  # 每 20s 进行一次探测

  In the above yaml configuration file, both types of probes are used. 5 seconds after the container is started, kubelet will send the first readinessProbe probe, which will connect to port 8080 of the container. If the detection is successful, the pod is healthy After ten seconds, kubelet will connect for the second time.

In addition to the readinessProbe probe, 15 seconds after the container is started, kubelet will send the first livenessProbe probe and still try to connect to port 8080 of the container. If the connection fails, restart the container.

• HTTPGetAction : Perform an HTTP Get request on the specified port and path on the IP address of the container. If the response status code is greater than or equal to 200 and less than 400, the diagnosis is considered successful.

The yaml file syntax of Httpget detection method is as follows:

spec:  
  containers:  
  - name: liveness  
    image: k8s.gcr.io/liveness  
    livenessProbe:              #采用livenessProbe机制探测  
      httpGet:                  #采用httpget的方式  
        scheme: HTTP         #指定协议，也支持https  
        path: /healthz          #检测是否可以访问到网页根目录下的healthz网页文件  
        port: 8080              #监听端口是8080  
      initialDelaySeconds: 3     #容器运行3秒后开始探测  
      periodSeconds: 3                #探测频率为3秒  

In the above configuration file, the detection method is that the item container sends an HTTP GET request. The healthz file under port 8080 is requested. Any status code greater than or equal to 200 and less than 400 is returned to indicate success, and any other code indicates abnormality.

Each probe will get one of the following three results:

• Success: The container passed the diagnosis.
• Failure: The container failed the diagnosis.

• Unknown: The diagnosis has failed, so no action will be taken.

  3. Configuration of k8s probe

  Probe has the following configuration fields, which can be used to precisely control the behavior of survival and readiness detection:

• initialDelaySeconds : How many seconds to wait for the survival and ready detectors to be initialized after the container is started, the default is 0 seconds, and the minimum is 0.
• periodSeconds : The time interval (in seconds) to perform detection. The default is 10 seconds. The minimum value is 1.
• timeoutSeconds : How many seconds to wait after the detection timeout. The default value is 1 second. The minimum value is 1.
• successThreshold : The minimum number of consecutive successes that the probe is considered to be successful after failure. The default value is 1. This value must be 1 for survival and start-up detection. The minimum value is 1.
• : The number of Kubernetes retry attempts when the probe fails. Abandoning in the case of survival detection means restarting the container. Abandoned Pod in the case of ready detection will be labeled as not ready. The default value is 3. The minimum value is 1.

Description : Before Kubernetes version 1.20, the exec probe will ignore timeoutSeconds: the probe will continue to run indefinitely, and may even exceed the configured deadline until the result is returned. This defect was fixed in the Kubernetes v1.20 version.

LivenessProbe configuration example:

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  initialDelaySeconds: 30        // 容器启动30s后启动第一次探测
  periodSeconds: 10              //  每隔10s启动一次探测
  timeoutSeconds: 3               // 超时时间3s
  successThreshold: 1             // 成功1次即表示容器健康
  failureThreshold: 5             // 连续5次失败，则判定容器不健康，默认3次

4. When to use

When to use the startup probe

The kubelet can use the startup detector to know when the application container has started. If this type of detector is configured, you can control the container to perform survivability and readiness checks after successful startup to ensure that these survival and ready detectors will not affect the startup of the application. This can be used to check the survivability of slow-start containers to prevent them from being killed before they start running.

When to use the Survival Probe

kubelet uses liveness detectors to know when to restart the container. For example, the survival detector can catch a deadlock (the application is running, but cannot continue to execute the next steps). Restarting the container in this situation helps to make the application more usable in the event of a problem.

when to use the ready probe

The kubelet can use the ready detector to know when the container is ready and can start receiving request traffic. When all the containers in a Pod are ready, the Pod can be regarded as ready. One use of this signal is to control which Pod serves as the backend of the Service. When the Pod is not ready, it will be removed from the Service load balancer.

Reference article:

official website k8s container probe