kubernetes临时存储ephemeral storage

1.临时存储的概念

临时存储即宿主机节点的本地存储。

早期版本的kubernetes提供了对container的CPU、内存的限制，并没有提供对container使用的本地存储的限制，这种情况下，可能存在某些container大量消耗宿主机的存储容量，导致宿主机没有足够的磁盘空间运行核心组件。

container使用的宿主机空间：

• 存放log的目录：/var/lib/kubelet、/var/log/pods
• 存放rootfs的目录: /var/lib/docker

kubernetes在1.8版本引入了ephemeral storage资源，以管理本地临时存储的使用。

2.临时存储的配置

Pod的container可以配置临时存储的requests和limits:

* spec.containers[].resources.limits.ephemeral-storage
* spec.containers[].resources.requests.ephemeral-storage

创建一个deploy，ephemeral-storage申请300Mi，限制300Mi:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:      
      containers:
      - name: nginx
        image: nginx:latest
        imagePullPolicy: IfNotPresent
        command:
        - nginx
        - -g
        - "daemon off;"
        workingDir: /usr/share/nginx/html
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
        resources:
          limits:
            ephemeral-storage: 300Mi
          requests:
            ephemeral-storage: 300Mi

创建出来2个pod:

# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-774779ddfb-89f5m   1/1     Running   0          12s
nginx-774779ddfb-k8r2r   1/1     Running   0          12s

在其中一个pod写入400Mi的文件：

# kubectl exec -it nginx-774779ddfb-89f5m /bin/sh
# dd if=/dev/zero of=/test bs=4096 count=102400
102400+0 records in
102400+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 16.2403 s, 25.8 MB/s

可以看到，写入文件的pod被Evicted，并新创了一个pod：

# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-774779ddfb-89f5m   0/1     Evicted   0          2m19s
nginx-774779ddfb-cv4zw   1/1     Running   0          4s
nginx-774779ddfb-k8r2r   1/1     Running   0          2m19s

3.临时存储的监控

容器临时存储的使用量：

• container_fs_usage_bytes

容器临时存储的申请量：

• kube_pod_container_resource_requests{resource="ephemeral_storage",unit="byte"}

节点的临时存储总量：

• kube_node_status_allocatable{resource="ephemeral_storage",unit="byte"}

节点已分配临时存储：

• sum by (node) (kube_pod_container_resource_requests{resource="ephemeral_storage"})

4.临时存储的源码

监控pod和container的临时存储使用率，若超过限制，则被evicted:

//pkg/kubelet/eviction/eviction_manager.go
func (m *managerImpl) localStorageEviction(pods []*v1.Pod, statsFunc statsFunc) []*v1.Pod {
    evicted := []*v1.Pod{}
    for _, pod := range pods {
        podStats, ok := statsFunc(pod)
        if !ok {
            continue
        }
        ...
        if m.podEphemeralStorageLimitEviction(podStats, pod) {
            evicted = append(evicted, pod)
            continue
        }
        if m.containerEphemeralStorageLimitEviction(podStats, pod) {
            evicted = append(evicted, pod)
        }
    }
    return evicted
}

重点看一下container的存储容量检查：

func (m *managerImpl) containerEphemeralStorageLimitEviction(podStats statsapi.PodStats, pod *v1.Pod) bool {
    //limits限制值
    thresholdsMap := make(map[string]*resource.Quantity)
    for _, container := range pod.Spec.Containers {
        ephemeralLimit := container.Resources.Limits.StorageEphemeral()
        if ephemeralLimit != nil && ephemeralLimit.Value() != 0 {
            thresholdsMap[container.Name] = ephemeralLimit
        }
    }
    //遍历所有的container
    for _, containerStat := range podStats.Containers {
        containerUsed := diskUsage(containerStat.Logs)
        if !*m.dedicatedImageFs {
            containerUsed.Add(*diskUsage(containerStat.Rootfs))
        }
        if ephemeralStorageThreshold, ok := thresholdsMap[containerStat.Name]; ok {
            //若limits < usage，则evicted
            if ephemeralStorageThreshold.Cmp(*containerUsed) < 0 {
                if m.evictPod(pod, 0, fmt.Sprintf(containerEphemeralStorageMessageFmt, containerStat.Name, ephemeralStorageThreshold.String()), nil) {
                    metrics.Evictions.WithLabelValues(signalEphemeralContainerFsLimit).Inc()
                    return true
                }
                return false
            }
        }
    }
    return false
}

参考：

1.https://ieevee.com/tech/2019/05/23/ephemeral-storage.html
2.https://www.bookstack.cn/read/okd-4.7-en/995d491a59697e3b.md