kubelet为cadvisor添加namespace/pod/container标签

在kubernetes中，kubelet中内嵌了cadvisor，prometheus对cadvisor指标的拉取时，使用的是kubelet的endpoints，只是metric_path改为/metrics/cadvisor。

kubelet在集成cadvisor的同时，还其添加了namespace、pod、container标签，用于定位它在kubernetes资源中的具体位置。

container_spec_cpu_period{
    container="prometheus", 
    endpoint="https-metrics", 
    id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304f6c86_d20b_45aa_95ad_db5edc2e8b43.slice/cri-containerd-cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf.scope", 
    image=“prometheus:latest", 
    instance=“192.168.0.1:10250", 
    job="kubelet", 
    metrics_path="/metrics/cadvisor", 
    name="cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf",     
    namespace="monitoring", 
    node=“node01", 
    pod="prometheus-k8s-0", 
    service="kubelet"}    100000

那么kubelet是如何为cadvisor添加这些标签的，cadvisor中有没有采集这些信息呢？

一.cadvisor的指标标签

为了验证cadvisor自身的采集能力，将cadvisor单独部署到kubernetes，配置其拉取的job=cadvisor，拉取到的指标如下：

container_cpu_usage_seconds_total{
    container_env_HOSTNAME="prometheus-k8s-0”, 
    container_env_KUBERNETES_PORT="tcp://10.233.0.1:443”, 
    container_env_KUBERNETES_PORT_443_TCP="tcp://10.233.0.1:443”, 
    container_env_KUBERNETES_PORT_443_TCP_ADDR="10.233.0.1”, 
    container_env_KUBERNETES_PORT_443_TCP_PORT="443”, 
    container_env_KUBERNETES_PORT_443_TCP_PROTO="tcp”, 
    container_env_KUBERNETES_SERVICE_HOST="10.233.0.1”, 
    …
    container_label_io_cri_containerd_kind="container”, 
    container_label_io_kubernetes_container_name="prometheus”,     // 容器名称
    container_label_io_kubernetes_pod_name="prometheus-k8s-0”,     // pod名称
    container_label_io_kubernetes_pod_namespace="monitoring”,      // namespace
    container_label_io_kubernetes_pod_uid="304f6c86-d20b-45aa-95ad-db5edc2e8b43”, 
    container_label_maintainer="The Prometheus Authors <[e-mail]>”, 
    ...
    cpu="cpu00”, 
    endpoint="http”, 
    id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304f6c86_d20b_45aa_95ad_db5edc2e8b43.slice/cri-containerd-cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf.scope”, 
    image=“prometheus:latest”, 
    instance="10.233.123.70:8080”, 
    job="cadvisor”, 
    name="cbde185ec5c5d6ff6928b10e9dc21de7548aec0e0bcbdff25a9ebf80a9fd3adf”, 
    service="cadvisor"}    709.899183253

可以看到拉取的指标中的label:

• container_env_*:

  • cadvisor采集了容器内的环境变量信息，并将其添加到指标的label中；

• container_label_io_*:

  • cadvisor采集了容器的label信息，并将其添加到指标的label中；
  • 这些label信息包含了容器的container名称、所在的pod、所在的namespace信息；

也就是说，cadvisor可以采集到容器的container/pod/namespace信息。

那这些信息是从何而来的呢，看cadvisor的代码：

• 对cadvisor采集的指标，使用containerLabelsFunc函数添加额外的标签；
• containerLabelsFunc默认=DefaultContainerLabels；

// cadvisor/metrics/prometheus.go
func NewPrometheusCollector(i infoProvider, f ContainerLabelsFunc, includedMetrics container.MetricSet, now clock.Clock, opts v2.RequestOptions) *PrometheusCollector {
    if f == nil {
        f = DefaultContainerLabels        // 默认的添加标签的函数
    }
    c := &PrometheusCollector{
        infoProvider:        i,
        containerLabelsFunc: f,
        ...
    }
    if includedMetrics.Has(container.CpuUsageMetrics) {
        c.containerMetrics = append(c.containerMetrics, []containerMetric{
            {
                name:      "container_cpu_user_seconds_total",
                help:      "Cumulative user cpu time consumed in seconds.",
                valueType: prometheus.CounterValue,
                getValues: func(s *info.ContainerStats) metricValues {
                    return metricValues{
                        {
                            value:     float64(s.Cpu.Usage.User) / float64(time.Second),
                            timestamp: s.Timestamp,
                        },
                    }
                },
            }, 
        ...
    }
    return c    
}

看一下DefaultContainerLabels函数的实现：

• 对container.spec中的label，添加了container_label_*前缀，这些label将被添加到最终的指标标签中；
• 对container.spec中的env，添加了container_label_env_*前缀，这些label将被添加到最终的指标标签中；

// cadvisor/metrics/prometheus.go
const (
   // ContainerLabelPrefix is the prefix added to all container labels.
   ContainerLabelPrefix = "container_label_"
   // ContainerEnvPrefix is the prefix added to all env variable labels.
   ContainerEnvPrefix = "container_env_"
   // LabelID is the name of the id label.
   LabelID = "id"
   // LabelName is the name of the name label.
   LabelName = "name"
   // LabelImage is the name of the image label.
   LabelImage = "image"
)

func DefaultContainerLabels(container *info.ContainerInfo) map[string]string {
   set := map[string]string{LabelID: container.Name}
   if len(container.Aliases) > 0 {
      set[LabelName] = container.Aliases[0]
   }
   if image := container.Spec.Image; len(image) > 0 {
      set[LabelImage] = image
   }
   for k, v := range container.Spec.Labels {
      set[ContainerLabelPrefix+k] = v          // container_label_*
   }
   for k, v := range container.Spec.Envs {
      set[ContainerEnvPrefix+k] = v            // container_env_*
   }
   return set
}

最后，再看一下containerInfo的spec中的label和env是什么样子：

"labels":{
    "io.kubernetes.container.name":"node-exporter",
    "io.kubernetes.pod.name":"node-exporter-gw4qg",
    "io.kubernetes.pod.namespace":"monitoring",
    "io.kubernetes.pod.uid":"2ac69f2e-cb3c-4e4f-b890-183e457ef891"
},

"env": [
  "HOSTNAME=node01",
  "KUBERNETES_SERVICE_PORT=443",
  "KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443",
  "KUBERNETES_SERVICE_HOST=10.233.0.1",
  "KUBERNETES_PORT_443_TCP_PORT=443",
  "KUBERNETES_SERVICE_PORT_HTTPS=443",
  "KUBERNETES_PORT=tcp://10.233.0.1:443",
  "KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1",
  ...
],

cadvisor对原始的label和env添加了前缀，返回给prometheus:

• 对label:

  • 原始的label=“io.kubernetes.container.name”，加上前缀，变成：container_label_io_kubernetes_container_name;

• 对env:

  • 原始的env=HOSTNAME，加上前缀，变成：container_env_HOSTNAME;

二. kubelet集成cadvisor的指标标签

kubelet在集成cadvisor时，自定义了为指标额外添加标签的逻辑containerLabelsFunc，没有使用DefaultContainerLabels。

kubelet传入的函数=containerPrometheusLabelsFunc:

// kubernetes/pkg/kubelet/server/server.go
r.RawMustRegister(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts))

containerPrometheusLabelsFunc函数的代码实现如下：

• 从containerInfo.Spec.label中，提取io.kubernetes.pod.name的值，作为pod名称的label-value；
• 从containerInfo.Spec.label中，提取io.kubernetes.pod.namespace的值，作为namespace名称的label-value；
• 从containerInfo.Spec.label中，提取io.kubernetes.container.name的值，作为container名称的label-value；

// kubernetes/pkg/kubelet/server/server.go
func containerPrometheusLabelsFunc(s stats.Provider) metrics.ContainerLabelsFunc {
   // containerPrometheusLabels maps cAdvisor labels to prometheus labels.
   return func(c *cadvisorapi.ContainerInfo) map[string]string {
      // Prometheus requires that all metrics in the same family have the same labels,
      // so we arrange to supply blank strings for missing labels
      var name, image, podName, namespace, containerName string
      if len(c.Aliases) > 0 {
         name = c.Aliases[0]
      }
      image = c.Spec.Image
      if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNameLabel]; ok {    // io.kubernetes.pod.name
         podName = v
      }
      if v, ok := c.Spec.Labels[kubelettypes.KubernetesPodNamespaceLabel]; ok {    // io.kubernetes.pod.namespace
         namespace = v
      }
      if v, ok := c.Spec.Labels[kubelettypes.KubernetesContainerNameLabel]; ok {    // io.kubernetes.container.name
         containerName = v
      }
      // Associate pod cgroup with pod so we have an accurate accounting of sandbox
      if podName == "" && namespace == "" {
         if pod, found := s.GetPodByCgroupfs(c.Name); found {
            podName = pod.Name
            namespace = pod.Namespace
         }
      }
      set := map[string]string{
         metrics.LabelID:    c.Name,
         metrics.LabelName:  name,
         metrics.LabelImage: image,
         "pod":              podName,        // pod名称
         "namespace":        namespace,        // namespace
         "container":        containerName,    // container名称
      }
      return set
   }
}

三. 总结

cadvisor通过容器的接口，查询到spec.labels的信息(包含container/pod/namespace)后，

• 对于普通的cadvisor:

  • 使用默认的DefaultContainerLabels函数添加标签，为label添加container_label_前缀；

• 对于kubelet集成的cadvisor：

  • 使用自定义的containerPrometheusLablesFunc函数添加标签，将原始的标签变为container=/pod=/namespace=；