目标:
用户启动的service或pod,在annotation中添加label后,可以自动被prometheus发现:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9121"
1. secret保存自动发现的配置
若要特定的annotation被发现,需要为prometheus增加如下配置:
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
上述配置会筛选endpoints:prometheus.io/scrape=True
将上述配置保存为secret:
$ kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
secret "additional-configs" created
2. 将配置添加到prometheus实例
修改prometheus CRD,将上面的secret添加进去:
# vi /etc/kubernetes/prometheus/prometheus-prometheus.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
......
additionalScrapeConfigs:
name: additional-configs
key: prometheus-additional.yaml
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.5.0
# kubectl apply -f prometheus-prometheus.yaml
prometheus CRD修改完毕,可以到prometheus dashboard查看config是否被修改。
3. prometheus实例增加clusterrole
添加了上述配置后,prometheus-k8s-0的log会发现很多的forbidden,这是因为其没有service/pod的list权限。老的权限:
# cat /etc/kubernetes/prometheus/prometheus-clusterRole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-k8s
rules:
- apiGroups:
- ""
resources:
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
需要修改其clusterRole,增加权限,新的权限:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-k8s
rules:
- apiGroups:
- ""
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
执行:kubectl apply -f prometheus-clusterRole.yaml进行更新。
参考:
1.Prometheus Operator高级配置:https://www.qikqiak.com/post/...
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用
。你还可以使用@
来通知其他用户。