prometheus-operator使用(七) -- 3个alertmanager实例

prometheus通过alertmanager这个CRD来管理和部署alertmanager实例，默认是部署3个实例，3个实例组成集群，通过Gossip协议保持一致性。

由于部署了headless service，每个alertmanager实例都有1个唯一的标识。

1. 3个alertmanager实例

alertmanager使用statefulset部署，部署了3个pod:

# kubectl get all -n monitoring |grep alertmanager-main
pod/alertmanager-main-0                    2/2     Running   0          19d
pod/alertmanager-main-1                    2/2     Running   0          19d
pod/alertmanager-main-2                    2/2     Running   1          18d
service/alertmanager-main       ClusterIP   10.233.5.73     <none>        9093/TCP                     19d
statefulset.apps/alertmanager-main   3/3     19d

3个alertmanager部署为集群，实例之间通过Gossip协议实现一致性：

# ps -ef|grep alertmanager
1000     51237 51205  0 Feb01 ?        01:34:55 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --storage.path=/alertmanager --data.retention=120h --cluster.listen-address=[10.233.97.7]:9094 --web.listen-address=:9093 --web.route-prefix=/ --cluster.peer=alertmanager-main-0.alertmanager-operated:9094 --cluster.peer=alertmanager-main-1.alertmanager-operated:9094 --cluster.peer=alertmanager-main-2.alertmanager-operated:9094

定义了一个headless service，让每个alertmanager实例有1个唯一的标识：

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-operated
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: web
    port: 9093
    protocol: TCP
    targetPort: web
  - name: tcp-mesh
    port: 9094
    protocol: TCP
    targetPort: 9094
  - name: udp-mesh
    port: 9094
    protocol: UDP
    targetPort: 9094
  selector:
    app: alertmanager
  sessionAffinity: None
  type: ClusterIP

2. prometheus中配置访问alertmanager

到prometheus POD上看一下：

# kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring

/etc/prometheus/config_out $ vi prometheus.env.yaml
......
alerting:
  alert_relabel_configs:
  - action: labeldrop
    regex: prometheus_replica
  alertmanagers:
  - path_prefix: /
    scheme: http
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - monitoring
    relabel_configs:
    - action: keep
      source_labels:
      - __meta_kubernetes_service_name
      regex: alertmanager-main
    - action: keep
      source_labels:
      - __meta_kubernetes_endpoint_port_name
      regex: web
.......

可以看到，alertManager的实例也是通过kubernetes_sd_configs来自动发现的，其筛选规则：

• service_name=alertmanager-main;
• endpoint_port_name=web；

看一下alertmanager-service.yaml，满足这一条件：

# cat alertmanager-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    alertmanager: main
  name: alertmanager-main
  namespace: monitoring
spec:
  ports:
  - name: web
    port: 9093
    targetPort: web
  selector:
    alertmanager: main
    app: alertmanager
  sessionAffinity: ClientIP

3. alertManager的CRD、service和serviceMonitor的定义

alertmanager的statefulset实际是通过alertmanager这个CRD来实现的:

# kubectl get crd -n monitoring
NAME                                                  CREATED AT
alertmanagers.monitoring.coreos.com                   2021-02-01T03:13:36Z
......

# cat alertmanager-alertmanager.yaml

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  labels:
    alertmanager: main
  name: main
  namespace: monitoring
spec:
  image: "178.104.162.39:443/dev/huayun/amd64/alertmanager:v0.21.0"
  nodeSelector:
    kubernetes.io/os: linux
  replicas: 3
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: alertmanager-main
  version: v0.21.0

可以看到，上面定义的replica=3。

service和serviceMonitor的定义如下：

# cat alertmanager-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    alertmanager: main
  name: alertmanager-main
  namespace: monitoring
spec:
  ports:
  - name: web
    port: 9093
    targetPort: web
  selector:
    alertmanager: main
    app: alertmanager
  sessionAffinity: ClientIP


# cat alertmanager-serviceMonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: alertmanager
  name: alertmanager
  namespace: monitoring
spec:
  endpoints:
  - interval: 30s
    port: web
  selector:
    matchLabels:
      alertmanager: main

定义了serviceMonitor，则alertmanager实例本身暴露的/metrics，是可以被Prometheus拉取的。