Prometheus+grafana监控告警实现（一）

方法一、此文档描述的安装方法不适用新版本的k8s

适用的k8s版本低于1.25,当前使用的版本为1.26
https://dev.to/aws-builders/monitoring-eks-cluster-with-prometheus-and-grafana-1kpb

PodDisruptionBudgets API version policy/v1beta1 removed in Kubernetes 1.25
With the release of Kubernetes v1.25, PodSecurityPolicy has been removed

方法二、所在区域不支持从模板快速安装

Prometheus for Amazon EKS Quick Start Reference Deployment
https://aws-quickstart.github.io/quickstart-eks-prometheus/

三、使用 Prometheus 控制平面指标

https://docs.aws.amazon.com/eks/latest/userguide/prometheus.h...

# 查看原始指标
kubectl get --raw /metrics

kubectl create namespace prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade -i prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"
kubectl get pods -n prometheus

# 添加ingress对外暴露服务
[root@ip-172-31-31-169 monitor]# cat prometheus-server-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-server-ingress
  namespace: prometheus
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip  
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-server
            port:
              number: 80


[root@ip-172-31-31-169 monitor]# kubectl apply -f prometheus-server-ingress.yaml 
ingress.networking.k8s.io/prometheus-server-ingress created
[root@ip-172-31-31-169 monitor]# kubectl get ing -n prometheus
NAME                        CLASS    HOSTS   ADDRESS                                                                      PORTS   AGEprometheus-server-ingress   <none>   *       k8s-promethe-promethe-84061c92b5-1244882.me-central-1.elb.amazonaws.com   80      10s

四、设置Grafana

https://catalog.workshops.aws/running-batch-on-eks/en-US/exer...
helm install

[root@ip-172-31-31-169 monitor]# mkdir grafana
[root@ip-172-31-31-169 grafana]# cat grafana.yaml 
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

[root@ip-172-31-31-169 grafana]# kubectl create namespace grafana
namespace/grafana created

[root@ip-172-31-31-169 grafana]# helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='YourPassWord' \
    --values /opt/monitor/grafana/grafana.yaml \
    --set service.type=ClusterIP
NAME: grafana
LAST DEPLOYED: Fri Jul  7 09:07:57 2023
NAMESPACE: grafana
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.grafana.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace grafana -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace grafana port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
[root@ip-172-31-31-169 grafana]# kubectl get all -n grafana
NAME                          READY   STATUS    RESTARTS   AGE
pod/grafana-5b4d59b6f-ckrqx   1/1     Running   0          59s

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/grafana   ClusterIP   10.100.198.81   <none>        80/TCP    59s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1/1     1            1           59s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-5b4d59b6f   1         1         1       59s

添加ingress对外访问

[root@ip-172-31-31-169 grafana]# cat grafana-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: grafana
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip  
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 80
[root@ip-172-31-31-169 grafana]# kubectl apply -f grafana-ingress.yaml

导入模板
https://grafana.com/grafana/dashboards/