Prometheus+grafana监控告警实现（二）

使用helm安装grafana

https://catalog.workshops.aws/running-batch-on-eks/en-US/exer...

[root@ip-172-31-31-169 monitor]# mkdir grafana
[root@ip-172-31-31-169 grafana]# cat grafana.yaml 
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

[root@ip-172-31-31-169 grafana]# kubectl create namespace grafana
namespace/grafana created

[root@ip-172-31-31-169 grafana]# helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='www' \
    --values /opt/monitor/grafana/grafana.yaml \
    --set service.type=ClusterIP
NAME: grafana
LAST DEPLOYED: Fri Jul  7 09:07:57 2023
NAMESPACE: grafana
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.grafana.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace grafana -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace grafana port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
[root@ip-172-31-31-169 grafana]# kubectl get all -n grafana
NAME                          READY   STATUS    RESTARTS   AGE
pod/grafana-5b4d59b6f-ckrqx   1/1     Running   0          59s

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/grafana   ClusterIP   10.100.198.81   <none>        80/TCP    59s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1/1     1            1           59s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-5b4d59b6f   1         1         1       59s

添加ingress对外访问

[root@ip-172-31-31-169 grafana]# cat grafana-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: grafana
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip  
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 80
[root@ip-172-31-31-169 grafana]# kubectl apply -f grafana-ingress.yaml

常用模板

https://github.com/dotdc/grafana-dashboards-kubernetes

Dashboard    ID
K8s Resource Monitoring    17375
Loki Kubernetes Logs    15141
k8s-addons-prometheus.json    19105
k8s-addons-trivy-operator.json    16337
k8s-system-api-server.json    15761
k8s-system-coredns.json    15762
k8s-views-global.json    15757
k8s-views-namespaces.json    15758
k8s-views-nodes.json    15759
k8s-views-pods.json    15760

添加jvm 监控 （可选）

a、制作包含jmx_exporter的基础镜像

cd /opt/templates/dockerfiles
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.15.0/jmx_prometheus_javaagent-0.15.0.jar
chmod 755 jmx_prometheus_javaagent-0.15.0.jar

vim Dockerfile-oracle-jdk-jmx 写入如下内容
FROM centos:7
WORKDIR /opt/
COPY jdk1.8.0_131 /opt/jdk1.8.0_131
ENV JAVA_HOME=/opt/jdk1.8.0_131
ENV PATH=$JAVA_HOME/bin:$PATH
RUN yum install kde-l10n-Chinese -y && \
    yum install glibc-common -y && \
    localedef -c -f UTF-8 -i zh_CN zh_CN.utf8
ENV LC_ALL zh_CN.UTF-8
COPY jmx_prometheus_javaagent-0.15.0.jar /opt/jmx_prometheus_javaagent-0.15.0.jar

# build docker image
build -t x.dkr.ecr.me-central-1.amazonaws.com/public:jdk8-jmx15 -f Dockerfile-oracle-jdk-jmx .
docker push x.dkr.ecr.me-central-1.amazonaws.com/public:jdk8-jmx15

b、创建configmap

[root@ip-172-31-31-169 jmx_exporter]# cat prometheus-jmx-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-jmx-config
  namespace: dev
data:
  prometheus-jmx-config.yaml: |
    lowercaseOutputLabelNames: true
    lowercaseOutputName: true
    whitelistObjectNames: ["java.lang:type=OperatingSystem"]
    blacklistObjectNames: []
    rules:
      - pattern: 'java.lang<type=OperatingSystem><>(committed_virtual_memory|free_physical_memory|free_swap_space|total_physical_memory|total_swap_space)_size:'
        name: os_$1_bytes
        labels: {}
        type: GAUGE
        attrNameSnakeCase: true
      - pattern: 'java.lang<type=OperatingSystem><>((?!process_cpu_time)\w+):'
        name: os_$1
        labels: {}
        type: GAUGE
        attrNameSnakeCase: true

kubectl apply -f prometheus-jmx-config.yaml

c、修改git仓库charts目录下文件配置

d、修改prometheus配置

helm upgrade -i prometheus prometheus-community/prometheus  --values prometheus-overrides.yaml    --namespace prometheus     --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

e、导入grafana模板 【id: 8563】