模块三 Kubernetes监控与日志管理

1、查看集群资源状况

[root@k8s-m ~]# kubectl get cs

[root@k8s-m ~]# kubectl api-resources

[root@k8s-m ~]# kubectl cluster-info

[root@k8s-m ~]# kubectl cluster-info dump

[root@k8s-m ~]# kubectl get pod
[root@k8s-m ~]# kubectl describe pod nginx-f89759699-krv78

[root@k8s-m ~]# kubectl get pod -w

2、监控集群资源利用率

以下命令需要metrics-server组件的支持

[root@k8s-m ~]# kubectl top pod/node

• metrics-server与cAdvisor监控集群资源的资源消耗
• Metrics Server是一个集群范围的资源使用情况的数据聚合器。作为一个应用部署在集群中。Metric server从每个节点上Kubelet API收集指标，通过Kubernetes聚合器注册在Master APIServer中。

https://github.com/kubernetes-sigs/metrics-server

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

# vim components.yaml
       containers:
 85       - name: metrics-server
 86         image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
 87         imagePullPolicy: IfNotPresent
 88         args:
 89           - --cert-dir=/tmp
 90           - --secure-port=4443
 91           - --kubelet-insecure-tls
 92           - --kubelet-preferred-address-types=InternalIP

[root@k8s-m ~]# kubectl apply -f components.yaml

[root@k8s-m ~]# kubectl get pod -n kube-system

[root@k8s-m ~]# kubectl get apiservices
[root@k8s-m ~]# kubectl describe apiservices v1beta1.metrics.k8s.io

[root@k8s-m ~]# kubectl top pod
NAME                    CPU(cores)   MEMORY(bytes)
nginx-f89759699-krv78   0m           4Mi
[root@k8s-m ~]# kubectl top node
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-m    286m         7%     920Mi           24%
k8s-n1   111m         2%     387Mi           10%
k8s-n2   118m         2%     375Mi           9%
[root@k8s-m ~]# kubectl top pod -A
NAMESPACE              NAME                                         CPU(cores)   MEMORY(bytes)
default                nginx-f89759699-krv78                        0m           4Mi
kube-system            calico-kube-controllers-578894d4cd-6dw4s     4m           12Mi
kube-system            calico-node-5bmc5                            37m          63Mi
kube-system            calico-node-n2ml6                            35m          65Mi
kube-system            calico-node-rkvbx                            35m          68Mi
kube-system            coredns-7ff77c879f-2hjsd                     3m           11Mi
kube-system            coredns-7ff77c879f-dm9rj                     4m           11Mi
kube-system            etcd-k8s-m                                   18m          56Mi
kube-system            kube-apiserver-k8s-m                         47m          414Mi
kube-system            kube-controller-manager-k8s-m                20m          48Mi
kube-system            kube-proxy-7qb5k                             1m           16Mi
kube-system            kube-proxy-jmm7c                             1m           16Mi
kube-system            kube-proxy-lglqx                             1m           16Mi
kube-system            kube-scheduler-k8s-m                         4m           16Mi
kube-system            metrics-server-5667498b7d-kzw25              1m           12Mi
kubernetes-dashboard   dashboard-metrics-scraper-6b4884c9d5-bqhrr   1m           11Mi
kubernetes-dashboard   kubernetes-dashboard-7f99b75bf4-qm5ft        1m           14Mi
[root@k8s-m ~]# 

3、K8S组件日志管理

• 系统组件日志

（1）systemd守护进程管理的组件

[root@k8s-m ~]# journalctl -u kubelet.service
-- Logs begin at Tue 2020-07-28 11:16:49 CST, end at Tue 2020-07-28 11:36:37 CST. --
Jul 28 11:16:53 k8s-m systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jul 28 11:16:53 k8s-m kubelet[694]: Flag --cgroup-driver has been deprecated, This parameter should
Jul 28 11:16:53 k8s-m kubelet[694]: Flag --cgroup-driver has been deprecated, This parameter should
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.906434     694 server.go:417] Version: v1.18.0
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.907231     694 plugins.go:100] No cloud provider
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.907806     694 server.go:837] Client rotation is
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.938954     694 certificate_store.go:130] Loading
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.957102     694 server.go:646] --cgroups-per-qos e
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.957703     694 container_manager_linux.go:266] co
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.957730     694 container_manager_linux.go:271] Cr
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.958596     694 topology_manager.go:126] [topology
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.958633     694 container_manager_linux.go:301] [t
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.958640     694 container_manager_linux.go:306] Cr
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.961434     694 client.go:75] Connecting to docker
Jul 28 11:16:53 k8s-m kubelet[694]: I0728 11:16:53.961637     694 client.go:92] Start docker client
Jul 28 11:16:53 k8s-m kubelet[694]: F0728 11:16:53.962633     694 server.go:274] failed to run Kubel
Jul 28 11:16:53 k8s-m systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Jul 28 11:16:53 k8s-m systemd[1]: Unit kubelet.service entered failed state.
Jul 28 11:16:53 k8s-m systemd[1]: kubelet.service failed.
Jul 28 11:17:04 k8s-m systemd[1]: kubelet.service holdoff time over, scheduling restart.
Jul 28 11:17:04 k8s-m systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jul 28 11:17:04 k8s-m systemd[1]: Started kubelet: The Kubernetes Node Agent.
....
[root@k8s-m ~]# journalctl -u kubelet.service -f 

查看进程状态

[root@k8s-m ~]# systemctl status kubelet.service

（2）Pod部署的组件

[root@k8s-m ~]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
nginx-f89759699-krv78   1/1     Running   1          37h
[root@k8s-m ~]# kubectl logs nginx-f89759699-krv78 -n default
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[root@k8s-m ~]# kubectl logs nginx-f89759699-krv78 -f

4、K8S应用日志管理

[root@k8s-m ~]# docker info
...
Logging Driver: json-file
...

[root@k8s-m ~]# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
nginx-f89759699-krv78   1/1     Running   1          37h   10.244.111.196   k8s-n2   <none>           <none>
[root@k8s-m ~]# kubectl logs nginx-f89759699-krv78
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up

[root@k8s-n2 ~]# docker ps | grep nginx-f89759699-krv78
e95f16bd027a        nginx                                               "/docker-entrypoint.…"   37 minutes ago      Up 37 minutes                           k8s_nginx_nginx-f89759699-krv78_default_2fa7ee67-48f0-4bfe-9d2c-f06ee57a9e8b_1
df8b1e289e68        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 38 minutes ago      Up 38 minutes                           k8s_POD_nginx-f89759699-krv78_default_2fa7ee67-48f0-4bfe-9d2c-f06ee57a9e8b_1
[root@k8s-n2 ~]# cd /var/lib/docker/containers/e95f16bd027a15f2fbf434b1eb101ff0ce54104ffe29e5648c132feb20db0eba/
[root@k8s-n2 e95f16bd027a15f2fbf434b1eb101ff0ce54104ffe29e5648c132feb20db0eba]# ll
total 16
drwx------ 2 root root    6 Jul 28 11:18 checkpoints
-rw------- 1 root root 5167 Jul 28 11:18 config.v2.json
-rw-r----- 1 root root 1073 Jul 28 11:18 e95f16bd027a15f2fbf434b1eb101ff0ce54104ffe29e5648c132feb20db0eba-json.log
-rw-r--r-- 1 root root 1995 Jul 28 11:18 hostconfig.json
drwx------ 2 root root    6 Jul 28 11:18 mounts
[root@k8s-n2 e95f16bd027a15f2fbf434b1eb101ff0ce54104ffe29e5648c132feb20db0eba]# cat e95f16bd027a15f2fbf434b1eb101ff0ce54104ffe29e5648c132feb20db0eba-json.log
{"log":"/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration\n","stream":"stdout","time":"2020-07-28T03:18:35.110164485Z"}
{"log":"/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/\n","stream":"stdout","time":"2020-07-28T03:18:35.110182952Z"}
{"log":"/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh\n","stream":"stdout","time":"2020-07-28T03:18:35.116430381Z"}
{"log":"10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf\n","stream":"stdout","time":"2020-07-28T03:18:35.127327527Z"}
{"log":"10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf\n","stream":"stdout","time":"2020-07-28T03:18:35.141154733Z"}
{"log":"/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh\n","stream":"stdout","time":"2020-07-28T03:18:35.141364541Z"}
{"log":"/docker-entrypoint.sh: Configuration complete; ready for start up\n","stream":"stdout","time":"2020-07-28T03:18:35.145517322Z"}

[root@k8s-m ~]# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
nginx-f89759699-krv78   1/1     Running   1          37h
[root@k8s-m ~]# kubectl exec -it nginx-f89759699-krv78 bash
root@nginx-f89759699-krv78:/# ls /var/log/
apt/      btmp      dpkg.log  faillog   lastlog   nginx/    wtmp

• 将日志文件通过数据卷挂载到宿主机目录，这样一来，就可以在宿主机上统一查看日志
• 编写YAML文件

# vim pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: web
    image: yejianfeng/nginx-php
    volumeMounts:
    - name: logs
      mountPath: /usr/local/nginx/logs
  volumes:
  - name: logs
    hostPath:
      path: /tmp/logs
      type: Directory

[root@k8s-m ~]# kubectl apply -f pod.yaml

[root@k8s-m ~]# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
my-pod                  1/1     Running   0          4m41s   10.244.111.199   k8s-n2   <none>           <none>
nginx-f89759699-krv78   1/1     Running   1          37h     10.244.111.196   k8s-n2   <none>           <none>

[root@k8s-m ~]# kubectl get pod -o wide                                                                                      NAME                    READY   STATUS    RESTARTS   AGE    IP               NODE     NOMINATED NODE   READINESS GATES
my-pod                  1/1     Running   0          8m6s   10.244.111.199   k8s-n2   <none>           <none>
nginx-f89759699-krv78   1/1     Running   1          37h    10.244.111.196   k8s-n2   <none>           <none>
[root@k8s-m ~]# curl http://10.244.111.199
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
[root@k8s-m ~]# curl http://10.244.111.199/status.html
ok

[root@k8s-n2 ~]# cd /tmp/logs/
[root@k8s-n2 logs]# ls
access.log  error.log
[root@k8s-n2 logs]# tail -f access.log
10.244.29.0 - - [28/Jul/2020:12:25:02 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0"

注意

[root@k8s-m ~]# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
my-pod                  1/1     Running   0          10m   10.244.111.199   k8s-n2   <none>           <none>
nginx-f89759699-krv78   1/1     Running   1          37h   10.244.111.196   k8s-n2   <none>           <none>
[root@k8s-m ~]# kubectl exec -it nginx-f89759699-krv78 bash
root@nginx-f89759699-krv78:/# cd /var/log/nginx/
root@nginx-f89759699-krv78:/var/log/nginx# ls
access.log  error.log
root@nginx-f89759699-krv78:/var/log/nginx# ls -l
total 0
lrwxrwxrwx 1 root root 11 Jul 22 03:23 access.log -> /dev/stdout
lrwxrwxrwx 1 root root 11 Jul 22 03:23 error.log -> /dev/stderr

vim pod2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod2
spec:
  containers:
  - name: web
    image: lizhenliang/nginx-php
    volumeMounts:
    - name: logs
      mountPath: /usr/local/nginx/logs
  volumes:
  - name: logs
    emptyDir: {}

[root@k8s-m ~]# kubectl apply -f pod2.yaml
[root@k8s-m ~]# kubectl get pod -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
my-pod                  1/1     Running   0          30m   10.244.111.199   k8s-n2   <none>           <none>
my-pod2                 1/1     Running   0          33s   10.244.111.200   k8s-n2   <none>           <none>
nginx-f89759699-krv78   1/1     Running   1          38h   10.244.111.196   k8s-n2   <none>           <none>

[root@k8s-n2 ~]# cd /var/lib/kubelet/pods/
[root@k8s-n2 pods]# ls
2fa7ee67-48f0-4bfe-9d2c-f06ee57a9e8b  9a208280-1a9f-43be-b7f6-c071e47b0ff4  cff5dd19-0fb2-4f58-9296-20fb3eccbbb5
44d83b61-928e-423d-8d03-b3c34dd80478  9c3174bf-ae42-4882-9283-33d540d34e22
7c532b51-c1f0-4ac7-8362-06f0765fbe0c  c18e5cb2-8861-4b2c-b788-18516d9a6f10
[root@k8s-n2 pods]# docker ps | grep my-pod2
37a3c4852932        yejianfeng/nginx-php                               "docker-entrypoint.s…"   4 minutes ago       Up 4 minutes                            k8s_web_my-pod2_default_c18e5cb2-8861-4b2c-b788-18516d9a6f10_0
baa2f259a50b        registry.aliyuncs.com/google_containers/pause:3.2   "/pause"                 4 minutes ago       Up 4 minutes                            k8s_POD_my-pod2_default_c18e5cb2-8861-4b2c-b788-18516d9a6f10_0

[root@k8s-n2 pods]# ls
2fa7ee67-48f0-4bfe-9d2c-f06ee57a9e8b  9a208280-1a9f-43be-b7f6-c071e47b0ff4  cff5dd19-0fb2-4f58-9296-20fb3eccbbb5
44d83b61-928e-423d-8d03-b3c34dd80478  9c3174bf-ae42-4882-9283-33d540d34e22
7c532b51-c1f0-4ac7-8362-06f0765fbe0c  c18e5cb2-8861-4b2c-b788-18516d9a6f10
[root@k8s-n2 pods]# cd c18e5cb2-8861-4b2c-b788-18516d9a6f10/
[root@k8s-n2 c18e5cb2-8861-4b2c-b788-18516d9a6f10]# ls
containers  etc-hosts  plugins  volumes
[root@k8s-n2 c18e5cb2-8861-4b2c-b788-18516d9a6f10]# cd volumes/kubernetes.io~
-bash: cd: volumes/kubernetes.io~: No such file or directory
[root@k8s-n2 c18e5cb2-8861-4b2c-b788-18516d9a6f10]# cd volumes/
[root@k8s-n2 volumes]# ls
kubernetes.io~empty-dir  kubernetes.io~secret
[root@k8s-n2 volumes]# cd kubernetes.io~empty-dir/
[root@k8s-n2 kubernetes.io~empty-dir]# ls
logs
[root@k8s-n2 kubernetes.io~empty-dir]# cat logs/
cat: logs/: Is a directory
[root@k8s-n2 kubernetes.io~empty-dir]# cd logs/
[root@k8s-n2 logs]# ls
access.log  error.log
[root@k8s-n2 logs]# tail -f access.log
10.244.29.0 - - [28/Jul/2020:12:53:15 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0"