nginx-ingress-controller日志持久化方案

最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化， 我们设计了一个较为无侵入的方案，通过这个方案可以将nginx-ingress-controller的日志输出，不论是accesslog还是klog库输出的日志，都能进行重定向落盘和轮转、清理。 我们发现这个日志持久化方案基本可以应对绝大多数的容器应用的日志持久化需求。

注：本方案需要在原有模板中增加一些容器，相关的镜像源码和Dockerfile记录在个人Github ，欢迎使用。

nginx-ingress-controller的日志

nginx-ingress-controller的日志包括三个部分：

• controller日志： 输出到stdout，通过启动参数中的–log_dir可已配置输出到文件，重定向到文件后会自动轮转，但不会自动清理
• accesslog：输出到stdout，通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理
• errorlog：输出到stderr，配置方式与accesslog类似。

给controller日志落盘

1. 给nginx-ingress-controller挂一个hostpath： /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ，
2. 给nginx-ingress-controller配置log-dir和logtostderr参数，将日志重定向到/var/log/nginx_ingress_controller/中。

controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的，会进行日志滚动，所以我们通过脚本定时清理一定时间之前的日志文件即可。

给nginx日志落盘

1. 修改configmap： nginx-configuration。配置accesslog和errorlog的输出路径，替换默认的stdout和stderr。输出路径我们可以与controller一致，便于查找。

2. accesslog和errorlog都只有一个日志文件，我们可以使用logrotate进行日志轮转，将输出到宿主机上的日志进行轮转和清理。配置如：

   $ cat /etc/logrotate.d/nginx.log
   /data/log/nginx_ingress_controller/access.log {
       su root list
       rotate 7
       daily
       maxsize 50M
       copytruncate
       missingok
       create 0644 www-data root
   }

3. 官方提供的模板中，nginx-ingress-controller默认都是以33这个用户登录启动容器的，因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -R 33:33 /data/log/nginx_ingress_controller.

自动化ops

nginx日志落盘中，第2、3两点均需要人工运维，有什么解决办法吗？

initContainer

问题的关键是：有什么办法可以在nginx-ingress-controller容器启动之前加一个hook，将宿主机的指定目录执行chown呢？

可以用initContainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。利用这一k8s特性，我们开发一个docker image：hub.c.163.com/combk8s/adddirperm:1.0.0 ，里面只执行如下脚本：

#!/bin/bash
logdir=$LOG_DIR
userID=$USER_ID
echo "try to set dir: $logdir 's group as $userID"
chown  -R  $userID:$userID $logdir

脚本读取一些环境变量， 确认需要修改哪个目录，改成怎样的user group。

将脚本打包成dockerimage， 放在nginx-ingress-controller的deploy yaml中，作为initcontainers。 注意要对该initcontainer配置环境变量和volumeMount.

sidecar

再说第二点，我们注意到nginx-ingress-controller的基础镜像中并不包含logrotate，所以我们初步的思路是在宿主机上运行并配置logrotate。 但机器上安装和运行logrotate有需要批量运维的脚本，并且有些环境的机器可能不允许我们安装这些东西。 所以还是应该思考一个容器化的方案。

我们于是设计了一个sidecar，使用我们自己构建好的镜像：hub.c.163.com/combk8s/logrotate:v1.1 。这个镜像启动的容器中，会每个6小时执行一次logrotate，定时时间可以通过环境变量CRON_EXPR注入，例如CRON_EXPR= */3 * * * * 表示每隔三分钟执行一次。

我们将针对nginx accesslog和errorlog的日志的轮转策略保存到configmap中，并以volume的方式mount到这个sidecar容器中， 同时，这个sidecar容器也要mount nginx-ingress-controller的日志目录。

归纳

一个deploy yaml如下：

---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    protocol: TCP
  selector:
    app: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: default
  namespace: kube-system
spec:
  backend:
    serviceName: default-http-backend
    servicePort: 80
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: kube-system
  labels:
    app: ingress-nginx
data:
  use-forwarded-headers: "true"
  # 此处配置nginx日志的重定向目标
  access-log-path: /var/log/nginx_ingress_controller/access.log
  error-log-path: /var/log/nginx_ingress_controller/error.log

---

# 创建一个configmap，配置nginx日志的轮转策略，对应的是nginx日志在容器内的日志文件
apiVersion: v1
data:
  nginx.log: |
    /var/log/nginx_ingress_controller/access.log {
        rotate 5
        daily
        maxsize 10M
        minsize 10M
        copytruncate
        missingok
        create 0644 root root
    }
    /var/log/nginx_ingress_controller/error.log {
        rotate 5
        daily
        maxsize 10M
        minsize 10M
        copytruncate
        missingok
        create 0644 root root
    }
kind: ConfigMap
metadata:
  name: nginx-ingress-logrotate
  namespace: kube-system
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      tolerations:
      - key: dedicated
        value: ingress-nginx
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "system/ingress"
                operator: In
                values:
                - "true"
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true
      # 配置initcontainer，确保在nginx-ingress-controller容器启动前将日志目录的权限配置好
      initContainers:
      - name: adddirperm
        image: "hub.c.163.com/combk8s/adddirperm:1.0.0"
        env:
        - name: LOG_DIR
          value: /var/log/nginx_ingress_controller
        - name: USER_ID
           value: "33"
        volumeMounts:
        - name: logdir
          mountPath: /var/log/nginx_ingress_controller
      containers:
      # 配置sidecar容器，运行logrotate进行日志轮转
      - image: hub.c.163.com/combk8s/logrotate:v1.1
        name: logrotate
        resources:
          requests:
            cpu: 100m
        volumeMounts:
        - name: logdir
          mountPath: /var/log/nginx_ingress_controller
        - name: logrotateconf
          mountPath: /etc/logrotate.d/nginx.log
          subPath: nginx.log
      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0
        imagePullPolicy: IfNotPresent
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx
        - --annotations-prefix=nginx.ingress.kubernetes.io
        
        # 设置controller日志的输出路径和方式
        - --log_dir=/var/log/nginx_ingress_controller
        - --logtostderr=false
        securityContext:
          capabilities:
              drop:
              - ALL
              add:
              - NET_BIND_SERVICE
          # www-data -> 33
          runAsUser: 33
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        volumeMounts:
        # 配置挂载容器中控制器组件和nginx的日志输出路径
        - name: logdir
          mountPath: /var/log/nginx_ingress_controller
        # 配置nginx日志的logrotate配置挂载路径
        - name: logrotateconf
          mountPath: /etc/logrotate.d/nginx.log
          subPath: nginx.log
      volumes:
      # 控制器组件和nginx的日志输出路径为宿主机的hostpath
      - name: logdir
        hostPath:
          path: /data/log/nginx_ingress_controller
          type: ""
      # nginx日志的轮转配置文件来自于configmap
      - name: logrotateconf
        configMap:
          name: nginx-ingress-logrotate
          items:
          - key: nginx.log
            path: nginx.log
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    app: default-http-backend
spec:
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      tolerations:
      - key: dedicated
        value: ingress-nginx
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "system/ingress"
                operator: In
                values:
                - "true"
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: "{{ image_registry.addr }}/{{ image.http_backend }}"
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

最后，有的人建议将initcontainer去掉，改为基于原有的nginx-ingress-controller镜像加一层layer，将配置路径权限的脚本放在该层执行。 个人认为这种方法既不美观，也不方便。唯一的好处仅在于deploy yaml仍然简洁（但少不了volumeMount之类的配置）。不过还是看个人使用感受吧~