Safeguard your K8s cluster with KubeEye

Author: KaliArch (Xue Lei), product manager of a Cloud MSP service provider, familiar with enterprise-level high availability/high concurrency architecture, including hybrid cloud architecture, remote disaster, proficient in enterprise DevOPS transformation and optimization, familiar with Shell/Python/Go and other development languages , familiar with Kubernetes, Docker, cloud native, microservice architecture, etc.

foreword

KubeEye is a Kubernetes security and configuration problem detection tool. It uses OPA for configuration detection of business applications deployed in K8s clusters, and Node-Problem-Detector for cluster-deployed Nodes. Predefined rules for common scenarios, and user-defined rules are also supported for cluster detection.

Architecture

KubeEye obtains cluster diagnostic data by calling the Kubernetes API and matching the keywords in the resource with the rules of the container syntax. For details, see the architecture diagram.

The detection of Node nodes needs to be installed on the detected Node host.

Features

characteristic

• KubeEye reviews your workload YAML specifications against industry best practices to help you stabilize your cluster.
• KubeEye can find problems with your cluster control plane, including kube-apiserver/kube-controller-manager/etcd, etc.
• KubeEye can help you detect various node issues, including memory/CPU/disk pressure, unexpected kernel error logs, etc.

Check items

whether	Check items	describe	level
✅	PrivilegeEscalationAllowed	Allow Privilege Escalation	urgent
✅	CanImpersonateUser	role/clusterrole has permissions to pretend to be other users	warn
✅	CanDeleteResources	role/clusterrole has permission to delete Kubernetes resources	warn
✅	CanModifyWorkloads	role/clusterrole has permission to modify Kubernetes resources	warn
✅	NoCPULimits	The resource does not have a CPU usage limit set	urgent
✅	NoCPURequests	The resource is not set to reserve CPU	urgent
✅	HighRiskCapabilities	High-risk functions are enabled, such as ALL/SYS_ADMIN/NET_ADMIN	urgent
✅	HostIPPCAllowed	Host IPC enabled	urgent
✅	HostNetworkAllowed	host network enabled	urgent
✅	HostPIDAllowed	host PID enabled	urgent
✅	HostPortAllowed	host port opened	urgent
✅	ImagePullPolicyNotAlways	Image pull strategy is not always	warn
✅	ImageTagIsLatest	The mirror tag is latest	warn
✅	ImageTagMiss	Mirror has no tags	urgent
✅	InsecureCapabilities	Unsafe features are turned on, such as KILL/SYS_CHROOT/CHOWN	warn
✅	NoLivenessProbe	No liveness check is set	warn
✅	NoMemoryLimits	The resource does not have a memory usage limit set	urgent
✅	NoMemoryRequests	The resource does not have reserved memory set	urgent
✅	NoPriorityClassName	No resource scheduling priority is set	Notice
✅	PrivilegedAllowed	Run the resource in privileged mode	urgent
✅	NoReadinessProbe	No readiness check is set	warn
✅	NotReadOnlyRootFilesystem	The root filesystem is not set to read-only	warn
✅	NotRunAsNonRoot	No setting prohibits starting processes as root	warn
✅	CertificateExpiredPeriod	Will check that the expiration date of the API Server certificate is less than 30 days	urgent
✅	EventAudit	event check	warn
✅	NodeStatus	Node Status Check	warn
✅	DockerStatus	Docker status check	warn
✅	KubeletStatus	Kubelet Status Check	warn

deploy

KubeEye itself is written in Golang, and the related components can be installed using the compiled binary executable.

Install

binary installation

 wget https://github.com/kubesphere/kubeeye/releases/download/v0.3.0/kubeeye-0.3.0-linux-amd64.tar.gz
tar -zxvf kubeeye-0.3.0-linux-amd64.tar.gz
mv kubeeye /usr/bin/

Source code compilation and installation

 git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye 
make installke

Install NPD

For the detection of clustered Node hosts, kubeEye uses Node-problem-Detector , which needs to be installed on the Node host node. KubeEye encapsulates the installation command, which can be installed with one click.

⚠️ NOTE: This will install NPD on your cluster and is only needed if you want detailed node reports.

 [root@VM-48-7-centos ~]# kubeeye install -e npd
kube-system      ConfigMap      node-problem-detector-config      created
kube-system      DaemonSet      node-problem-detector      created

It mainly creates the ConfigMap of node-problem-detector-config and the node-problem-detector DaemonSet in the kube-system namespace.

Running KubeEye in a cluster

In addition to the one-time use of kubeEye tools, kubeEye is also an operator that can run inside the cluster for long-term continuous cluster detection.

Deploy KubeEye in Kubernetes

 kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml

View KubeEye inspection results

 $ kubectl get clusterinsight -o yaml

apiVersion: v1
items:
- apiVersion: kubeeye.kubesphere.io/v1alpha1
  kind: ClusterInsight
  metadata:
    name: clusterinsight-sample
    namespace: default
  spec:
    auditPeriod: 24h
  status:
    auditResults:
      auditResults:
      - resourcesType: Node
        resultInfos:
        - namespace: ""
          resourceInfos:
          - items:
            - level: waring
              message: KubeletHasNoSufficientMemory
              reason: kubelet has no sufficient memory available
            - level: waring
              message: KubeletHasNoSufficientPID
              reason: kubelet has no sufficient PID available
            - level: waring
              message: KubeletHasDiskPressure
              reason: kubelet has disk pressure
            name: kubeeyeNode

test

command options

 [root@VM-48-7-centos ~]# kubeeye -h
KubeEye finds various problems on Kubernetes cluster.

Usage:
  ke [command]

Available Commands:
  audit       audit resources from the cluster
  completion  generate the autocompletion script for the specified shell
  help        Help about any command
  install     A brief description of your command
  uninstall   A brief description of your command

Flags:
  -f, --config string         Specify the path of kubeconfig.
  -h, --help                  help for ke
      --kubeconfig string     Paths to a kubeconfig. Only required if out-of-cluster.
      --master --kubeconfig   (Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.

It can be seen that KubeEye currently mainly supports two commands, one is to install package such as NPD, and the other is to perform audit to scan the configuration of cluster applications.

audit

 [root@VM-48-7-centos ~]# kubeeye audit
KIND         NAMESPACE         NAME                                           MESSAGE
Deployment   dddd              jenkins-1644220286                             [NoCPULimits ImagePullPolicyNotAlways NoMemoryLimits NoPriorityClassName NotReadOnlyRootFilesystem NotRunAsNonRoot]
Deployment   jenkins           jenkins-1644220286                             [NoCPULimits ImagePullPolicyNotAlways NoMemoryLimits NoPriorityClassName NotReadOnlyRootFilesystem NotRunAsNonRoot]
Deployment   smartkm-api-k8s   velero                                         [ImageTagIsLatest NoLivenessProbe NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]
DaemonSet    smartkm-api-k8s   restic                                         [ImageTagIsLatest NoLivenessProbe NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]
Node                           minikube                                       [KernelHasNoDeadlock FilesystemIsNotReadOnly KubeletHasSufficientMemory KubeletHasNoDiskPressure KubeletHasSufficientPID]
Event        kube-system       node-problem-detector-dmsws.16d844532f662318   [Failed to pull image "k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7": rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
Event        kube-system       node-problem-detector-dmsws.16d844532f66703e   [Error: ErrImagePull]
Event        kube-system       node-problem-detector-dmsws.16d84453351b8b19   [Error: ImagePullBackOff]

Add custom inspection rules

We use the command to view the predefined OPA inspection rules.

 kubectl get cm -n kube-system node-problem-detector-config -oyaml

At the same time, you can also create custom inspection rules according to your own business.

• Create OPA rule storage directory

 mkdir opa

• Add custom OPA rules file

Note: The package name must be the OPA rule set by kubeeye_workloads_rego for checking RBAC, the package name must be the OPA rule set by kubeeye_RBAC_rego for checking nodes, and the package name must be kubeeye_nodes_rego

• The following are the rules for checking the address of the mirror warehouse, save the following rules to the rule file imageRegistryRule.rego

 package kubeeye_workloads_rego

deny[msg] {
    resource := input
    type := resource.Object.kind
    resourcename := resource.Object.metadata.name
    resourcenamespace := resource.Object.metadata.namespace
    workloadsType := {"Deployment","ReplicaSet","DaemonSet","StatefulSet","Job"}
    workloadsType[type]

    not workloadsImageRegistryRule(resource)

    msg := {
        "Name": sprintf("%v", [resourcename]),
        "Namespace": sprintf("%v", [resourcenamespace]),
        "Type": sprintf("%v", [type]),
        "Message": "ImageRegistryNotmyregistry"
    }
}

workloadsImageRegistryRule(resource) {
    regex.match("^myregistry.public.kubesphere/basic/.+", resource.Object.spec.template.spec.containers[_].image)
}

• Run KubeEye with extra rules

Tip: KubeEye will read all files ending in .rego in the specified directory

 kubeeye audit -p ./opa

Troubleshoot

• NPD installation is abnormal, k8s.gcr.io is used by default, if the installation server cannot connect to the public network, you can use my mirror warehouse: 1832990/node-problem-detector:v0.8.7.
• The kubEye installation uses the default host $HOME/.kube/config file. If the K8s config file does not exist, it will not work properly.

Reference link

• https://github.com/kubesphere/kubeeye/

  This article is published by OpenWrite , a multi-post blog platform!