Summary of issues that need attention when Kubernetes integrates KubeEdge

Author: Zhu Han

Recently, friends have a lot of questions about using KubeSphere v3.1 to integrate edge nodes. Here is the guide document address, and then you can summarize the problems in this area here to facilitate follow-up partners to troubleshoot problems, and you are welcome to continue to add .

Official guide document portal

• activate kubeedge
• edge node join

1. IP and port opening issues

If you use ks-installer to install, you need to activate KubeEdge and configure the external IP address corresponding to the master node.

Assuming that the IP of the cluster master node is 192.168.10.7, the external network ports that need to be opened are as follows:

serial number	Intranet IP	Intranet port (NodePort)	External network port
1	192.168.10.7	30000	10000	https protocol port
2	192.168.10.7	30001	10001	Quic protocol port
3	192.168.10.7	30002	10002	Cloudhub obtains a certificate for the first time token
4	192.168.10.7	30003	10003	cloudstream port
5	192.168.10.7	30004	10004	tunnel port (edgestream connection)

The external network port needs to pass through the firewall.

If you have forgotten to set the Internet access IP of the KubeEdge component in ks-installer, cloudhub cannot get up, you can use the following command to remedy:

 kubectl -n kubeedge edit cm cloudcore

2. Obtain edge node logs and metrics

There are many cases where the edge node and the master node are not in the same local area network, so we designed to use the external network port communication above. In addition, because KubeEdge completes the functions of kubectl exec to obtain logs and metrics, it relies on the virtual IP to forward the corresponding iptable to the corresponding port of cloudhub, and then obtains the edge data, which is realized through edgemesh, so the virtual IP bound to the edge node and The edge node name must be unique and must be maintained according to the law. Note that this virtual IP cannot be filled with the edge node internal network IP. It is best to choose the network terminal that does not conflict with the internal network. At the same time, must ensure that the metrics-server component is turned on Status , need to be updated to 0.4.1 or above to adapt to KubeEdge (the current version mainly obtains edge metrics through metrics-server).

3. If the edge node uses the external network IP and port communication, some daemonsets are more tolerant, such as Calico, you need to patch them to avoid scheduling to the edge

#!/bin/bash

NodeSelectorPatchJson='{"spec":{"template":{"spec":{"nodeSelector":{"node-role.kubernetes.io/master": "","node-role.kubernetes.io/worker": ""}}}}}'
NoShedulePatchJson='{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/edge","operator":"DoesNotExist"}]}]}}}}}}}'

edgenode="edgenode"
if [ $1 ]; then
        edgenode="$1"
fi

namespaces=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $1}' ))
pods=($(kubectl get pods -A -o wide |egrep -i $edgenode | awk '{print $2}' ))
length=${#namespaces[@]}

for((i=0;i<$length;i++));  
do
        ns=${namespaces[$i]}
        pod=${pods[$i]}
        resources=$(kubectl -n $ns describe pod $pod | grep "Controlled By" |awk '{print $3}')
        echo "Patching for ns: $ns, resources: $resources"
        kubectl -n $ns patch $resources --type merge --patch "$NoShedulePatchJson"
        sleep 1
done

4. kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

Reason: the cgroup used by kubelet and master is inconsistent

Solution:

Add the configuration "exec-opts": ["native.cgroupdriver=systemd"], then restart Docker, systemctl daemon-reload &&
systemctl restart kubelet

Reference description:

• kubeedge/kubeedge#1772 (comment)
  cgroup driver shouldn't be taken care by the keadm, I think this is also true for "kubeadm", all that care about the cgroup driver is actually which cri you are using, for example, in your case, you are using docker, so you just need to update the docker configuration to use "systemd" instead.

5. When the edge node and Kubernetes are in a local area network, how to join KubeSphere to be managed?

If the edge node and the Kubernetes cluster are in a local area network, you can use the nodePort method to join the edge node. The default nodeport port is 30000-30004, so when the edge node joins the cluster, the external port 10000-10004 must be changed to 30000-30004. It is to change the --cloudcore-ipport=192.168.x.x:10000 --quicport 10001 --certport 10002 --tunnelport 10004 to --cloudcore-ipport=192.168.x.x:30000 --quicport 30001 --certport 30002 --tunnelport 30004 , but it needs to be pointed out that the design is based on the external network. Please clarify the application scenarios of the edge nodes.

For more information, please refer to the guide.

6. The minimum Docker version supported by edge node pod metrics

The current Docker version supported by the edge should be greater than or equal to v19.3.0. For specific reasons, please refer to https://kubesphere.com.cn/forum/d/4494-kubesphere-v310/8

KubeEdge v1.6.2 has been released, and the bug has been fixed in the current test, as shown in the figure:

The edge end needs to be upgraded to v1.6.2, or you can add the following configuration to modify:

apiVersion: v1
kind: ConfigMap
metadata:
  name: edge-watcher-config
  namespace: kubeedge
data:
  version: v1.6.2
  region: zh

7. Node metrics abnormal display exclusion guide

• Check whether the metrics-server service is normal
  Check whether the metrics-server deployment is normal, whether the version meets the requirements (v0.4.1 or above), check whether the kubectl top is abnormal, if it is abnormal, you can restart it, or install it manually:

  kubectl apply  -f       https://raw.githubusercontent.com/kubesphere/ks-installer/master/roles/metrics-server/files/metrics-server/metrics-server.yaml

• Other nodes have metrics, only edge nodes do not have metrics

  • To check whether iptables has established the correct forwarding, use the following command to see which node the iptables pod is deployed on:

  kubectl get pods -n kubeedge -o wide

• Obtain the 10350 nat forwarding rule on the corresponding node, and confirm that the forwarding destination is on k8s master or node ip

• If the forwarding destination is wrong, you can use the following approach:

  kubectl -n kubeedge edit cm edge-watcher

After modification, you need to restart edge-watcher-controller-manager deployment

8. Certificate Issue

kubectl delete secret casecret cloudcoresecret -n kubeedge

Need to restart cloudcore.

This article is published by the blog one article multi-posting OpenWrite