Use edgeadm to install edge K8s cluster and native K8s cluster with one click

background

At present, many edge computing container open source projects have a default premise in use: users need to prepare a standard or specific tool-built Kubernetes cluster in advance, and then deploy corresponding components in the cluster through specific tools or other methods to experience the edge ability. This undoubtedly raises the threshold of user experience edge capabilities, and there are many restrictions on use, making it difficult for users to get started. Simply organize, there will probably be the following problems:

The threshold is too high

Users need to prepare a Kubernetes cluster in advance. For many users, the threshold is too high, the construction process is complicated and easy to fail, and many people who want to use edge capabilities are shut out;

Too restrictive

Often require specific tools to build a specific version of the Kubernetes cluster, which is too versatility, and users want to be too restrictive in the actual production environment;

Adding edge nodes is more troublesome

Adding edge nodes needs to rely on the tools to build the Kubernetes cluster itself to add Kubernetes native nodes and then convert, which is highly dependent on third-party tools, and the operation process is more troublesome and error-prone;

Poor automation

Regardless of the establishment of a Kubernetes cluster or the addition of edge nodes, it is difficult to automate in the production environment. The related processes also require secondary development by their own team, which is difficult to integrate;

In order to lower the threshold of user experience edge capabilities, students in the cloud native community plan to develop a method that can deploy edge Kubernetes clusters with one click, so that users can experience edge Kubernetes clusters more easily and simply.

Architecture design

In response to the above problems, in order to lower the threshold for users to use the edge Kubernetes cluster and make the edge Kubernetes cluster capable of production, we designed a one-click solution to deploy an edge Kubernetes cluster, completely shielding the installation details, so that users can have a zero-threshold experience Edge capacity.

Original intention

Let users use edge Kubernetes clusters easily and without barriers, and can truly use edge capabilities in the production environment;

the goal

One-click use

Able to build an edge Kubernetes cluster with one click;
It is very simple and flexible to add edge nodes with one key;

two installations Chuangjing

Support online installation;
Support offline installation, so that the privatization environment can also be very simple;

can be produced and used

Don't encapsulate too much, so that the team who wants to use the edge Kubernetes cluster can perform simple integration in the internal system, and then it is available for production;

Zero learning cost

Try to be consistent with kubeadm's usage as much as possible, so that users will use kubeadm and edgeadm without additional learning costs;

in principle

Don't modify kubeadm source code
- Try to quote and reuse the source code of kubeadm, and try not to modify the source code of kubeadm to avoid hidden dangers of subsequent upgrades;
- Based on kubeadm but higher than kubeadm, it does not have to be limited by the design of kubeadm, it can be allowed as long as it can make it easier for users to use;
Allow users to choose whether to deploy edge capability components;
Allow users to customize the configuration of edge capability components;

Design and implementation

We studied the source code of kubeadm and found that we can borrow kubeadm to create native Kubernetes clusters, join nodes, and workflow ideas to deploy edge Kubernetes clusters with one click, and perform the installation steps step by step. This is exactly what we want for a simple, flexible, and low learning cost deployment solution. So we stood on the shoulders of giants, used Kubedam's ideas, reused the source code of kubeadm, and designed the following solution.

The kubeadm init cluster/join node part completely reuses the source code of kubadm, and all logic is exactly the same as kubeadm.

This program has the following advantages:

Fully compatible with kubeadm

We just stand on the shoulders of kubeadm, set some configuration parameters required by edge clusters before kubeadm init/join, initialize the Master or Node nodes automatically, and install the container runtime. After the completion of kubeadm init/join, the CNI network plug-in was installed and the corresponding edge capability components were deployed.

We quoted the source code of kubeadm in Go Mod mode. During the whole process, we did not modify the source code of kubeadm one line. It is completely native and ready to upgrade to a higher version of kubeadm in the future.

One-click, easy to use, flexible and automated

The edgeadm init cluster and join nodes completely retain the original parameters and procedures of kubeadm init/join, but automatically initialize the node and install the container when running, you can use the edgeadm --enable-edge=fasle parameter to install the native Kubernetes cluster in one click, or you can use the edgeadm --enable-edge=true parameter One-click installation of edge Kubernetes clusters.

You can join any node as long as you can access the node where the Kube-apiserver is located, or you can join the master. Join master also continues the kubeadm approach. To build highly available nodes, you can directly use join master to expand Master nodes when needed to achieve high availability.

No learning cost, exactly the same as using kubeadm

Because the kubeadm init cluster/join node part completely reuses the source code of kubadm, all logic is exactly the same as kubeadm, and the usage habits of kubeadm and all flag parameters are completely retained. The usage is exactly the same as that of kubeadm. There is no new learning cost. Users can press kubeadm parameters. Or use kubeadm.config to customize the edge Kubernetes cluster.

Edge node security enhancement

With the help of Kubernetes Node authentication mechanism, we have enabled NodeRestriction access plug-in by default to ensure that each node has a unique identity and only has a minimum set of permissions. Even if an edge node is compromised, other edge nodes cannot be operated.

Kubelet We also enabled by default Kubelet configure certificates rotation mechanism, when Kubelet certificate is about to expire, will automatically generate a new secret key, and apply for a new certificate from Kubernetes API. Once the new certificate is available, it will be used to authenticate the connection with the Kubernetes API.

Install edge Kubernetes cluster with edgeadm

The following process communities have recorded detailed tutorial videos, which can be installed in conjunction with the documentation:

Video resource link: Use edgeadm to install edge Kubernetes clusters and native Kubernetes clusters with one click

Installation conditions

Follow kubeadm minimum requirements , the Node Master && lowest 2C2G, disk space is not less than 1G;
Currently supports amd64 and arm64 two systems;
Other systems can compile edgeadm and make corresponding system installation packages, please refer to 5. Customize Kubernetes static installation package
Supported Kubernetes version: greater than or equal to v1.18, the provided installation package only provides Kubernetes v1.18.2 version;
For other Kubernetes versions, please refer to 5. Customize the Kubernetes static installation package and make it yourself.

Download the edgeadm static installation package

Download the edgeadm static installation package and copy it to all Master and Node nodes.

Pay attention to modify the "arch=amd64" parameter, currently supports [amd64, amd64], download the architecture corresponding to your own machine, and other parameters remain unchanged

arch=amd64 version=v0.3.0-beta.0 && rm -rf edgeadm-linux-* && wget https://superedge-1253687700.cos.ap-guangzhou.myqcloud.com/$version/$arch/edgeadm-linux-$arch-$version.tgz && tar -xzvf edgeadm-linux-* && cd edgeadm-linux-$arch-$version && ./edgeadm

The installation package is about 200M. For detailed information about the installation package, please 5. Customize the Kubernetes static installation package .

If downloading the installation package is slow, you can directly view the corresponding SuperEdge version , download edgeadm-linux-amd64/arm64-*.0.tgz , and decompress the same.
One-click installation of the edge independent Kubernetes cluster function is supported starting from SuperEdge-v0.3.0-beta.0, pay attention to download v0.3.0-beta.0 and later versions.

Install edge Kubernetes master node

./edgeadm init --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --install-pkg-path ./kube-linux-*.tar.gz --apiserver-cert-extra-sans=<Master节点外网IP> --apiserver-advertise-address=<Master节点内网IP> --enable-edge=true -v=6

among them:

--enable-edge=true: Whether to deploy edge capability components, the default is true

--enable-edge=false means to install a native Kubernetes cluster, which is exactly the same as the cluster built by kubeadm;

--install-pkg-path: Kubernetes static installation package address

The value of --install-pkg-path can be the path on the machine or the network address (for example: http://xxx/xxx/kube-linux-arm64/amd64- *.tar.gz, can be free The secret wget can be reached), pay attention to use the Kubernetes static installation package that matches the machine system;

--apiserver-cert-extra-sans: kube-apiserver certificate extension address
- Recommend signing outside the Master node IP network or domain, as long as IP or domain name can be entered into the edge node Master node access to can, of course, also be allowed within the IP network, provided by the IP edge node can access Kube-apiserver . If you customize the domain name, you can configure hosts on all Matser and Node nodes by yourself;
- Sign the external network IP and domain name, because the edge node and the Master node are generally not in the same local area network, and need to join and access the Master through the external network;
--image-repository: image repository address

If superedge.tencentcloudcr.com/superedge is slower, you can switch to other accelerated mirror warehouses, as long as you can pull down kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, etcd, pause... mirroring.

Other parameters have the same meaning as kubeadm and can be configured according to kubeadm's requirements.

You can also use kubeadm.config to configure the original parameters of edgeadm init --config kubeadm.config --install-pkg-path ./kube-linux-*.tar.gz and create an edge Kubernetes cluster through 060a7223d5faf9.

If there is no problem during execution and the cluster is successfully initialized, the following content will be output:

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
  
Then you can join any number of worker nodes by running the following on each as root:

edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \
    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx
    --install-pkg-path <Path of edgeadm kube-* install package>

If there is a problem during the execution, the corresponding error message will be directly returned and the initialization of the cluster will be interrupted. You can use the ./edgeadm reset command to roll back the initialization operation of the cluster.

Set up the Master kube-config file

To enable non-root users to run kubectl, run the following commands, which are also part of the edgeadm init output:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Or, if you are a root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

Note saved ./edgeadm init output ./edgeadm join command, will be used when the node is added Node behind.

The validity period of the token is the same as that of 24h . After the expiration, you can use ./edgeadm token create create a new token.

The value generation of --discovery-token-ca-cert-hash is also the same as kubeadm, which can be generated by executing the following command on the Master node.

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

Join edge node

<2> on the edge node. Download the edgeadm static installation package, or upload the edgeadm static installation package to the edge node by other means, and then execute the following command:

./edgeadm join <Master节点外网IP/Master节点内网IP/域名>:Port --token xxxx \
     --discovery-token-ca-cert-hash sha256:xxxxxxxxxx 
     --install-pkg-path <edgeadm Kube-*静态安装包地址/FTP路径> --enable-edge=true

among them:

<Master node external network IP/Master node internal network IP/domain name>: Port is the address of the node to access the Kube-apiserver service

You can change the edgeadm init to the Kube-apiserver service prompted by the node to Master node external network IP/Master node internal network IP/domain name, depending on whether you want the node to access the Kube-apiserver service through the external network or the internal network.

--enable-edge=true: Whether the added node is used as an edge node (whether to deploy edge capability components), the default is true

--enable-edge=false means to join the native Kubernetes cluster node, which is exactly the same as the node of kubeadm join;

If there is no problem in the execution process, the new Node successfully joins the cluster, and the following content will be output:

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

If there is a problem in the execution process, the corresponding error message will be directly returned, and the addition of the node will be interrupted. You can use the ./edgeadm reset command to roll back the operation of joining the node and rejoin.

Tip: If the edge node joins successfully, it will give the edge node a label: superedge.io/edge-node=enable , which is convenient for subsequent applications to use nodeSelector to select the application and schedule to the edge node; Native Kubernetes nodes, like kubeadm's join, do not do anything.

`Use edgeadm to install an edge high-availability Kubernetes cluster`

`Installation prerequisites`

Prepare a Master VIP as a unified entrance for available load balancing;
3 sets satisfy kubeadm minimum requirements machine as Master node;
Three meet kubeadm minimum requirements machines do worker node;

`Install Haproxy`

Install Haproxy on the Master as the main entrance of the cluster

Note: Replace <Master VIP> in the configuration file

# yum install -y haproxy# cat << EOF >/etc/haproxy/haproxy.cfgglobal    log         127.0.0.1 local2    chroot      /var/lib/haproxy    pidfile     /var/run/haproxy.pid    maxconn     4000    user        haproxy    group       haproxy    daemon    stats socket /var/lib/haproxy/statsdefaults    mode                    http    log                     global    option                  httplog    option                  dontlognull    option http-server-close    option forwardfor       except 127.0.0.0/8    option                  redispatch    retries                 3    timeout http-request    10s    timeout queue           1m    timeout connect         10s    timeout client          1m    timeout server          1m    timeout http-keep-alive 10s    timeout check           10s    maxconn                 3000frontend  main *:5000    acl url_static       path_beg       -i /static /images /javascript /stylesheets    acl url_static       path_end       -i .jpg .gif .png .css .js    use_backend static          if url_static    default_backend             appfrontend kubernetes-apiserver    mode                 tcp    bind                 *:16443    option               tcplog    default_backend      kubernetes-apiserverbackend kubernetes-apiserver    mode        tcp    balance     roundrobin    server  master-0  <Master VIP>:6443 check # 这里替换 Master VIP 为用户自己的 VIPbackend static    balance     roundrobin    server      static 127.0.0.1:4331 checkbackend app    balance     roundrobin    server  app1 127.0.0.1:5001 check    server  app2 127.0.0.1:5002 check    server  app3 127.0.0.1:5003 check    server  app4 127.0.0.1:5004 checkEOF

`Install Keepalived`

Install Keepalived on all Masters and perform the same operation: note:

Replace <Master VIP> in the configuration file
In the following keepalived.conf configuration file, <Master's local network IP> and <Other Master's external network IP> need to be exchanged in the configuration of different Masters, do not fill in the error.

  ## 安装keepalived  yum install -y keepalived  cat << EOF >/etc/keepalived/keepalived.conf   ! Configuration File for keepalived  global_defs {     smtp_connect_timeout 30     router_id LVS_DEVEL_EDGE_1  }  vrrp_script checkhaproxy{  script "/etc/keepalived/do_sth.sh"  interval 5  }  vrrp_instance VI_1 {      state BACKUP      interface eth0      nopreempt      virtual_router_id 51      priority 100      advert_int 1      authentication {          auth_type PASS          auth_pass aaa      }      virtual_ipaddress {          <master VIP> # 这里替换 Master VIP 为用户自己的 VIP      }      unicast_src_ip <Master 本机外网 IP>      unicast_peer {        <其他 Master 外网 IP>        <其他 Master 外网 IP>      }  notify_master "/etc/keepalived/notify_action.sh master"  notify_backup "/etc/keepalived/notify_action.sh BACKUP"  notify_fault "/etc/keepalived/notify_action.sh FAULT"  notify_stop "/etc/keepalived/notify_action.sh STOP"  garp_master_delay 1  garp_master_refresh 5     track_interface {       eth0     }     track_script {       checkhaproxy      }  }  EOF

`Install the high-availability edge Kubernetes Master`

Perform cluster initialization operations in one of the Masters

./edgeadm init --control-plane-endpoint <Master VIP> --upload-certs --kubernetes-version=1.18.2 --image-repository superedge.tencentcloudcr.com/superedge --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16 --apiserver-cert-extra-sans=<Master节点外网IP/Master节点内网IP/域名/> --install-pkg-path <edegadm Kube-*静态安装包地址> -v=6

The meaning of the parameters is the same as 3. Use edgeadm to install the edge Kubernetes cluster. Others are the same as kubeadm, so I won’t explain it here;

If there is no problem during execution and the cluster is successfully initialized, the following content will be output:

Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:  mkdir -p $HOME/.kube  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  sudo chown $(id -u):$(id -g) $HOME/.kube/config  You should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:  https://kubernetes.io/docs/concepts/cluster-administration/addons/  You can now join any number of the control-plane node running the following command on each as root:  edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx \    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx \    --control-plane --certificate-key xxxxxxxxxx    --install-pkg-path <Path of edgeadm kube-* install package>    Please note that the certificate-key gives access to cluster sensitive data, keep it secret!As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use"edgeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:edgeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx      --install-pkg-path <Path of edgeadm kube-* install package>

If there is a problem during the execution, the corresponding error message will be directly returned, and the initialization of the cluster will be interrupted. Use the ./edgeadm reset command to roll back the initialization operation of the cluster.

To enable non-root users to run kubectl, run the following commands, which are also part of the edgeadm init output:

mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config

Or, if you are a root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

Note saved ./edgeadm init output ./edgeadm join command, add behind Master node and edge nodes need to use.

`Join Master node`

./edgeadm join command on another Master

./edgeadm join xxx.xxx.xxx.xxx:xxx --token xxxx    \    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx \    --control-plane --certificate-key xxxxxxxxxx     \    --install-pkg-path <edgeadm Kube-*静态安装包地址

If there is no problem in the execution process, the new Master successfully joins the cluster, and the following content will be output:

This node has joined the cluster and a new control plane instance was created:* Certificate signing request was sent to apiserver and approval was received.* The Kubelet was informed of the new secure connection details.* Control plane (master) label and taint were applied to the new node.* The Kubernetes control plane instances scaled up.* A new etcd member was added to the local/stacked etcd cluster.To start administering your cluster from this node, you need to run the following as a regular user:        mkdir -p $HOME/.kube        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config        sudo chown $(id -u):$(id -g) $HOME/.kube/config        Run 'kubectl get nodes' to see this node join the cluster.

If there is a problem during the execution, the corresponding error message will be directly returned, and the addition of the node will be interrupted. Use the ./edgeadm reset command to roll back the initialization operation of the cluster.

`Join node edge node`

./edgeadm join xxx.xxx.xxx.xxx:xxxx --token xxxx \    --discovery-token-ca-cert-hash sha256:xxxxxxxxxx     --install-pkg-path <edgeadm Kube-*静态安装包地址/FTP路径>

If there is no problem in the execution process, and the new node successfully joins the cluster, the output will be as follows:

This node has joined the cluster:* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

`Customize the Kubernetes static installation package`

The directory structure of the Kubernetes static installation package is as follows:

kube-linux-arm64-v1.18.2.tar.gz ## kube-v1.18.2 arm64的Kubernetes静态安装包├── bin                         ## 二进制目录│   ├── conntrack               ## 连接跟踪的二进制文件│   ├── kubectl                 ## kube-v1.18.2的kubectl│   ├── kubelet                 ## kube-v1.18.2的kubelet│   └── lite-apiserver          ## 相应版本的lite-apiserver，可编译SuperEdge的lite-apiserver生成├── cni                         ## cin的配置│   └── cni-plugins-linux-v0.8.3.tar.gz ## v0.8.3的CNI插件二进制压缩包└── container                   ## 容器运行时目录    └── docker-19.03-linux-arm64.tar.gz ## docker 19.03 arm64体系的安装脚本和安装包

`Customize other Kubernetes versions`

There are two things you need to do to customize other Kubernetes versions:

Replace binary directory, the version needs to be greater than or equal to Kubernetes v1.18.0;
Ensure that the mirror warehouse used by init has the basic image of the corresponding Kubernetes version;

`Customize other system Kubernetes static installation packages`

Three things need to be done to customize other systems of the Kubernetes static installation package:

Replace all the binaries of the Kubernetes static installation package with the target system, including the binaries in the corresponding installation packages of cni and container;
Ensure that the mirror warehouse used by init has the basic image of the Kubernetes version of the corresponding system. It is recommended to use the multi-system image ;
Fully test to ensure that there are no compatibility issues. If there are related problems, you can also mention Issues in the SuperEdge community to fix them.

[Tencent Cloud Native] Yunshuo new products, Yunyan new technology, Yunyou Xinhuo, Yunxiang information, scan the QR code to follow the public account of the same name, and get more dry goods in time! !