Clusternet: An open source cross-cloud multi-cluster cloud native control tool!

author

Xu Di, the initiator of the Clusternet project, and an expert in Tencent Cloud container technology.

Summary

Clusternet (Cluster Internet) is an open source cloud-native management and control platform that combines multi-cluster management and cross-cluster application orchestration. It solves the problem of cluster management across clouds, regions, and availability zones. In the project planning stage, it is designed for future scenarios such as hybrid cloud, distributed cloud, and edge computing, and supports access and management of massive clusters, application distribution, and traffic governance (under development).

How to register a cluster

When Clusternet was designed, it completely adopts add-on , which supports one-click deployment and installation. For more installation methods of each module, see official document .

When registering a cluster, it is also very simple. By installing clusternet-agent , you can complete the registration of a cluster, see the following command,

helm repo add clusternet https://clusternet.github.io/charts
helm install clusternet-agent -n clusternet-system --create-namespace \
  --set parentURL=PLEASE-CHANGE-ME \
  --set registrationToken=PLEASE-CHANGE-ME \
  clusternet/clusternet-agent

Here, you need to replace PLEASE-CHANGE-ME with a reasonable configuration for the corresponding cluster.

• parentURL is the apiserver address of the control cluster
• registrationToken is a token that can access the management cluster, which can be bootstrap token or ServiceAccount token .

The main function of these tokens is only to register the cluster, so the permissions can be set very low. The following are the default permission settings.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: clusternet:system:bootstrapping
rules:
  - apiGroups:
      - "clusters.clusternet.io"
    resources:
      - clusterregistrationrequests
    verbs:
      - get
      - create

Group is system:bootstrappers:clusternet:register-cluster-token 161ea842f28050 automatically have the permission to register the cluster. For an example of creating the bootstrap token, you can refer to the following example,

$ # 如下命令会创建一个 bootstrap token "07401b.f395accd246ae52d"
$ # 这里你可以更改 yaml 文件，创建出你设定的值
$ kubectl apply -f manifests/samples/cluster_bootstrap_token.yaml

If you use ServiceAccount token for registration, like k3s does not support using bootstrap token, you can refer to the following example to create ServiceAccount Token for cluster registration.

$ # 你可以更改如下的 yaml 文件，再进行 apply 操作
$ # 如下命令可以创建一个 ServiceAccount token
$ kubectl apply -f manifests/samples/cluster_serviceaccount_token.yaml
$ # 通过如下的命令，即可拿到对应的 ServiceAccount token
$ kubectl get secret -n clusternet-system -o=jsonpath='{.items[?(@.metadata.annotations.kubernetes\.io/service-account\.name=="cluster-bootstrap-use")].data.token}' | base64 --decode; echo

When the clusternet-agent installed, the cluster will be automatically registered in the control cluster specified by the parentURL, and identified by the object ClusterRegistrationRequest . Each cluster has a unique Cluster ID that identifies the cluster. clusternet-agent Restart or rebuild will not change the ID of the currently registered cluster.

Then you can run the following command to view the currently registered cluster,

$ # clsrr is an alias for ClusterRegistrationRequest 
$ kubectl get clsrr
NAME                                              CLUSTER ID                             STATUS     AGE
clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118   dc91021d-2361-4f6d-a404-7c33b9e01118   Approved   3d6h 
$ kubectl get clsrr clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118 -o yaml 
apiVersion: clusters.clusternet.io/v1beta1 
kind: ClusterRegistrationRequest 
metadata: 
  labels: 
    clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118 
    clusters.clusternet.io/cluster-name: clusternet-cluster-dzqkw
    clusters.clusternet.io/registered-by: clusternet-agent
    name: clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118 
spec: 
  clusterId: dc91021d-2361-4f6d-a404-7c33b9e01118
  clusterName: clusternet-cluster-dzqkw
  clusterType: EdgeCluster
status:
  caCertificate: REDACTED
  dedicatedNamespace: clusternet-dhxfs
  managedClusterName: clusternet-cluster-dzqkw
  result: Approved
  token: REDACTED

Once status.result becomes Approved, it means that the cluster has been successfully registered. At this time, clusternet-hub will create a dedicated namespace for the cluster. For example, in the above example, a namespace named clusternet-dhxfs is allocated, and there is an object ManagedCluster The cluster is associated, and all the state of the cluster is reported to this object.

apiVersion: clusters.clusternet.io/v1beta1
kind: ManagedCluster
metadata:
  creationTimestamp: "2022-01-20T09:20:30Z"
  generation: 1
  labels:
    clusternet.io/created-by: clusternet-agent
    clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118
    clusters.clusternet.io/cluster-name: cls-bx2ro4ak
  name: clusternet-cluster-dzqkw
  namespace: clusternet-dhxfs
  resourceVersion: "545410287"
  selfLink: /apis/clusters.clusternet.io/v1beta1/namespaces/clusternet-dhxfs/managedclusters/clusternet-cluster-dzqkw
  uid: 1e6a1003-8309-40c5-8969-c15cdf274a5a
spec:
  clusterId: dc91021d-2361-4f6d-a404-7c33b9e01118
  clusterType: EdgeCluster
  syncMode: Dual
status:
  allocatable:
    cpu: 2820m
    memory: 8657308Ki
  apiserverURL: https://10.8.0.1:443
  appPusher: true
  capacity:
    cpu: "6"
    memory: 12094876Ki
  conditions:
  - lastTransitionTime: "2022-01-21T03:33:59Z"
    message: managed cluster is ready.
    reason: ManagedClusterReady
    status: "True"
    type: Ready
  healthz: true
  heartbeatFrequencySeconds: 180
  k8sVersion: v1.21.5
  lastObservedTime: "2022-01-21T03:33:59Z"
  livez: true
  nodeStatistics:
    readyNodes: 3
  platform: linux/amd64
  readyz: true
  serviceCIDR: 10.4.0.0/14
  useSocket: true

After the cluster is registered, the cluster can be managed and distributed. We will introduce how to do application distribution in the next article.

How to access subclusters

Through Clusternet, you can further manage and control the successfully registered cluster. In some operation and maintenance scenarios, it may be necessary to perform additional separate operations on a sub-cluster, such as viewing logs, events, node status, and so on.

The target subcluster that needs to be managed may:

• Running on edge nodes or edge clusters, the network conditions are poor, and the external network address is not exposed
• In a VPC running on the cloud, in order to ensure security, there is no network connection or port mapping
• Running in a self-built machine room
• other situations

In order to provide a consistent management experience, Clusternet provides a general solution for accessing sub-clusters, that is, the parent cluster can be used as an access proxy to forward requests to the sub-cluster, but dynamic RBAC can still be used. Here, the RBAC used by Clusternet is the sub-cluster's own RBAC, so the sensitive information used in these RBACs does not need to be retained in the parent cluster to achieve real dynamic access. The detailed access link is shown in the following figure.

For convenience, Clusternet also provides command line support, which can be experienced through the kubectl-clusternet plugin.

$ # 安装 kubectl-clusternet 插件
$ kubectl krew install clusternet
$ kubectl get mcls -A 
NAMESPACE          NAME       CLUSTER ID                             SYNC MODE   KUBERNETES                   READYZ   AGE 
clusternet-ml6wg   aws-cd     6c085c18-3baf-443c-abff-459751f5e3d3   Dual        v1.18.4                      true     4d6h 
clusternet-z5vqv   azure-cd   7dc5966e-6736-48dd-9a82-2e4d74d30443   Dual        v1.20.4                      true     43h 
$ # 通过指定 Cluster ID，以及对应 Cluster 的 kubeconfig 文件 (这里的 apiserver 地址可以是内网地址)
$ kubectl clusternet --cluster-id=7dc5966e-6736-48dd-9a82-2e4d74d30443 --child-kubeconfig=./azure-cd-kubeconfig get ns 
NAME                STATUS   AGE
clusternet-system   Active   4d20h 
default             Active   24d 
kube-node-lease     Active   24d 
kube-public         Active   24d 
kube-system         Active   24d 
test-nginx          Active   11d 
test-systemd        Active   11d

For more usage details, please scan the QR code below to learn.

join us

Please pay attention to the Clusternet project https://github.com/clusternet/clusternet , like and support, and welcome all forms of discussion and cooperation.

about us

For more cases and knowledge about cloud native, you can pay attention to the public account of the same name [Tencent Cloud Native]~

Welfare:

① Reply to the [Manual] in the background of the official account, you can get the "Tencent Cloud Native Roadmap Manual" & "Tencent Cloud Native Best Practices"~

②The official account will reply to [series] in the background, and you can get "15 series of 100+ super practical cloud native original dry goods collection", including Kubernetes cost reduction and efficiency enhancement, K8s performance optimization practices, best practices and other series.

③If you reply to the [White Paper] in the background of the official account, you can get the "Tencent Cloud Container Security White Paper" & "The Source of Cost Reduction - Cloud Native Cost Management White Paper v1.0"

③ Reply to [Introduction to the Speed of Light] in the background of the official account, you can get a 50,000-word essence tutorial of Tencent Cloud experts, Prometheus and Grafana of the speed of light.

[Tencent Cloud Native] New products of Yunshuo, new techniques of Yunyan, new activities of Yunyou, and information of cloud appreciation, scan the code to follow the public account of the same name, and get more dry goods in time! !