Still worrying about multi-cluster management? OCM is coming!

About the author: Feng Yong (Hua Ming Lu Jing), senior technical expert, has a doctorate in computer science from Northwestern Polytechnical University, has more than ten years of design and development experience in the field of high-performance computing, big data and cloud computing, focusing on scheduling and resources And application management field. He is also deeply involved in the development and commercialization of related open source projects, such as OpenStack, Mesos, Swarm, Kubernetes, Spark, etc., and has led related development teams at IBM.

Preface

In the field of cloud computing, if anyone hasn’t heard of Kubernetes, it’s as if someone doesn’t know that Chongqing hot pot must have chili. Kubernetes has become the de facto standard platform for managing data centers like Android on mobile phones and Windows on laptops. Around Kubernetes, the open source community has built a rich technology ecosystem. Whether it is CI/CD, monitoring operation and maintenance, or application framework, security anti-intrusion, users can find projects and products that suit them. However, once the scenario is extended to a multi-cluster and hybrid cloud environment, there are only a handful of open source technologies that users can rely on, and they are often not mature and comprehensive.

In order to allow developers and users to use their familiar open source projects and products to easily develop functions in a multi-cluster and hybrid environment, just like on a single Kubernetes cluster platform, RedHat, Ant and Alibaba Cloud jointly initiated and open sourced OCM (Open Cluster Management) aims to solve the life cycle management problems of resources, applications, configurations, policies and other objects in a multi-cluster and mixed environment. At present, OCM has submitted an application for the incubation of Sandbox-level projects to CNCF TOC.

Project official website: https://open-cluster-management.io/

Multi-cluster management development history

Let's bring the time back to a few years ago. When the focus of the industry's attention/controversy was still on whether Kubernetes was available at production level, the first batch of players who landed on the "multi-cluster federation" technology appeared. Most of them are pioneers of Kubernetes practice that are far above average in size. From the earliest Redhat and Google entering the market, they tried KubeFed v1, and then later joined hands with IBM to learn from experience and launch KubeFed v2. In addition to these large companies exploring multi-cluster federation technology in the production and practice of Kubernetes, in the commercial market, most of the service products packaged by various vendors based on Kubernetes have also experienced the evolution from single-cluster product services to multi-cluster forms and hybrid cloud scenarios. . In fact, both companies and business users have common needs, focusing on the following aspects:

Multi-regional problem: When the cluster needs to be deployed on a heterogeneous infrastructure or across a wider region

The Kubernetes cluster relies on etcd as the data persistence layer, and etcd as a distributed system has requirements on the network delay between the members of the system, and there are some restrictions on the number of members, although the delay can be tolerated by adjusting the heartbeat, etc. Parameter adaptation, but cannot meet the global deployment requirements of transnational and transcontinental, nor can it guarantee the number of available zones in large-scale scenarios, so in order to make etcd at least stable operation, etcd is generally planned into multiple clusters by region. In addition, based on business availability and security, hybrid cloud architecture is increasingly accepted by users. It is difficult to deploy a single etcd cluster across cloud service providers, and correspondingly, the Kubernetes cluster is also split into multiple. When the number of clusters gradually increases and administrators are tired of coping, it is natural to need an aggregated control system to manage and coordinate multiple clusters at the same time.

Scale problem: When a single cluster encounters a bottleneck

It is true that the open source version of Kubernetes has obvious scale bottlenecks, but even worse, it is difficult for us to truly quantify the scale of Kubernetes. At the beginning, the community provided the kubemark suite to verify the performance of the cluster, but the reality is very skinny. What kubemark does is based on the repeated scaling and scheduling of workloads under different numbers of nodes. However, in practice, the reasons for the Kubernetes performance bottleneck are complex and there are many scenarios. It is difficult for kubemark to comprehensively and objectively describe the scale of multi-clusters, and can only be used as a reference solution under very coarse-grained conditions. Later, the community supported the use of scale envelopes to measure cluster capacity in multiple dimensions, and later there was a more advanced cluster stress testing suite perf-tests. When users have a clearer understanding of the problem of scale, they can plan the distribution of multiple Kubernetes clusters in advance according to actual scenarios (such as IDC scale, network topology, etc.), and the demand for multi-cluster federation will also emerge.

Disaster tolerance/isolation issues: When more granular isolation and disaster tolerance requirements

Disaster recovery of business applications is achieved through scheduling strategies in the cluster and deploying applications to infrastructure availability zones of different granularities. Combining technologies such as network routing, storage, and access control can solve the problem of business continuity after the availability zone fails. But how to solve the cluster level, or even the cluster management control platform itself?

As a distributed system, etcd can naturally solve the problem of most node failures, but unfortunately in practice, etcd services may still be down, which may be due to management errors or network partitions. In order to prevent etcd from "destroying the world" when there is a problem, the "explosion radius" is often reduced to provide a more granular disaster recovery strategy. For example, in practice, it is more inclined to build multiple clusters within a single data center to avoid split-brain problems, and at the same time make each cluster an independent autonomous system, which can run completely even in the presence of network partitions or higher-level control offline, at least stable on site. This naturally forms the need to control multiple Kubernetes clusters at the same time.

On the other hand, the isolation requirement also comes from the cluster's lack of multi-tenant capabilities, so the cluster-level isolation strategy is directly adopted. By the way, the good news is that the control plane fairness/multi-tenant isolation of Kubernetes is being built brick by brick. By entering the APIPriorityAndFairness feature of Beta in version 1.20, you can actively customize the traffic soft isolation strategy according to the scenario, rather than passively Punish and limit traffic through ACLs. If it is divided into multiple clusters at the beginning of cluster planning, then the problem of isolation will naturally be solved. For example, we can assign exclusive clusters to big data according to the business, or assign exclusive clusters to specific business applications.

Main functions and architecture of OCM

OCM aims to simplify the management of multiple Kubernetes clusters deployed in a mixed environment. It can be used to expand multi-cluster management capabilities for different management tools in the Kubernetes ecosystem. OCM summarizes the basic concepts required for multi-cluster management and believes that in multi-cluster management, any management tool needs to have the following capabilities:

1. Understand the definition of clusters;

2. Select one or more clusters through a certain scheduling method;

3. Distribute configuration or workload to one or more clusters;

4. Govern user's access control to the cluster;

5. Deploy management probes to multiple clusters.

OCM adopts the hub-agent architecture and includes several primitives and basic components for multi-cluster management to meet the above requirements:

●Define the managed clusters through the ManagedCluster API. At the same time, OCM will install an agent named Klusterlet in each cluster to complete cluster registration, life cycle management and other functions.

●Define how to schedule the configuration or workload to which clusters through the Placement API. The scheduling result will be stored in the PlacementDecision API. Other configuration management and application deployment tools can use PlacementDecisiono to determine which clusters need to be configured and deployed.

●Define the configuration and resource information distributed to a certain cluster through the ManifestWork API.

●Group clusters through the ManagedClusterSet API, and provide the boundaries for users to access the cluster.

● Through the ManagedClusterAddon API, define how the management probe is deployed to multiple clusters and how it communicates securely and reliably with the control plane of the hub.

The architecture is shown in the figure below, where registration is responsible for cluster registration, cluster life cycle management, registration of management plug-ins, and life cycle management; work is responsible for resource distribution; placement is responsible for cluster load scheduling. On top of this, developers or SRE teams can conveniently develop and deploy management tools in different scenarios based on the API primitives provided by OCM.

By using OCM's API primitives, the deployment and operation and maintenance of many other open source multi-cluster management projects can be simplified, and the multi-cluster management capabilities of many Kubernetes single-cluster management tools can also be expanded. E.g:

1. Simplify the management of submariner and other multi-cluster network solutions. Use OCM's plug-in management function to centralize the deployment and configuration of submariner on a unified management platform.

2. Provide a wealth of multi-cluster scheduling strategies and reliable resource distribution engines for application deployment tools (KubeVela, ArgoCD, etc.).

3. Expand the existing kuberenetes single-cluster security policy management tools (Open Policy Agent, Falco, etc.) to make it capable of multi-cluster security policy management.

OCM also has two built-in management plug-ins for application deployment and security policy management. The application deployment plug-in adopts the subscriber model, which can obtain application deployment resource information from different sources by defining a subscription channel (Channel). Its architecture is shown in the following figure:

At the same time, in order to closely integrate with the kubernetes ecosystem, OCM has implemented multiple design schemes of kubernetes sig-multicluster, including KEP-2149 Cluster ID:
https://github.com/Kubernetes/enhancements/tree/master/keps/sig-multicluster/2149-clusterid

And the concept of clusterset in KEP-1645 Multi-Cluster Services API: https://github.com/Kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api

Also working with other developers in the community to promote the development of Work API https://github.com/Kubernetes-sigs/work-api .

The main advantages of OCM

highly modular --- freely selectable/

The entire OCM architecture is very similar to a "microkernel" operating system. The OCM chassis provides services such as core capabilities cluster metadata abstraction, while other expansion capabilities are detachable and deployed as independent components. As shown in the figure above, except for the core capabilities of the entire OCM solution, other upper-level capabilities can be tailored according to actual needs. For example, if we don’t need complex cluster topology, then we can tailor the cluster grouping. If we don’t need to distribute any resources through OCM only as metadata, then we can even cut out the Agent component distributed by the entire resource. This is also helpful to guide users to log in to OCM gradually. In the initial stage, users may only need to use a small part of the functions, and then gradually introduce more feature components as the scene expands, and even support heating on the running control surface at the same time. Plug and unplug.

more inclusive --- Swiss army knife for complex usage scenarios

At the beginning of the design of the entire OCM solution, it was considered to build advanced capabilities in some complex scenarios by integrating some mainstream third-party technical solutions. For example, in order to support the rendering and delivery of more complex application resources, OCM supports the installation of applications in the form of Helm Chart and supports loading into remote Chart warehouses. At the same time, the Addon framework is also provided to support users to customize and develop their own needs through the provided extensibility interface. For example, Submarine is a multi-cluster network trust solution developed based on the Addon framework.

ease of use --- reduce the complexity of use

In order to reduce the complexity of users and the ease of migrating to the OCM solution, OCM provides a traditional command-style multi-cluster federated control process. It is worth noting that the following mentioned functions are still in the research and development process, and will officially meet with you in subsequent versions:

●Through ManagedClusterAction, we can issue atomic commands to the managed clusters one by one. This is also the most intuitive way to automatically arrange each cluster as a central control system. A ManagedClusterAction can have its own instruction type, instruction content, and specific status of instruction execution.

●Through ManagedClusterView, we can actively "project" the resources in the managed cluster to the multi-cluster federated central system. By reading the “projection” of these resources in the central, we can make more dynamic and accurate decisions in the federated system .

Practice of OCM in Ant Group

OCM technology has been applied to the Ant Group’s infrastructure. As the first step, OCM Klusterlets are deployed to the managed clusters one by one through the use of some operation and maintenance methods similar to the community Cluster API, thereby integrating dozens of lines in the ant domain. The meta-information of the online and offline clusters is integrated into the OCM. These OCM Klusterlets provide the upper product platform with the basic capabilities of multi-cluster management, operation and maintenance to facilitate future function expansion. Specifically, the first step of OCM includes the following aspects:

●Certificateless: In the traditional multi-cluster federation system, we often need to configure the corresponding cluster access certificate for the metadata of each cluster, which is also a required field in the cluster metadata model of KubeFed v2. Since OCM adopts the Pull architecture as a whole, the Agents deployed in each cluster pull tasks from the hub and there is no process for the hub to actively access the actual cluster. Therefore, the metadata of each cluster is only a completely "desensitized" placeholder. symbol. At the same time, because the certificate information does not need to be stored, there is no risk of the certificate being copied and misappropriated in the OCM scheme

●Automated cluster registration: In the previous cluster registration process, there were many manual interventions, which lengthened the collaboration and communication time and lost the flexibility of changes, such as site-level or computer room-level flexibility. In many scenarios, manual verification is indispensable. You can make full use of the review and verification capabilities provided by OCM cluster registration and integrate them into the approval process tools in the domain to achieve the entire cluster registration process and achieve the following goals:

(1) Simplify the cluster initialization/takeover process;

(2) Clearly control the authority of the control center.

●Automatic cluster resource installation/uninstallation: The so-called takeover mainly includes two things (a) installing the application resources required by the management platform in the cluster (b) entering the cluster metadata into the management platform. (A) can be further divided into Cluster-level and Namespace-level resources, while (b) is generally a critical operation for the upper-level management and control system. From the moment the metadata is entered, the product is considered to have taken over the cluster. Before the introduction of OCM, all preparations required manual preparation step by step. Through OCM, the entire process can be automated, simplifying the cost of manual collaboration and communication. The essence of this matter is to sort out the cluster management into a process operation, and define the concept of state on the cluster metadata so that the product hub can automatically take over the "trivial tasks" that need to be done in a process. After registering the cluster in OCM, the installation and uninstallation procedures of resources are clearly defined.

Through the above work, dozens of clusters in the ant domain are all within the management scope of OCM. In the double eleven and other big promotion activities, automatically created and deleted clusters also realized automatic access and deletion. Later, it plans to integrate with application management technologies such as KubeVela, and collaborate to complete the cloud-native management capabilities of applications and security policies in Ant Domain.

Practice of OCM in Alibaba Cloud

In Alibaba Cloud, the OCM project is KubeVela
https://github.com/oam-dev/kubevela

One of the core dependencies of undifferentiated application delivery for mixed environments. KubeVela is a "one-stop" application management and delivery platform based on the Open Application Model (OAM), and it is currently the only cloud native application platform project hosted by the CNCF Foundation. In terms of function, KubeVela can provide developers with an end-to-end application delivery model, as well as multiple cluster-oriented operation and maintenance capabilities such as gray-scale release, elastic scaling, and observability, and can be applied to a mixed environment with a unified workflow Delivery and management. In the whole process, OCM is KubeVela's main technology to implement Kubernetes cluster registration, management, and application distribution strategies.

On the public cloud, the above features of KubeVela combined with the Alibaba Cloud ACK multi-cluster management capabilities can provide users with a powerful application delivery control plane, which can be easily implemented:

●One-click site building in a mixed environment. For example, a typical hybrid environment can be a public cloud ACK cluster (production cluster) plus a local Kubernetes cluster (test cluster) managed by ACK multi-cluster. In these two environments, the providers of application components are often different. For example, the database component may be MySQL in the test cluster, and the Alibaba Cloud RDS product on the public cloud. In such a mixed environment, traditional application deployment and operation and maintenance are extremely complicated. KubeVela allows users to easily define the products to be deployed, deliver the workflow, and declare the differentiated configuration of different environments in a deployment plan. This not only eliminates the tedious manual configuration process, but also greatly reduces the release and operation and maintenance risks with the help of Kubernetes' powerful automation and certainty.

●Multi-cluster microservice application delivery: Microservice applications under the cloud native architecture are often composed of diverse components, such as container components, Helm components, middleware components, cloud service components, etc. KubeVela provides users with a micro-service-oriented multi-component application delivery model. With the help of the distribution strategy provided by OCM, unified application delivery is carried out in a multi-cluster and mixed environment, which greatly reduces the difficulty of operation, maintenance and management of micro-service applications.

In the future, the Alibaba Cloud team will work with partners such as the RedHat/OCM community, Oracle, and Microsoft to further improve KubeVela's application orchestration, delivery, and operation and maintenance capabilities for hybrid environments, so that the delivery and management of microservice applications in the cloud-native era can be truly achieved. "Fast and good."

Join the community

At present, the OCM community is still in the early stages of rapid development, and interested companies, organizations, schools and individuals are very welcome to participate. Here, you can become partners with technical experts from Ant Group, RedHat, and Alibaba Cloud, as well as the core Contributor of Kubernetes, to learn, build, and promote the popularization of OCM together.

●GitHub address:
https://github.com/open-cluster-management-io

●Learn about OCM through video:
https://www.youtube.com/channel/UC7xxOh2jBM5Jfwt3fsBzOZw

● Come to the community weekly meeting to meet everyone:
https://docs.google.com/document/d/1CPXPOEybBwFbJx9F03QytSzsFQImQxeEtm8UjhqYPNg

●Free communication in the Kubernetes Slack channel # open-cluster-mgmt:
https://slack.k8s.io/

●Join the mail group to browse key discussions:
https://groups.google.com/g/open-cluster-management

●Visit the official website of the community for more information:
https://open-cluster-management.io/

On September 10 this year, INCLUSION·The Bund Conference will be held as scheduled. As a global financial technology event, it will continue to maintain the original intention of making technology more inclusive. In the multi-cluster and hybrid cloud architecture open source special session on the afternoon of the 11th, the main developers of the OCM community will bring you the best practices of multi-cluster and hybrid cloud built around OCM. You are welcome to participate offline and have face-to-face communication.
Thank you for your attention and participation in OCM, and welcome to share with more friends who have the same needs. Let us work together to further the multi-cluster and hybrid cloud experience!

Recommended reading this week

• RFC8998+BabaSSL--- Let the national secret drive to the farther sea of stars
• MOSN sub-project Layotto: opening a new chapter in service grid + application runtime
• opens a new chapter in cloud native MOSN-integrating Envoy and GoLang ecology
• MOSN multi-protocol extension development practice

For more articles, please scan the QR code to follow the "Financial Grade Distributed Architecture" public account