Cluster mirroring: to achieve efficient distributed application delivery

Author | fanux. Zhongyi
Source | Alibaba Cloud Native Public

What is cluster mirroring

​
As the name suggests, similar to the operating system .iso image or Docker image, the cluster image is uses certain technical means to format all the files of the entire cluster into a resource pack .
​

Comparing single machine and cluster will find some interesting phenomena:
​

• A single machine has drivers for computing, storage, and networking; a cluster has CNI/CSI/CRI to achieve a driver like a cluster.
• A single machine has the ubuntu centos operating system; in a cluster, Kubernetes can be regarded as a cloud operating system.
• A single machine can run a docker container or a virtual machine; it is equivalent to a running instance, and there are also instances of K8s running on the cluster.
• There are virtual machine mirroring and docker mirroring on a single machine; with the development of cloud computing technology, similar mirroring technology will also be abstracted on the cluster.

Take the Kubernetes-based cluster image as an example, which contains all files except the operating system:
​

• Docker depends on the binary and systemd configuration, dockerd configuration, and a private container mirror warehouse.
• Kubernetes core component binary, container image, kubelet system configuration, etc.
• The yaml configuration or helm chart that the application needs, and the container image of the application.
• Other scripts, configuration, and binary tools, and all the dependencies needed for the application to run.

Similarly, when the cluster mirror is running, it is definitely not a container or installed on a machine, but the mirror can be directly installed on multiple servers or directly connected to the infrastructure of the public cloud.

Introduction to sealer

​
Sealer is an implementation of Alibaba's open source cluster mirroring, project address: _ https://github.com/alibaba/sealer_ .
​
Docker solves the problem of mirroring a single container, and sealer realizes the Build Share Run of distributed software by packaging the entire cluster!!!
​
Imagine that we are going to deliver a SaaS application that relies on databases and middleware such as MySQL/ES/Redis. Everything is orchestrated on Kubernetes. If there is no cluster mirroring, do the following:
​

1. Find a tool to install K8s cluster
2. helm install mysql es redis...If it is an offline environment, you may also need to import the container image
3. kubectl apply yoursaas

It may not seem that complicated, but in fact, from the perspective of the delivery of the entire project, the above operations are process-oriented and error-prone.
​
Now if you provide another way to solve the above problem with just one command, would you use it?
​
sealer run your-saas-application-with-mysql-redis-es:latest
​
It can be seen that only one cluster image needs to be run, and the entire cluster is delivered, complicated operations are shielded, and any application can be run in the same way. How did this cluster mirroring come from?
​

As shown in the figure above: we only need to define a file similar to Dockerfile, call it Kubefile, and then execute the build command:
​
sealer build -t your-saas-application-with-mysql-redis-es:latest .
​
From the comparison of the two latitudes of the single machine and the cluster, it can be clear at a glance:
​

• Docker builds a docker image through Dockerfile, and can run the container using compose.
• Sealer builds a CloudImage through Kubefile, and uses Clusterfile to start the entire cluster.

Quick experience

​
Let's make and run a cluster image of the Kubernetes dashboard together to experience a complete process.
​
Write Kubefile:
​

# 基础镜像，已经被制作好了里面包含所有的kubernetes启动相关的依赖
FROM registry.cn-qingdao.aliyuncs.com/sealer-io/cloudrootfs:v1.16.9-alpha.7
# 下载官方的dashboard yaml编排文件，已经下载了可以使用COPY指令
RUN wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
# 指定运行方式，可以使用kubectl helm kustomiz等
CMD kubectl apply -f recommended.yaml

​
Build dashboard cluster mirror:
​
sealer build -t kubernetes-with-dashobard:latest .
​
Run cluster mirroring:
​

# 下面命令会在服务器上安装k8s集群并apply dashboard, passwd指定服务器ssh密码,也可以使用密钥
sealer run kubernetes-with-dashobard:latest \
  --master 192.168.0.2,192.168.0.3,192.168.0.4 \
  --node 192.168.0.5,192.168.0.6 \
  --passwd xxx
# 检查pod
kubectl get pod -A |grep dashboard

​
Push the created image to the mirror warehouse, compatible with docker registry:
​

sealer tag kubernetes-with-dashobard:latest docker.io/fanux/dashobard:latest
sealer push docker.io/fanux/dashobard:latest

In this way, the produced image can be delivered or provided to others for reuse.

scenes to be used

​
What exactly can sealer do for us? Here are a few main scenarios:
​

1. Install Kubernetes and cluster lifecycle management (upgrade/backup/restore/scaling)

This is the simplest scenario, whether you need to install a development and test environment on a single machine, or install a high-availability cluster in a production environment; whether it is bare metal or docking with public clouds, or various architecture operating systems, you can use it Install the sealer. If you only install Kubernetes, you can select a basic image.

Compared with other installation tools, sealer has the following advantages:
​

1. It's horribly simple: the sealer run command ends.
2. The speed is suffocating: 6 nodes are installed in 3 minutes, maybe you have installed sealer before you finish downloading when you use other tools. Not only that, we will also optimize the black technology to 2min or even 1min later.
3. Compatibility and stability: Compatible with various operating systems, and supports x86 arm and other architectures.
4. Consistent design: Let the cluster maintain the defined state in the Clusterfile. Taking upgrade as an example, you only need to change the version number in the Clusterfile to achieve the upgrade.

The speed is fast because first of all it is implemented by golang, which means that we can do concurrent processing on many very detailed places, which has more advantages than ansible; and can also do more detailed error handling, and then distribute it on the mirror. Abandon the previous load method; subsequent file distribution will also be optimized to achieve the ultimate in installation performance.
​
In terms of compatibility, docker kubelet uses binary + systemd to install the core components in full containerization, so that you no longer need to rely on yum/apt and other operating system-aware installation tools. ARM and x86 use different mirroring support to decouple it from sealer itself, and the adaptation of the public cloud is also implemented in a separate module. Here we did not connect to terraform, the reason is still performance. In our scenario, terraform started the infrastructure for nearly 3 minutes, and we optimized the infrastructure startup to within 30s through backoff and retry. In addition, in the cluster mirroring scenario, there is no need for such complex infrastructure management capabilities. We Don't want to make sealer heavy and don't want to rely on a command line tool.
​
The consistent design concept is worth mentioning in sealer. Cluster mirroring and Clusterfile determine what a cluster looks like. The same mirroring and Clusterfile can run the same cluster. Change or change the Clusterfile, such as adding nodes, changing node specifications, or changing the mirror. When changing the mirror, because the cluster mirror is also a hierarchical structure, the layer with the same hash value will not change, and a change in the hash will help reapply the layer .
​

2. Packaging/installation of cloud native ecological software, such as prometheus mysql cluster

​
sealer run prometheus:latest can create a cluster with prometheus, or install prometheus in an existing cluster.
​
So the question is: what is the difference between it and helm?
​

1. Sealer does not care about orchestration, but pays more attention to packaging. In the example above, prometheus can use helm for orchestration. Sealer will package the chart and all the container images needed in the chart. This is done through black technology during the build process, because the build process will Start a temporary Kubernetes cluster like docker build, and then we know which container images the cluster depends on, and finally package these container images.
2. Packaged with Kubernetes, it may not be able to install successfully if you take a chart. For example, it uses an obsolete api version, but Kubnernetes is also packaged as a mirror. As long as the build is okay, the run is okay. This is the same as docker. The operating system rootfs packaged together has the same effect.
3. Integration, cluster mirroring pays more attention to the overall packaging of the entire distributed application cluster, such as making the prometheus ELK mysql cluster into a mirroring service and business.

Therefore sealer and helm are in a collaborative relationship with a clear division of labor.
​
In the future, you can find these general cluster mirrors in the official mirror repository of sealer, and you can use them directly.
​

3. SaaS software packaged/delivered offline and delivered to the proprietary cloud

​
From the perspective of distributed applications, usually from top to bottom, ranging from a few to hundreds of components, most of the existing overall delivery methods are process-oriented, and there are a lot of interventions in the middle, and sealer can take these Everything is packaged together for one-click delivery.
​
You may ask: Can we make a tar.gz and add an ansible script with one-click? The answer is yes. Just as before the emergence of docker mirroring, everyone delivered through tar.gz, you will find that the standard and technology has solved the problem of collaboration between people, can directly reuse the results of others with cluster mirroring. You can also make good things for others to use.
​
The proprietary cloud scenario is very suitable for using sealer. Many client rooms are offline, and cluster mirroring will put all dependencies into the mirror. As long as the mirroring is made well, all sites can be delivered in the same way with one click. , Get an excellent consistency experience.
​

4. Practice the above scenarios on the public cloud

​
Sealer comes with attributes for connecting to public clouds. In many cases, connecting to public clouds will have a better user experience. For example, when installing a cluster, you only need to specify the number and specifications of servers and do not care about IP. You can directly modify the number defined in the Clusterfile for scaling.
​

Introduction to Technical Principles

​

1. Copy-on-write

​
The storage of cluster mirroring is also realized by copy-on-write. This has two advantages: we can put different distributed software in the same cluster on different layers to achieve reuse; we can also push the cluster image directly to the docker image warehouse.
​

2. Container image cache

​
During the build process, how does sealer know which container images are in the cluster image to be built, and how to store the container images? There are some difficult issues:
​

1. How to know which container images are in distributed software? Because we need to cache these images, whether it is scanning the user's yaml file or scanning after using the helm template is not perfect. First of all, it is not sure what the user's layout method is. Secondly, some software does not write the mirror address in the layout file, but pulls it up through its own program. There is no guarantee that the build will run successfully and there will be no problem.
2. The container image needs to be stored in a private warehouse and packaged in the cluster image. The address of the container image warehouse is bound to be different from that written in the orchestration file. In particular, how to ensure that the image can be downloaded in the private warehouse when the user alwaysPull is used?

To deal with the first problem, sealer's solution is: the sealer build process is the same as Docker build, it will pull up a temporary Kubernetes cluster and execute the apply instruction defined by the user in the Kubefile.
​

As shown in the figure above, this ensures that all the images that the user depends on are packaged, no matter what kind of arrangement the user uses.
​
The second question is that we package the container image into a private mirror warehouse. How to use this private mirror is also a difficult problem. Assuming the private mirror warehouse is named localhost:5000, it will definitely be inconsistent with the one written in the orchestration file. Way to solve:
​

• The first is hack and docker. As long as there is a private mirror warehouse, it will be pulled directly from the private mirror, and there is no ability to pull the mirror from the public network.
• The second solution is to not invade the proxy of docker, and send all docker requests to the proxy, and let the proxy decide if the private warehouse has it, it will pull it from the private warehouse. At the same time, we have also enhanced the capabilities of the registry so that the registry can cache multiple remote warehouses.

This solution of sealer perfectly solves the problem of mirroring packaging in offline scenes.
​

3. Load Balancing​

​
Sealer's cluster high availability uses lightweight load balancing lvscare. Compared with other load balancing, lvscare is very small and only has a few hundred lines of code, and lvscare only guards the ipvs rules. It does not do the load and is very stable. It directly monitors the apiserver on the node. If you kneel down, remove the corresponding rules and restart. It will be automatically added back after it is up, which is equivalent to a dedicated load balancer. It has been used for more than two years in the sealos project and has extensive practice.
​

4. Runtime

​
The runtime is the environment that supports the running of the application. For example, the runtime sealer of base on Kuberentes can transparently support it is very simple. Taking istio as an example, the user only needs to:
​

FROM kubernetes:v1.18.3
RUN curl -L https://istio.io/downloadIstio | sh -

You can build an istio runtime for your own applications.
​
For runtimes that are not base on Kuberentes, such as k0s k3s, you can extend the interface in sealer.Runtime so that you can:
​

FROM k3s:v1.18.3
RUN curl -L https://istio.io/downloadIstio | sh -

More powerful extensions such as extending the runtime of ACK:

FROM aliyum.com/ACK:v1.16.9
RUN curl -L https://istio.io/downloadIstio | sh -

This kind of mirroring will directly help the user application to run on the ACK. Some of the above capabilities are in the roadmap.
​

5. Infrastructure​

​
Many users now want to run their own cluster mirroring in the cloud. Sealer comes with the ability to connect to the public cloud. Sealer's own infrastructure manager, thanks to our more refined backoff and retry mechanism, can complete the infrastructure construction in 30 seconds ( Alibaba Cloud 6 nodes) performance is the best among similar tools, and the number of API calls is greatly reduced, and the configuration is compatible with Clusterfile.
​

to sum up

​
Some of sealer's future visions and value withdrawals:
​

• Sealer can allow users to customize clusters in an extremely simple way to solve the problem of collaboration between distributed software producers and users.
• The extremely simple and friendly User Interface can shield and be compatible with various underlying technical details and run everywhere.
• Ecological construction, the official warehouse will cover commonly used distributed software.

Finally, we summarize:
​

• If you want to deliver your distributed SaaS as a whole, please use sealer.
• If you want to integrate multiple distributed services, such as database message queues or microservice runtimes, please use sealer.
• If you want to install a distributed application such as mysql active/standby cluster, please use sealer.
• If you need to install/manage a Kubernetes high-availability cluster, please use sealer.
• If you want to initialize multiple data centers and keep the state of multiple data centers strongly consistent, please use sealer.
• If you need to implement the above scenarios on the public cloud, please use sealer.

Sealer will announce open source in the near future. Those who are interested can follow: https://github.com/alibaba/sealer .
​
If you have any questions, welcome to Dingding search group number: 34619594 to join Dingding exchange group.