Cluster mirroring-cluster as a whole package, distributed application build share run

Author | fanux. Zhongyi

What is cluster mirroring

As the name suggests and operating system image, or .iso Docker mirroring a similar cluster mirroring is a resource pack with certain technical means all files of the entire cluster labeled in a certain format .

Comparing single machine and cluster will find some interesting phenomena:

• A single machine has drivers such as computing and storage networks, and a cluster has CNI/CSI/CRI implementations like cluster drivers
• There are ubuntu centos as stand-alone operating system, we can regard kubernetes as a cloud operating system
• A single machine can run a docker container or a virtual machine, which is equivalent to a running instance, and the cluster also has instances running k8s
• Virtual machine mirroring on a single machine, docker mirroring, so with the development of cloud computing technology, similar mirroring technology will be abstracted at the latitude of the cluster.

​

Take the cluster image based on kubernetes as an example, which contains all files except the operating system:

• Docker's dependent binary and systemd configuration, dockerd configuration, and a private container mirror warehouse
• Kubernetes core component binary, container image, kubelet system configuration, etc.
• The yaml configuration or helm chart that the application needs to use, and the container image of the application
• Other scripts, configuration and binary tools and all the dependencies needed for the application to run

Similarly, when the cluster mirror is running, it is definitely not a container or installed on a machine, but the mirror can be directly installed on multiple servers or directly connected to the infrastructure of the public cloud.

Introduction to sealer

Sealer is an implementation of Alibaba's open source cluster mirroring. Project address: https://github.com/alibaba/sealer To achieve clustering.
Docker solves the problem of mirroring a single container, and sealer realizes the Build Share Run of distributed software by packaging the entire
​

Imagine that we are going to deliver a SaaS application that relies on databases and middleware such as mysql es redis. Everything is orchestrated on kubernetes. If there is no cluster mirroring, do the following:

1. Find a tool to install k8s cluster
2. helm install mysql es redis...If it is an offline environment, you may also need to import the container image
3. kubectl apply yoursaas

It may not seem that complicated, but from the perspective of the delivery of the entire project, it is process-oriented and error-prone.
​

Now if you provide another way to solve the above problem with just one command, would you use it?
sealer run your-saas-application-with-mysql-redis-es:latest
You can see that you only need to run a cluster to mirror the entire cluster and the entire cluster is delivered as a whole, detailed and complex operations are blocked, and any application can be run in the same way. So how did this cluster mirror come from:

We only need to define a file similar to Dockerfile, we call it Kubefile, and then execute the build command:
sealer build -t your-saas-application-with-mysql-redis-es:latest .
A comparison between the two latitudes of a single machine and a cluster can be clear at a glance:

Docker can build a docker image through Dockerfile, and use compose to run the container.
Sealer builds a CloudImage through Kubefile and uses Clusterfile to start the entire cluster.

Quick experience

Make and run a cluster image of kubernetes dashboard to experience a complete process.
Write Kubefile

# 基础镜像，已经被制作好了里面包含所有的kubernetes启动相关的依赖
FROM registry.cn-qingdao.aliyuncs.com/sealer-io/cloudrootfs:v1.16.9-alpha.7
# 下载官方的dashboard yaml编排文件，已经下载了可以使用COPY指令
RUN wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
# 指定运行方式，可以使用kubectl helm kustomiz等
CMD kubectl apply -f recommended.yaml

build dashboard cluster mirror
sealer build -t kubernetes-with-dashobard:latest .
​

Run cluster mirroring

# 下面命令会在服务器上安装k8s集群并apply dashboard, passwd指定服务器ssh密码,也可以使用密钥
sealer run kubernetes-with-dashobard:latest \
    --master 192.168.0.2,192.168.0.3,192.168.0.4 \
  --node 192.168.0.5,192.168.0.6 \
  --passwd xxx
# 检查pod
kubectl get pod -A |grep dashboard

​

Push the produced image to the mirror warehouse, compatible with docker registry

sealer tag kubernetes-with-dashobard:latest docker.io/fanux/dashobard:latest
sealer push docker.io/fanux/dashobard:latest

In this way, the produced image can be delivered or provided to others for reuse.

scenes to be used

What exactly can sealer do for us? Here are a few main scenarios:
​

| Install kubernetes and cluster lifecycle management (upgrade/backup/restore/scaling)
This is the simplest scenario, whether you need to install a development and test environment on a single machine or install a high-availability cluster in a production environment, whether it is bare metal or docking with public clouds, or various architecture operating systems, you can use sealer For installation, if only kubernetes is installed here, just select a base image.
Compared with other installation tools, sealer has the following advantages:

1. Simple enough to sealr run, one command ends
2. The speed is suffocating. After installing 6 nodes in 3 minutes, you may have installed sealer before using other tools. However, we will continue to optimize the black technology to within 2 minutes or even 1 minute.
3. Compatibility and stability Compatible with various operating systems, support x86 arm and other architectures
4. The consistent design will keep the cluster in the defined state in the Clusterfile. Taking upgrade as an example, you only need to change the version number in the Clusterfile to achieve the upgrade.

​

The speed is fast because it is first implemented in golang, which means that we can do concurrent processing in many very detailed places. This has more advantages than ansible, and it can also do more detailed error handling, and then distribute it on the mirror. Abandoning the previous load method, subsequent file distribution will be optimized to achieve the ultimate in installation performance.
​

In terms of compatibility, docker kubelet here uses binary + systemd to install core components and full containerization, so that you no longer need to rely on yum apt and other operating system-aware installation tools. ARM and x86 use different mirroring support to decouple from sealer itself. The adaptation to the public cloud is also implemented in separate modules. The reason why we did not connect to terraform here is for performance. In our scenario, terraform started the infrastructure for nearly 3 minutes, and we optimized the infrastructure startup to 30s through backoff and retry. In addition, there is no need for such complex infrastructure management capabilities in the scenario of cluster mirroring. We don't want to make sealer heavier and don't want to rely on a command line tool.
​

The consistent design concept is worth mentioning in sealer. Cluster mirroring and Clusterfile determine what a cluster looks like. The same mirroring and Clusterfile can run the same cluster. Changes are either to change the Clusterfile, such as adding nodes, changing node specifications, or changing the mirror. When changing the mirror, because the cluster mirror is also a hierarchical structure, the layer with the same hash value will not change, and a change in the hash will help reapply the layer.
​

| Packaging/installation of cloud native ecological software, such as prometheus mysql cluster, etc.
sealer run prometheus:latest can create a cluster with prometheus, or install prometheus in an existing cluster.
So the question is: what is the difference with helm?

1. Sealer doesn’t care about orchestration, but pays more attention to packaging. In the example above, Prometheus can be orchestrated by helm. Sealer will package the chart and all the container images needed in the chart. This is done through black technology during the build process because of the build process. A temporary kubernetes cluster will be set up like a docker build, and then we will know which container images the cluster depends on, and finally package these container images.
2. Packaged with kubernetes, if you take a chart, it may not be able to install successfully. For example, if you use an obsolete api version, but make a mirror to package kubnernetes together, as long as the build is okay, the run is okay. This is the same as docker. The operating system rootfs is packaged together in the same way.
3. Integration, cluster mirroring pays more attention to the overall packaging of the entire distributed application cluster, such as making the prometheus ELK mysql cluster into a mirroring service and business.

Therefore, sealer and helm are in a collaborative relationship with a clear division of labor.
In the future, you can find these common cluster mirrors in the official mirror repository of sealer and use them directly.
​

| SaaS software overall packaging/delivery Proprietary cloud offline delivery
From the perspective of distributed applications, usually from top to bottom, ranging from a few to hundreds of components, most of the existing overall delivery methods are process-oriented, and there are many things that need to be intervened in the middle, and sealer can take these Everything is packaged together for one-click delivery.
​

You may ask, if I make a tar.gz and add an ansible script, it can also be one-clicked. That's for sure, just like everyone delivered through tar.gz before the docker image appeared, you will find the standard and technology The emergence of this solution solves the problem of collaboration between people. With cluster mirroring, you can directly reuse the results of others, and you can also make good things for others to use.
​

Private cloud scenarios are very suitable for using sealer. Many client rooms are offline, and cluster mirroring will put all dependencies into the mirror. As long as the image is made well, all sites can be delivered in the same way with one click, and an excellent consistent experience can be obtained.
​

| Practice the above scenario on the public cloud
Sealer comes with the attributes of connecting to the public cloud. In many cases, connecting to the public cloud will have a better user experience. For example, when installing a cluster, you only need to specify the number and specifications of servers and do not care about the IP. You can directly modify the number defined in the Clusterfile for scaling.

Introduction to Technical Principles

| Copy-on-write
The storage of cluster mirroring is also through copy-on-write. This has two advantages. We can put different distributed software in a cluster on different layers to achieve reuse, and we can also directly push the cluster mirror. Go to the docker mirror warehouse.
​

| Container image cache
During the build process, how does sealer know which container images are in the cluster image to be built and how to store the container images? There are some difficult issues:

1. How to know which container images are in the distributed software, because we need to cache these images, whether it is scanning the user's yaml file or scanning after using the helm template, it is not perfect. First of all, it is not possible to determine the user's arrangement method, and secondly Some software does not even write the mirror address in the layout file, but pulls it up through its own program. There is no guarantee that the build will run successfully.
2. The container image needs to be stored in a private warehouse and packaged in the cluster image. The address of the container image warehouse is bound to be different from that written in the orchestration file, especially how to ensure that the user can still download the image in the private warehouse when the user is alwaysPull.

To deal with the first problem, the sealer solution is to set up a temporary kubernetes cluster during the sealer build process, just like Docker build, and execute the apply command defined by the user in the Kubefile.

In this way, it can be ensured that all the images that the user depends on are packaged in, no matter what kind of arrangement the user uses.
The second question is that we package the container image into a private mirror warehouse. How to use this private mirror is also a problem, because if the private mirror warehouse is named localhost:5000, it will definitely be different from what is written in the orchestration file. We have two The first solution is hack and docker. As long as there is a private mirror warehouse, it will be directly pulled from the private mirror, and there is no ability to pull the mirror from the public network.

Another solution is to not invade the docker proxy, send all docker requests to the proxy, and let the proxy decide if the private warehouse has it, it will pull it from the private warehouse. At the same time, we have also enhanced the capabilities of the registry so that the registry can cache multiple remote warehouses.
This solution of sealer perfectly solves the problem of mirroring packaging in offline scenes.
​

| Load Balancing
Sealer's cluster high availability uses lightweight load balancing lvscare. First, compared to other load balancing lvscare, it is very small by a few hundred lines of code. Moreover, lvscare only guards the ipvs rules. It does not do the load and is very stable. It is directly monitored on the node. Apiserver, if you kneel down, remove the corresponding rules, and automatically add them back after restarting, which is equivalent to a dedicated load balancer. It has been used in the sealos project for more than two years and has extensive practice.

| runtime
The runtime is the environment that supports the running of the application. Like the runtime sealer of base on kuberentes, the transparent support is very simple. Taking istio as an example, the user only needs to:

FROM kubernetes:v1.18.3
RUN curl -L https://istio.io/downloadIstio | sh -

You can build an istio runtime for your own applications.
​

For runtimes that are not base on kuberentes, such as k0s k3s, you can extend the interface in sealer.Runtime so that you can:

FROM k3s:v1.18.3
RUN curl -L https://istio.io/downloadIstio | sh -

More powerful extensions such as extending the runtime of ACK

FROM aliyum.com/ACK:v1.16.9
RUN curl -L https://istio.io/downloadIstio | sh -

This kind of mirroring will directly help the user application to run on the ACK. Some of the above capabilities are in the roadmap
​

| Infrastructure
Many users now want to run their own cluster mirroring in the cloud. Sealer comes with the ability to connect to the public cloud. Sealer's own infrastructure manager, thanks to our more refined backoff and retry mechanism, can complete the infrastructure construction in 30 seconds ( Alibaba Cloud 6 nodes) performance is the best among similar tools, and the number of API calls is greatly reduced, and the configuration is compatible with Clusterfile.

to sum up

Some of sealer's future visions and value withdrawals:

• Sealer can allow users to customize clusters in an extremely simple way to solve the problem of collaboration between distributed software producers and users.
• Extremely simple and friendly User Interface, which can shield and be compatible with various low-level technical details, and run everywhere
• Ecological construction, the official warehouse will cover commonly used distributed software

​

Finally, we summarize:

• If you want to deliver your distributed SaaS as a whole, please use sealer
• If you want to integrate multiple distributed services, such as database message queues or microservice runtimes, please use sealer
• If you want to install a distributed application such as mysql active/standby cluster, please use sealer
• If you need to install/manage a kubernetes high-availability cluster, please use sealer
• If you want to initialize multiple data centers and keep the state of multiple data centers strongly consistent, please use sealer
• If you need to implement the above scenarios on the public cloud, please use sealer

​

Sealer will announce open source in the near future. Those who are interested can follow https://github.com/alibaba/sealer, exchange group :