Zheng Caiyun's privatized business delivery practice based on sealer

Introduction to by the Alibaba Cloud Intelligent Cloud Native Application Platform team. The sealer open source project co-built by Zhengcaiyun and Harmony Cloud Technology complements the shortcomings of Kubernetes in the field of integrated delivery. Sealer has a very elegant design The plan considers the overall delivery of cluster + distributed applications. As a representative of the government procurement industry, Zhengcai Cloud has successfully used sealer to complete the overall privatization delivery of large-scale distributed applications. The delivery practice fully proves that sealer has flexible and powerful integrated delivery capabilities.

Author | Zheng Caiyun | Wang Xun

In recent years, the Internet has developed rapidly. In order to keep up with the rapid growth of the business, new technologies have sprung up like bamboo shoots after a rain. At first glance, the sky is full of stars and the stars are vying for beauty. Cloud native technology with containers as the core is growing rapidly. Kubernetes is the new infrastructure and the de facto standard for container orchestration is undoubtedly the most dazzling star.

However, although Kubernetes solves the problems of large-scale application deployment, resource management and scheduling, it is not friendly to business delivery. Kubernetes' own deployment is also more complicated. In the emerging applications surrounding the Kubernetes ecosystem, it has always been There is a lack of applications that can integrate services, middleware, and clusters for integrated delivery.

At the moment, the sealer open source project initiated by the Alibaba Cloud Intelligent Cloud Native Application Platform team and co-built by Zheng Caiyun and Harmony Cloud Technology complements Kubernetes’ shortcomings in the field of integrated delivery. Sealer considers clusters with a very elegant design plan. + Overall delivery of distributed applications. As a representative of the government procurement industry, Zheng Caiyun has successfully used sealer to complete the overall privatization delivery of large-scale distributed applications. The delivery practice fully proves that sealer has flexible and powerful integrated delivery capabilities.

background

The privatization delivery customers of Zhengcai Cloud are government-enterprise scenarios, requiring a large scale of delivery: 300+ business components, 20+ middleware, the delivery target infrastructure is different and uncontrollable, network restrictions are strict, and some sensitive scenarios even It is a completely isolated network. In this context, the biggest pain point of service delivery is the processing of deployment dependencies and the issue of delivery consistency. Although the unified business delivery based on Kubernetes achieves the consistency of the operating environment, how to solve a series of problems such as all the images that depend on the deployment process, the unified processing of various packages, and the consistency of the delivery system itself, urgently need to be resolved.

As shown in the figure above, the process of Zhengcai Cloud's localization delivery is mainly divided into: user needs confirmation -> resource demand is proposed to the user -> the resource list provided by the user is obtained -> the preparation configuration is generated according to the resource list -> the deployment script is prepared And Dependence -> Six steps of actual delivery. During the pre-preparation and actual delivery, it takes a lot of manpower and time to prepare and deploy.

privatization delivery

In the cloud native era, the emergence of docker solves the environmental consistency and packaging problems of a single application, and the delivery of services no longer spends a lot of time on deployment environment dependencies like traditional delivery. Later, the emergence of container orchestration systems such as Kubernetes solved the problem of unified scheduling of underlying resources and unified orchestration of application runtimes. However, for a complex business, the delivery itself is a huge engineering problem. Take the scenario as an example: the deployment and configuration of various resource objects such as helm chart, RBAC, istio gateway, cni, middleware, etc., plus the delivery of more than 300 business components, each privatization delivery brings a lot of Consumption of manpower and time costs.

Zhengcai Cloud is in a period of rapid business development. The demand for privatized deployment projects is constantly increasing. High-cost delivery methods are becoming more and more difficult to support actual needs. How to reduce delivery costs and ensure delivery consistency is the operation and maintenance team The most urgent problem to be solved.

found sealer

In the early days, Zheng Caiyun used ansible for business delivery. The ansible solution achieved a certain degree of automation and reduced delivery costs, but there were several problems:

1. Ansible only solves the problem of the deployment process, and needs to separately prepare the dependencies required for deployment. The preparation of the dependencies and the availability verification incur additional costs, and the localization scenarios of Zhengcai Cloud basically strictly restrict the external network. It is also not feasible to obtain dependencies directly from the external network.

2. Using ansible to respond to differentiated needs will be very tiring. In the privatization delivery scenario of Zhengcaiyun, the needs of each user and business dependence are different, and it will cost a lot to re-edit the ansible playbook for each delivery. Time to debug.

3. When it comes to complex control logic, ansible's declaration language is relatively weak.

4. Ansible's operating environment needs to be prepared before deployment and delivery. Zero dependencies that cannot be delivered.

Ansible is more about doing some glueing things, and it is more suitable for operation and maintenance work with relatively simple logic. With the continuous addition of localization projects, the shortcomings of ansible delivery began to appear. Each localization project requires a lot of time investment. The Zhengcaiyun operation and maintenance team began to explore and think about the optimization direction of the delivery system. We have investigated many technical solutions, but the existing Kubernetes delivery tools focus on the delivery of the cluster itself without considering the delivery of the business layer. Although it can be packaged based on cluster deployment tools, there is no essential difference between this solution and the deployment of upper-level services after cluster deployment using ansible.

Fortunately, we discovered the sealer project. Sealer is a solution for packaged delivery of distributed applications. It solves the delivery problem of complex applications by packaging distributed applications and their dependencies together. The design concept is very elegant, and the packaging and delivery of the entire cluster can be managed with the ecology of container mirroring.

When using Docker, we use Dockerfile to define the operating environment and packaging of a single application. Correspondingly, the technical principles of sealer can be explained by analogy with Docker. The entire cluster is regarded as a machine and Kubernetes is defined as the operating system. Use Kubefile to define the applications in this "operating system" and finally package them into an image, and then, like Docker run to deliver a single application, sealer run can deliver the entire cluster and application.

.png")

After discovering the surprise of sealer, we invited partners from the community to communicate with the company. Sealer is still a new project. It has only been born for a few months. In the actual experience, we encountered a lot of problems. There were a lot of pits, and when we tried to land, we found a lot of places that did not meet the needs, but we did not give up, because we have great expectations and confidence in the sealer design mode, we chose to collaborate with the community Build together and grow together. The final successful landing practice also proved that our choice is very correct!

Community collaboration

At the beginning of deciding to cooperate with the community, we conducted a comprehensive test and evaluation of sealer. Combined with our demand scenarios, there are mainly the following problems:

1. The cost of image caching is too high. Initially sealer only provided the cloud build method, that is, the premise of packaging the sealer cluster image is to pull up a cluster based on cloud resources. We think this method is too costly. Based on this demand, we propose and Contributed to the construction method of lite build, which supports parsing and direct caching of images by parsing helm, resource definition yaml, and mirroring lists. Lite build is the lowest cost build method, no need to pull up the cluster, only a host that can run sealer can complete the build.

2. After the business is delivered, there is a lack of a check mechanism. It is necessary to manually check the status of each component of the Kubernetes cluster. Then, we contributed the check function of the cluster and component status.

3. Some of the early sealer configurations are solidified in rootfs. For example, the deployment host of the registry is fixed on the first master node. In actual scenarios, we have custom requirements, so we contributed a custom registry configuration Function.

4. After the cluster is deployed based on sealer, there will be a need to add nodes to the cluster, so we have contributed the function of sealer join.

In addition, it is necessary to talk about several very practical and powerful sealer features when we landed:

1. One thing that must be mentioned is that the cluster image built by sealer can be directly pushed to a private docker image warehouse such as harbor. Then, like a docker image, you can expand the functions based on the existing image and build it again.

2. The sealer community optimized the registry and docker to support multi-source and multi-domain proxy caching. This is a very practical function. When dealing with mirror dependencies, we cache a mirror that needs to change the address of the mirror. For example, we need to change the mirror address. When a public mirror is cached in a private mirror, the mirror address referenced by the corresponding resource object also needs to be synchronously changed to the address of the private mirror warehouse, but the built-in registry of sealer has been optimized to realize the function of matching the cache without modifying the mirror address. In addition, when the registry built into sealer acts as a proxy, it can proxy multiple privatized mirror repositories, which is a very practical function in scenarios with multiple private repositories.

Landing Practice

Using sealer, we redefine the delivery process. Through Kubefile, the delivery of business components, containerized middleware, image caching and other components is directly completed by sealer. Use the sealed lite build mode to automatically complete the resolution of dependent images and the built-in cache.

Use sealer to shield a large number of complex process logic and dependent processing logic of application delivery, which greatly simplifies the difficulty of implementation. The continuous simplification of the implementation logic makes it possible to deliver on a large scale. In our practical scenario, using the new delivery system, the delivery cycle was shortened from 15 days/person to 2 days/person, and the successful delivery of a cluster with a scale of 2000G+memory and 800+core CPUs including 20G business mirror cache was realized. In the next step, we plan to continue to simplify the delivery process so that a novice can complete the delivery of the entire project with only simple training.

Future Outlook

The success of the landing is not only the result of the delivery system, but also the power of open source, and a new model of cooperation and construction with the community has been explored. In the future, Zheng Caiyun will continue to support and participate in the construction of the sealer community, combining actual business scenarios to provide more contributions to the community.

As a new open source project, sealer is not perfect now. There are still some problems to be solved and optimized. There are more requirements and business scenarios waiting for us to realize. We hope that through continuous contributions, sealer can serve more There are many user scenarios, and we also hope that more partners can participate in the construction of the community, so that the star of sealer will be even more dazzling!

Copyright Notice: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users, and the copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.