Best Practice丨It is not a dream to use Rancher to easily manage tens of thousands of resources!

Preface

Rancher is an open source enterprise-level Kubernetes cluster management platform. You can import existing clusters, such as ACK, TKE, EKS, GKE, or use RKE, RKE2, K3s to customize the deployment cluster.

As the industry's leading multi-cluster management platform, Rancher can manage thousands of clusters and tens of thousands of nodes at the same time. At the same time, you don't have to worry about the additional burden of operating and maintaining large-scale clusters and nodes. Social communication software LINE 5 people is enough to manage 2000 nodes in 130 clusters.

How to manage large-scale clusters and nodes in Rancher will not be discussed this time. It has been introduced many times before. Today we only talk about how to manage large-scale projects, namespaces and other resources in a single cluster.

How to manage large-scale projects and namespaces in a single cluster

• Namespace : Namespace is a concept of Kubernetes, which allows a virtual cluster to be established in a cluster. This is useful for dividing the cluster into separate "virtual clusters", each virtual cluster has its own access control and Resource quota.
• project : A project is a set of namespaces, which is a concept introduced by Rancher. The project allows you to manage multiple namespaces as a group and perform Kubernetes operations in them. You can use projects to support multi-tenancy. For example, you can set up a team to access a certain project in the cluster, but not other projects in the same cluster.

Hierarchically, the cluster contains projects, and projects contain namespaces. If the company is understood as a cluster, then the project corresponds to a department or team, and we can divide a certain department or team for different projects to manage multiple namespaces.
Therefore, we will create a large number of projects and namespaces in the production environment, so how to plan these projects and namespaces?
It is recommended that projects be divided according to departments or teams, and relevant access control is granted for different projects

Use namespaces to isolate environments with different purposes in each project, for example: production environment and test environment

In order to verify the ability to manage large-scale projects and namespaces in a single Rancher cluster, we created 500 projects in a cluster, and created 10 namespaces in each project, and finally, created in each namespace : 1 Deployment, 2 Secrets, 1 Service, 2 ConfigMaps to simulate the use in real scenarios.

The following is the test cluster size:

In this way, we created a total of 500+ projects, 5000+ namespaces, 5000+ Deployment/pods,

Next, let's experience the management of these resources through Rancher UI:

First log in to the Rancher UI, because there is only one cluster, no matter how many resources are under the cluster, it will not be loaded, so the loading speed is the same as that of a normal-scale cluster

When there are many items, we can quickly locate the item we need to query in the search box:

Enter a project, 1.5 seconds to complete the loading

Next, let's create a Deployment. From the results, the loading speed is no different from that of a normal-scale cluster.

The management of other resources will not be shown to you one by one, and the test results are almost the same as the loading speed of a normal-scale cluster. From the above scenario analysis, single cluster contains large-scale projects, namespaces and other resources and does not bring too much performance pressure to Rancher .
Due to the limited host resources of the test environment, only 5,000 deployments were created in this verification. As long as resources are allocated to the project reasonably, I believe that using Rancher to manage several times the resource scale will be very easy.

A scene that is not recommended

Although it is not a recommended scenario, it should be explained to avoid pitfalls when you manage large-scale projects and namespaces in a single cluster.

It is strongly not recommended that you accumulate all resources in the same project, such as putting 5000+ namespaces in the same project.

Why not recommend it? Simply put, when entering a project from the Rancher Cluster Manager, all resources contained in the project will be loaded. The more resources contained in the project, the slower the speed of loading the resources in the project. If the amount of data is particularly large, sometimes timeouts may occur. The Rancher management plane communicates with the API of the downstream cluster through the cluster tunnel. If the amount of resources is too large, the burden on the websocket tunnel will be heavy.

Although it is not recommended that everyone accumulate all resources in the same project, it does not mean that there is no way to manage this kind of scene in Rancher. If you must put all namespaces and other resources in the same project in your production environment, and the amount of data is very large. Then it is recommended that you use the newly added Cluster Explorer of Rancher v2.5. Cluster Explorer is optimized for loading resources in a single project, and there is almost no timeout.

The following figure shows an example of loading 3000 namespaces, 3000 Deployments, 3000 Services, 6000 ConfigMaps, and 6000 Secrets in the same project in Cluster Explorer:

It can be seen that through Rancher 2.5's new Dashboard, Cluster Explorer, it is also possible to manage thousands of resources in a single project.

Tuning recommendations for large-scale cluster management

As a multi-cluster management platform, whether it is the increase in the number of clusters or the increase in the number of workloads in a single cluster, it will pose certain performance challenges to the management plane. The default parameters of all components are the best product configuration for common scales. For large-scale scenarios, some platform parameters and even system kernel parameters need to be tuned so that the core of the management plane can obtain the best performance.

Users can take corresponding optimization schemes according to their own scale of use. All optimization measures are not necessary, and they need to be tailored to their own scenarios. Rancher's default configuration parameters are already the best solution for most usage scenarios. If you are managing a large-scale cluster, you need to adjust the relevant configuration. The following are some common optimization parameters in some large-scale clusters:
For the large number of TCP connections on the management plane, the OS Kernel level has been optimized to maximize the performance of the hardware configuration. Common configurations and their functions are described as follows:

The following parameters are personal summary, for reference only

Kernel Tuning

Kube-apiserver
The concurrency and cache optimization of APIs for native resources and CRD improve the performance of API List-Watch:

The focus of this article is not tuning, so only some commonly used parameters are introduced. For more detailed parameters (for example: kube-controller-manager, kube-scheduler, kubelet, kube-proxy), please refer to the Kubernetes official website.

postscript

Judging from the above verification results, Rancher has the ability to manage large-scale projects and namespaces in a single cluster. As long as the correct way to build the cluster and deploy the business is used reasonably, it will not bring too much pressure to Rancher, nor There are performance problems, and at the same time, the host and Kubernetes parameters must be tuned according to the cluster size to make it play the best performance.

about the author
Wang Hailong, SUSE Rancher Community Technical Manager, is responsible for the maintenance and operation of the Rancher China Technical Community. He has 6 years of experience in the cloud computing field, and has experienced technological changes from OpenStack to Kubernetes. He has rich operation and maintenance and practical experience no matter the underlying operating system Linux, virtualized KVM or Docker container technology.