Author: Xu Wei, R&D engineer of Aerospace Netcom
K8s is the leader in the field of container orchestration and distributed application deployment. In the K8s environment, we only need to care about the business logic of the application, which reduces the management burden of our server network and storage. For a user, K8s is a very complex container orchestration platform, and the learning cost is very high. KubeSphere abstracts the underlying K8s and conducts a high degree of productization to build a full-stack multi-tenant container cloud platform, providing users with a robust, secure, feature-rich, and ultimate experience Web console, solving K8s Pain points such as the high threshold for use and the complexity of cloud native ecological tools allow us to focus on the rapid iteration of the business, and its multi-dimensional data monitoring has provided great help in locating problems.
Why implement hierarchical management on KuberSphere
In KubeSphere, resources can be shared among tenants, and various resources can be operated according to different roles assigned. There is a high degree of freedom between tenants and resources and between resources and resources, and the granularity of permissions is relatively large. In our system, resources have permission levels. For example, low-level users can operate high-level resources by inviting, granting permissions, etc., or for example, Pods in low-level projects can be scheduled to high-level nodes. to resources. For issues such as cross-level operation resources, we implement hierarchical management on the basis of KubeSphere.
What is a grading system
Grading, as the name suggests, is to decompose and classify the whole according to the established standards. We abstract it into a pyramid model. There will be many levels from the foundation to the top of the tower. We use public resources as the foundation of the pyramid, the admin with the highest authority as the top of the tower, and other resources are divided into different levels according to the level of authority. Low-level resources cannot access high-level resources, and high-level resources can obtain all resources under its level, constructing such a hierarchical system with diminishing rights and isolation between levels.
How to realize hierarchical management
We define a label representing the level kubernetes.io/level
. Take a multi-node cluster as an example. First, we will label resources such as users, enterprise spaces, and nodes with representative levels. When inviting users to join an enterprise space or project, the level of the added enterprise space or project must not be higher than the user’s level. Similarly, when a project is bound to an enterprise space, the level of the project must not be higher than the level of the enterprise space. Resources are managed; we believe that the level of resources under the same project is the same, and the level of resources such as load, Pod, and service created based on the project is consistent with the project; at the same time, node affinity is added to the Pod, so that the Pod is scheduled to on a node no higher than its authority level.
For example, here, we have created a user with permission level 3 demo-user
, who can join the enterprise space or project whose permission level is not higher than 3.
kind: User
apiVersion: iam.kubesphere.io/v1alpha2
metadata:
name: demo-user
labels:
kubernetes.io/level: 3
spec:
email: demo-user@kubesphere.io
Create a project with a permission level of 2 demo-ns
, then the permission level of resources such as load, Pod, and storage created by the project is also 2.
apiVersion: v1
kind: Namespace
metadata:
name: demo-ns
labels:
kubernetes.io/level: 2
Based on the demo-ns
project, a Pod of nginx
85e658dcaaa050ad443141d87d8233e1--- has been created, and its permission level is also 2. At the same time, node affinity is added, and it is required to be scheduled to a node whose permission level is not higher than 2.
apiVersion: apps/v1
kind: Pod
metadata:
labels:
kubernetes.io/level: 2
name: nginx
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
protocol: TCP
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/level
operator: Lt
values:
- 2
- matchExpressions:
- key: kubernetes.io/level
operator: In
values:
- 2
How to realize the upgrading and upgrading of resources
In the hierarchical management system, infinite division of levels is supported, and a new level can be inserted between two levels only by defining an intermediate value without operating other resources; when upgrading or upgrading resources, only need to modify the corresponding Resource label
tag, you can upgrade or upgrade the resource. Of course, when upgrading or downgrading resources, we need to check the resources to ensure that when upgrading, the authority level of its upper-level resources must not be lower than the target level; at the same time, when downgrading, the authority level of its lower-level resources must not be higher than the target level. . When the upgrade and upgrade operating conditions are not met, the corresponding resources need to be adjusted accordingly.
Network isolation of Pods between different tiers
In the hierarchical system, we require that high-level Pods can access low-level Pods, but low-level Pods cannot access high-level Pods. How do we ensure network communication between Pods at different levels?
When the project does not enable network isolation, the network between Pods is interconnected, so a blacklist network policy will be added here.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: demo-ns
labels:
kubernetes.io/level: 2
spec:
podSelector: {}
policyTypes:
- Ingress
podSelector:{}
to all Pods in the project, preventing the inflow of all traffic.
Then allow traffic with a label level greater than the target level (here 2) to flow in (we do not limit the Ingress traffic).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: level-match-network-policy
namespace: demo-ns
labels:
kubernetes.io/level: 2
spec:
podSelector:
matchExpressions:
- key: kubernetes.io/level
operator: Gt
values:
- 2
policyTypes:
- Ingress
Summarize
KubeSphere solves the pain points of user building, deployment, management and observability, and its resources can be shared among multiple tenants. However, in a scenario where resources have permission levels, low-level resources can operate high-level resources, resulting in the problem of resource unauthorized management. In order to solve this problem, we have carried out transformation on the basis of KubeSphere to adapt to the hierarchical management between tenants and resources and between resources and resources. The network isolation between projects makes the management of resources more secure.
This article is published by OpenWrite , a multi-post blog platform!
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用
。你还可以使用@
来通知其他用户。