Tencent Cloud Cloud Native Hybrid Cloud-TKE Release Version

background

The TKE release (TKE Kubernetes Distro) is a K8S release released by Tencent Cloud TKE to help users create a safe and reliable K8S cluster. Users can rely on the TKE release version to run K8S services that are exactly the same as TKE on self-built or hosted computer rooms, physical machines or virtual machines. The TKE release cluster can be seamlessly integrated with Tencent Cloud TKE to form a hybrid cloud. Users can extend their business under the cloud to the cloud through the TKE release cluster, and use Tencent Cloud's TKE, EKS and other cloud service elastic capabilities to provide strong resource guarantees for their business. The current TKE release has been open sourced on GitHub: [[ https://github.com/tkestack/tke-k8s-distro]](https://github.com/tkestack/tke-k8s-distro])( https://github.com/tkestack/tke-k8s-distro ).

Usage scenarios and positioning

In the hybrid cloud scenario, the K8S provided by different cloud service providers are not exactly the same. And users can only use the community version of K8S in environments other than cloud service providers, and any small differences in the operating environment may cause business failures. Therefore, how to ensure the consistency of basic components in multiple environments becomes particularly important. The TKE release is a good choice to solve this problem. Users don't need to pay attention to the differences in the capabilities of K8S in different environments and fix the problems encountered in K8S by themselves.

Relying on the TKE release version, users can now compile and build the same K8S version as Tencent Cloud TKE. This means that users can now manually deploy reliable and secure clusters without the need to continuously test and track K8S updates, dependencies, and security patches. Each TKE release follows the compatibility process of Tencent Cloud TKE and the new version of the K8S community standard.

The TKE release version expands K8S on the basis of ensuring compatibility, and keeps the version consistent with Tencent Cloud TKE service. Users can deploy the TKE release version on their IDC or hybrid cloud, and use the reliable and secure K8S service that has been verified on a large scale by enterprise users.

Each version of the TKE release will pass the official conformance test of the K8S community to ensure compatibility. At the same time, the source code is provided in the form of patch, and a build tool is provided to help users compile. The TKE release currently supports the v1.20 version.

Our advantage

Large-scale production cluster verification

The TKE release provides the same installable version and open source code as Tencent Cloud TKE, and its functions and stability have been tempered by a large number of enterprise users, public clouds and self-developed clouds. Users can use the provided source code and compilation tools to build and deploy.

Seamlessly integrate public cloud TKE

The TKE release version can support users to run K8S services that are completely consistent with Tencent Cloud TKE on self-built or hosted computer rooms, physical or virtual machines. And it can be seamlessly integrated with Tencent Cloud TKE to form a hybrid cloud cluster.

Longer support period

The TKE release version has a longer support period than the community version. After the community version stops supporting, the TKE release will continue to be supported, including important issues and fixes for security vulnerabilities.

More practical ability enhancement

The TKE release version combines Tencent's own business characteristics and experience to achieve enhanced capabilities for some scenarios (flexible expansion, off-line mixing, resource isolation, etc.). And the TKE release version closely follows the trend of the community, leading or deeply participating in the design and implementation of the community KEP. For practical value KEP will precede community support, allowing users to enjoy the advancement of cloud native technology in advance.

Function enhancement

Support elastic expansion to Tencent Cloud EKS service

EKS is Tencent Cloud's elastic container service. Users can deploy workloads without purchasing nodes, which is very suitable for sudden and other temporary expansion needs. When temporary expansion is needed, the workload can be quickly expanded from the TKE release cluster to the EKS on the cloud in seconds to deal with sudden and temporary traffic, improve resource utilization, and reduce operation and maintenance and capital costs.

Support dynamic modification of kube-controller-manager log level

In the operation and maintenance of K8S production clusters, we generally set a lower log level (0~2), and we need to increase the log level when troubleshooting. The TKE release version implements the function of dynamically modifying the log level, thereby avoiding the loss of critical logs due to component restarts. At present, the official version of K8S supports kube-apiserver, kubelet and kube-scheduler settings. The TKE release additionally implements the dynamic log level adjustment of the kube-controller-manager component.

This feature has been submitted to the community: https://github.com/kubernetes/kubernetes/pull/98262

Support Memory QoS with cgroups v2 feature (in progress)

Memory QoS with cgroups v2 is a memory QoS function designed and implemented by the TKE team. It uses memory.min/memory.high v2 memory controller to provide a full range of memory protection for pod/container/node.

At present, the KEP has been accepted by the K8S community, and the alpha version is expected to be implemented in v1.22. The TKE release version will support this feature before the community to provide better memory protection for user Pods, cluster nodes, etc.

Memory QoS with cgroups v2：https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2570-memory-qos

Support TencentOS QoS features (in progress)

TencentOS is a Linux operating system developed by Tencent for cloud scenarios. It provides native priority support and resource isolation enhancements specifically for container scenarios, including CPU, memory, disk IO, and network IO. The TKE release version has built-in support for TencentOS QoS features, and implements K8S resource isolation and QoS hierarchical offload to TencentOS. This feature is under development and is expected to be supported in the next version.

TencentOS：https://github.com/Tencent/TencentOS-kernel

Stability enhancement

Based on a large amount of production experience, the TKE release version fixes many production-level bugs and supports the stable operation of tens of millions of core clusters in Tencent. The main bugfixes include:

1. Fix the problem of missing cluster monitoring indicators when using containerd

In a cluster where containerd is used as a container runtime, kubelet does not correctly set the container name for collecting indicators, which leads to failure to be classified and analyzed. The TKE release version fixes this problem. Submit a community PR: https://github.com/kubernetes/kubernetes/pull/90260

2. Fix the problem that the Pod created and scheduled to the same node immediately after deleting it may cause the problem to fail to mount successfully

The K8S statefulset pod is recreated after being deleted. If the pod is scheduled to the same node, it will fail to start because the volume fails to mount. The TKE release version fixes this problem. Submit a community PR: https://github.com/kubernetes/kubernetes/pull/72914

3. Fix the issue that creating containers under CentOS will cause cgroup leaks

The TKE release version disabled kernel memory accouting to avoid cgroup leaks.

4. Fix kubectl describe deployment <xxx> NewReplicaSet displayed as \<none\>

kubectl describe will sort the volumes after obtaining the deployment object, which sometimes results in the failure to match the latest replicaset. The TKE release version fixes this problem, submit a community PR: https://github.com/kubernetes/kubernetes/pull/97752

​

5. When the Pod container image has multiple tags, the Pod status image tag will not match

When a Pod container image has multiple tags, it will cause the pod spec container image tag to be inconsistent with the kubelet report. TKE release version backport community PR fixes this problem: https://github.com/kubernetes/kubernetes/pull/94833

​

6. Fix the aws credential provider causing a 20s delay in kubelet startup

AWS credential provider will try to connect to the AWS metadata service during initialization, which will cause up to 20s startup delay for non-AWS clusters. TKE release version reported bug https://github.com/kubernetes/kubernetes/issues/92162, and backport community PR https://github.com/kubernetes/kubernetes/pull/93260

7. Fix the issue of pod exit caused by lxcfs upgrade on

The K8S cluster installs the lower version of lxcfs by default under Ubuntu 16.04. After upgrading to lxcfs, the pod will not run normally. The reason is that the lower version of lxcfs mounts cgroupfs, and kubelet will use lxcfs to mount cgroupfs when it starts, instead of the system /sys/fs/cgroup. After upgrading the new version of lxcfs, the old cgroupfs will be unhooked, causing the kubelet to fail to operate the pod cgroup. The TKE release fixes this problem.

How to use the TKE distribution

The TKE release version provides installation tool scripts to help users automatically compile and build the release version image, which greatly reduces the threshold for using the TKE release version.

The compilation and build process includes:

1. Pull patch code

git clone [https://github.com/tkestack/tke-k8s-distro](https://github.com/tkestack/tke-k8s-distro)

2. Compile components

make

or

make \<release\>

Only version 1.20 is currently supported.

3. Component output

During the compilation process, the source code path is _src/<release> , and the compiled output path is _output/<release> . The components include kubeadm , kube-apiserver , kube-controller-manager , kubectl , kubelet , kube-proxy , kube-scheduler .

Next step

The launch of the TKE release makes it possible to integrate IDC offline and TKE on Tencent Cloud. We hope that the TKE release will become the cornerstone of hybrid cloud and multi-cloud in the future, allowing users to enjoy the same experience as TKE on the cloud anytime and anywhere in a hybrid cloud environment.

The TKE release will operate in an open source manner in the future, and users can provide any feedback through GitHub, not limited to Issue and PR.

​

GitHub：[[https://github.com/tkestack/tke-k8s-distro]](https://github.com/tkestack/tke-k8s-distro])(https://github.com/tkestack/tke-k8s-distro)

[Tencent Cloud Native] Yunshuo new products, Yunyan new technology, Yunyou Xinhuo, Yunxiang information, scan the QR code to follow the public account of the same name, and get more dry goods in time! !