OpenKruise v1.0: Cloud-native application automation reaches a new peak

Cloud native application automation management suite, CNCF Sandbox project-OpenKruise, recently released a major version of v1.0.

OpenKruise [1] is an enhanced capability suite for Kubernetes, focusing on the deployment, upgrade, operation and maintenance, and stability protection of cloud-native applications. All functions are extended through standard methods such as CRD, and can be applied to any Kubernetes cluster of version 1.16 and above. A single helm command can complete the one-click deployment of Kruise, no more configuration is required.

Overall, the capabilities currently provided by OpenKruise are divided into several areas:

• application workload: an advanced deployment and release strategy for stateless, stateful, daemon and other types of applications, such as in-place upgrades, grayscale streaming releases, etc.
• Sidecar container management: supports independent definition of sidecar containers, and completes functions such as dynamic injection, independent in-place upgrade, and hot upgrade.
• enhances operation and maintenance capabilities: includes container restart in place, image pre-pull, and container startup sequence guarantee.
• application partition management: manages the deployment ratio, order, priority, etc. of applications in multiple partitions (availability zones, different models, etc.).
• application security protection: helps applications obtain higher security and availability protection on Kubernetes.

Version resolution

In the v1.0 major version, OpenKruise brings a variety of new features, and at the same time it also enhances and optimizes many existing functions.

The first thing to say is that starting from v1.0, OpenKruise has upgraded the version of resource configuration such as CRD/WehhookConfiguration from v1beta1 to v1 , so it can support clusters of Kubernetes v1.22 and above, but it also requires that the version of Kubernetes cannot be lower In v1.16.

The following is a brief introduction to some of the functions of v1.0. For the detailed ChangeLog list, please refer to the release instructions on OpenKruise Github and the official website documents.

Supports in-situ upgrade of environment variables

Author: @FillZpp [2]

OpenKruise has supported the "in-place upgrade" function since its early version, which is mainly applied to CloneSet and Advanced StatefulSet workloads. To put it simply, in-situ upgrade makes it unnecessary to delete or create new Pod objects during the upgrade process of the application, but to achieve the purpose of upgrading by modifying the container configuration in the Pod.

As shown in the figure above, only the fields in the Pod are modified during the in-place upgrade process, therefore:

1. It can avoid additional operations and costs scheduling, assigning IP, assigning, and mounting disk
2. Faster image pull, because open source reuses most of the layer layers of the old image, and only need to pull some layers that are changed by the new image.
3. When a container is upgraded in situ, the Pod's network, mount disk, and other containers in the Pod will not be affected and will still be running.

However, in the past, OpenKruise could only perform in-situ upgrades for the update of the image field in the Pod. For other fields, it can still only use reconstruction and upgrade similar to Deployment. For a long time, we have received many feedbacks from users, hoping to support in-situ upgrades to more fields such as env-due to the limitation of kube-apiserver, this is difficult to do.

After our unremitting efforts, OpenKruise finally supports the in-situ upgrade of env environment variables through the Downward API in the v1.0 version. For example, for the following CloneSet YAML, the user defines the configuration in the annotation and associates it with the corresponding env. When you subsequently modify the configuration, you only need to update the value in the annotation value, and Kruise will trigger an in-situ rebuild for all containers in the Pod that reference this annotation in the env, so that the new value configuration will take effect.

apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
  ...
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        app-config: "... the real env value ..."
    spec:
      containers:
      - name: app
        env:
        - name: APP_CONFIG
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['app-config']
  updateStrategy:
    type: InPlaceIfPossible

At the same time, in this version, we have also removed the previous imageID restriction on in-situ mirror upgrades, which means that two mirrors with the same imageID can be replaced and upgraded.

For specific usage, please refer to the document [3] .

Configure cross-namespace distribution

​Author: ​ @veophi [4]

In the scenario of cross-namespace distribution and synchronization of namespace-scoped resources such as Secret and ConfigMap, native kubernetes currently only supports one-by-one manual distribution and synchronization by users, which is very inconvenient.

Typical cases are:

• When users need to use the imagePullSecrets capability of SidecarSet, they must first repeatedly create corresponding Secrets in the relevant namespaces, and ensure the correctness and consistency of these Secret configurations.
• When users want to use ConfigMap to configure some general environment variables, they often need to distribute the ConfigMap in multiple namespaces, and subsequent modifications often require synchronization between multiple namespaces.

So, faced with these resources need to be distributed across namespaces and multiple synchronous scenario, we expect a more convenient tool to automate the distribution and synchronization to do it, for which we designed and implemented a new CRD --- ResourceDistribution .

ResourceDistribution currently supports the distribution and synchronization of two types of resources: Secret and ConfigMap

apiVersion: apps.kruise.io/v1alpha1
kind: ResourceDistribution
metadata:
  name: sample
spec:
  resource:
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: game-demo
    data:
      ...
  targets:
   namespaceLabelSelector:
      ...
    # or includedNamespaces, excludedNamespaces

As shown in the above YAML, ResourceDistribution is a type of cluster-scoped CRD, which is mainly composed of two fields, resource and targets. The resource field is used to describe the resources that users want to distribute, and the targets field is used to describe the resources that users want to distribute. Target namespace.

For specific usage, please refer to document [5] .

Container startup sequence control

Author: @Concurrensee [6]

For a Pod of Kubernetes, multiple containers may have dependencies. For example, the operation of the application process in container B depends on the application in container A. Therefore, there is a need for sequential relationships among multiple containers:

• Container A is started first, and container B can be started only after successful startup
• Container B exits first, and can stop container A after exiting

Generally speaking, the startup and exit sequence of Pod containers is managed by Kubelet. Kubernetes once had a KEP plan to add a type field to the container to identify the start and stop priority of different types of containers. However, due to sig-node's consideration of making too many changes to the existing code architecture, the KEP has been rejected at present.

Thus, OpenKruise provided named in V1.0 Container the Launch the Priority function, for controlling a plurality of containers Pod forced startup sequence:

1. For any Pod object, you only need to define apps.kruise.io/container-launch-priority: Ordered in annotations, then Kruise will follow the order of the containers in the Pod to ensure the serial launch of the containers.
2. If you want to customize the startup sequence of multiple containers in containers, add the KRUISE_CONTAINER_PRIORITY environment variable to the container env. The value is an integer in the range [-2147483647, 2147483647]. The higher the priority value of a container, the earlier it will be guaranteed to start.

For specific usage, please refer to document [7] .

kubectl-kruise command line tool

Author: @hantmac [8]

In the past, OpenKruise provided Kruise API definitions and client packages in languages ​​such as Go and Java through repositories such as kruise-api, client-java, etc., for users to introduce and use in their own applications. However, there are still many users who need to flexibly use the command line to manipulate workload resources in the test environment.

However, the rollout, set image and other commands provided by the native kubectl tool can only be applied to native workload types, such as Deployment, StatefulSet, and cannot recognize the workload types extended in OpenKruise.

Therefore, OpenKruise recently provides the kubectl-kruise command-line tool, which is a standard plug-in of kubectl and provides many functions suitable for OpenKruise workload.

# rollout undo cloneset
$ kubectl kruise rollout undo cloneset/nginx
#  rollout status advanced statefulset
$ kubectl kruise rollout status statefulsets.apps.kruise.io/sts-demo
# set image of a cloneset
$ kubectl kruise set image cloneset/nginx busybox=busybox nginx=nginx:1.9.1

Please refer to the document [9] specific usage.

Improvement and optimization of the rest of the functions

CloneSet:

• Support streaming expansion through the scaleStrategy.maxUnavailable strategy
• Stable revision judges the logic change, when all Pod versions are consistent with updateRevision, it is marked as currentRevision

WorkloadSpread:

• Support to take over the stock Pod into the matching subset group
• Optimize the update and retry logic of webhook during Pod injection

Advanced DaemonSet:

• Support for in-situ upgrade of Daemon Pod
• Introduce progressive annotation to choose whether to restrict Pod creation by partition

SidecarSet:

• Solve the problem of SidecarSet filtering and blocking inactive Pod
• Added the SourceContainerNameFrom and EnvNames fields to transferenv to solve the problem of inconsistency in container names and redundancy in the case of a large number of envs

PodUnavailableBudget:

• Added "skip protection" logo
• The PodUnavailableBudget controller pays attention to the changes in the replicas of the workload

NodeImage:

• Add the --nodeimage-creation-delay parameter, and wait for the new Node ready by default to create a NodeImage synchronously after a period of time

UnitedDeployment:

• Solve the problem that the length of Pod NodeSelectorTerms is 0 when NodeSelectorTerms is nil

Other optimization:

• kruise-daemon uses protobuf protocol to operate Pod resources
• Expose cache resync as a command line parameter, and set the default value to 0 in the chart
• Solve the problem of http checker refresh when certs are updated
• Remove the dependency on forked controller-tools and use native controller-tools with marker annotations instead

Community Involvement

You are very welcome to join us through Github/Slack/Dingding/WeChat to participate in the OpenKruise open source community. Do you already have something you want to communicate with our community? You can share your voice on our community bi-weekly meeting [10] , or participate in the discussion through the following channels:

• Join the community Slack channel [11] (English)
• Join the community nail group: search group number 23330762 (Chinese)
• Join the community WeChat group: add user openkruise and let the robot pull you into the group (Chinese)

Reference

[1]  OpenKruise:

 ​​https://openkruise.io​​

[2]  @FillZpp:

 ​​https://github.com/FillZpp​​

[3] Documentation:

/docs/core-concepts/inplace-update

[4]  @veophi:

 ​​https://github.com/veophi​​

[5] Documentation:

/docs/user-manuals/resourcedistribution

[6]  @Concurrensee:

 ​​https://github.com/Concurrensee​​

[7] Documentation:

  /docs/user-manuals/containerlaunchpriority

[8]  @hantmac: 

​​https://github.com/hantmac​​

[9] Documentation:

  /docs/cli-tool/kubectl-plugin

[10] Community biweekly meeting:

​​https://shimo.im/docs/gXqmeQOYBehZ4vqo​​

[11]  Slack channel:

 ​​https://kubernetes.slack.com/channels/openkruise​​

Stamp original , see OpenKruise project official home page of the document!