Implement Chaos Engineering in Kubernetes-Chaos Mesh® Principle Analysis and Control Surface Development

Chaos Mesh® is a Chaos Engineering system developed by the PingCAP company behind TiDB and running on Kubernetes. In short, Chaos Mesh® creates chaos (simulated failure) in the cluster through the "privileged" container running in the K8s cluster and based on the test scenarios in the CRD resource [1].

This article explores the practice of chaos engineering on Kubernetes clusters, understands the working principle of Chaos Mesh® based on source code analysis, and explains how to develop the control plane of Chaos Mesh® with code examples. If you lack the basic knowledge and want to have a macro understanding of the architecture of Chaos Mesh®, please refer to the link in the note at the end of the article.

The test code for this article is located in the mayocream/chaos-mesh-controlpanel-demo repository.

How to create chaos

Chaos Mesh® is a powerful tool for implementing chaos engineering on Kubernetes, how does it work?

Privileged Mode

As mentioned above, Chaos Mesh® runs Kubernetes privileged containers to cause failures. The Pod running in Daemon Set mode authorizes the Capabilities of the container when it is running.

apiVersion:apps/v1kind:DaemonSetspec:  template:    metadata:...    spec:      containers:        -name:chaos-daemon          securityContext:            {{-if.Values.chaosDaemon.privileged}}            privileged:true            capabilities:              add:                -SYS_PTRACE            {{-else}}            capabilities:              add:                -SYS_PTRACE                -NET_ADMIN                -MKNOD                -SYS_CHROOT                -SYS_ADMIN                -KILL                # CAP_IPC_LOCK is used to lock memory                -IPC_LOCK            {{-end}}

These Linux capability words are used to grant container privileges to create and access the /dev/fuse FUSE pipe [2] (FUSE is the Linux user space file system interface, which enables unprivileged users to create their own file system without editing the kernel code ).
Refer to #1109 Pull Request, Daemon Set program uses CGO to call Linux makedev function to create FUSE pipeline.

// #include <sys/sysmacros.h>// #include <sys/types.h>// // makedev is a macro, so a wrapper is needed// dev_t Makedev(unsigned int maj, unsigned int min) {//   return makedev(maj, min);// }// EnsureFuseDev ensures /dev/fuse exists. If not, it will create onefunc EnsureFuseDev() { if _, err := os.Open("/dev/fuse"); os.IsNotExist(err) {  // 10, 229 according to https://www.kernel.org/doc/Documentation/admin-guide/devices.txt  fuse := C.Makedev(10, 229)  syscall.Mknod("/dev/fuse", 0o666|syscall.S_IFCHR, int(fuse)) }}

At the same time, in #1103 PR, Chaos Daemon enabled privileged mode by default, that is, set privileged: true in the securityContext of the container.

Kill pod

PodKill, PodFailure, and ContainerKill all belong to the PodChaos category. PodKill kills Pods randomly.
The concrete realization of PodKill is actually to send Kill command by calling API Server.

import ( "context" v1 "k8s.io/api/core/v1" "sigs.k8s.io/controller-runtime/pkg/client")type Impl struct { client.Client}func (impl *Impl) Apply(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) { ... err = impl.Get(ctx, namespacedName, &pod) if err != nil {  // TODO: handle this error  return v1alpha1.NotInjected, err } err = impl.Delete(ctx, &pod, &client.DeleteOptions{  GracePeriodSeconds: &podchaos.Spec.GracePeriod, // PeriodSeconds has to be set specifically }) ... return v1alpha1.Injected, nil}

The GracePeriodSeconds parameter is suitable for K8s to terminate Pod forcibly. For example, when we need to delete a pod quickly, we use the kubectl delete pod --grace-period=0 --force command.
PodFailure is to replace the mirror in the Pod with the wrong mirror through the Patch Pod object resource. Chaos only modified the image fields of containers and initContainers. This is also because most fields of Pod cannot be changed. For details, please refer to Pod update and replacement.

func (impl *Impl) Apply(ctx context.Context, index int, records []*v1alpha1.Record, obj v1alpha1.InnerObject) (v1alpha1.Phase, error) { ... pod := origin.DeepCopy() for index := range pod.Spec.Containers {  originImage := pod.Spec.Containers[index].Image  name := pod.Spec.Containers[index].Name  key := annotation.GenKeyForImage(podchaos, name, false)  if pod.Annotations == nil {   pod.Annotations = make(map[string]string)  }  // If the annotation is already existed, we could skip the reconcile for this container  if _, ok := pod.Annotations[key]; ok {   continue  }  pod.Annotations[key] = originImage  pod.Spec.Containers[index].Image = config.ControllerCfg.PodFailurePauseImage } for index := range pod.Spec.InitContainers {  originImage := pod.Spec.InitContainers[index].Image  name := pod.Spec.InitContainers[index].Name  key := annotation.GenKeyForImage(podchaos, name, true)  if pod.Annotations == nil {   pod.Annotations = make(map[string]string)  }  // If the annotation is already existed, we could skip the reconcile for this container  if _, ok := pod.Annotations[key]; ok {   continue  }  pod.Annotations[key] = originImage  pod.Spec.InitContainers[index].Image = config.ControllerCfg.PodFailurePauseImage } err = impl.Patch(ctx, pod, client.MergeFrom(&origin)) if err != nil {  // TODO: handle this error  return v1alpha1.NotInjected, err } return v1alpha1.Injected, nil}

The default container image used to cause failure is gcr.io/google-containers/pause:latest. If it is used in a domestic environment, there is a high probability that it will be unacceptable. You can replace gcr.io with registry.aliyuncs.com.
ContainerKill is different from PodKill and PodFailure. The latter two control the Pod life cycle through the K8s API Server, while ContainerKill is completed by the Chaos Daemon program running on the cluster Node. Specifically, ContainerKill uses Chaos Controller Manager to run the client to initiate grpc calls to Chaos Daemon.

func (b *ChaosDaemonClientBuilder) Build(ctx context.Context, pod *v1.Pod) (chaosdaemonclient.ChaosDaemonClientInterface, error) { ... daemonIP, err := b.FindDaemonIP(ctx, pod) if err != nil {  return nil, err } builder := grpcUtils.Builder(daemonIP, config.ControllerCfg.ChaosDaemonPort).WithDefaultTimeout() if config.ControllerCfg.TLSConfig.ChaosMeshCACert != "" {  builder.TLSFromFile(config.ControllerCfg.TLSConfig.ChaosMeshCACert, config.ControllerCfg.TLSConfig.ChaosDaemonClientCert, config.ControllerCfg.TLSConfig.ChaosDaemonClientKey) } else {  builder.Insecure() } cc, err := builder.Build() if err != nil {  return nil, err } return chaosdaemonclient.New(cc), nil}

When sending a command to Chaos Daemon, the corresponding client will be created based on the Pod information. For example, if you want to control a Pod on a Node, you will get the ClusterIP of the Node where the Pod is located to create a client. If the TLS certificate configuration exists, Controller Manager will add the TLS certificate for the client.
If Chaos Daemon has a TLS certificate when it starts, it will attach the certificate to enable grpcs. The TLS verification configuration RequireAndVerifyClientCert indicates that mutual TLS authentication (mTLS) is enabled.

func newGRPCServer(containerRuntime string, reg prometheus.Registerer, tlsConf tlsConfig) (*grpc.Server, error) { ... if tlsConf != (tlsConfig{}) {  caCert, err := ioutil.ReadFile(tlsConf.CaCert)  if err != nil {   return nil, err  }  caCertPool := x509.NewCertPool()  caCertPool.AppendCertsFromPEM(caCert)  serverCert, err := tls.LoadX509KeyPair(tlsConf.Cert, tlsConf.Key)  if err != nil {   return nil, err  }  creds := credentials.NewTLS(&tls.Config{   Certificates: []tls.Certificate{serverCert},   ClientCAs:    caCertPool,   ClientAuth:   tls.RequireAndVerifyClientCert,  })  grpcOpts = append(grpcOpts, grpc.Creds(creds)) } s := grpc.NewServer(grpcOpts...) grpcMetrics.InitializeMetrics(s) pb.RegisterChaosDaemonServer(s, ds) reflection.Register(s) return s, nil}

Chaos Daemon provides the following grpc calling interfaces:

// ChaosDaemonClient is the client API for ChaosDaemon service.//// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.type ChaosDaemonClient interface { SetTcs(ctx context.Context, in *TcsRequest, opts ...grpc.CallOption) (*empty.Empty, error) FlushIPSets(ctx context.Context, in *IPSetsRequest, opts ...grpc.CallOption) (*empty.Empty, error) SetIptablesChains(ctx context.Context, in *IptablesChainsRequest, opts ...grpc.CallOption) (*empty.Empty, error) SetTimeOffset(ctx context.Context, in *TimeRequest, opts ...grpc.CallOption) (*empty.Empty, error) RecoverTimeOffset(ctx context.Context, in *TimeRequest, opts ...grpc.CallOption) (*empty.Empty, error) ContainerKill(ctx context.Context, in *ContainerRequest, opts ...grpc.CallOption) (*empty.Empty, error) ContainerGetPid(ctx context.Context, in *ContainerRequest, opts ...grpc.CallOption) (*ContainerResponse, error) ExecStressors(ctx context.Context, in *ExecStressRequest, opts ...grpc.CallOption) (*ExecStressResponse, error) CancelStressors(ctx context.Context, in *CancelStressRequest, opts ...grpc.CallOption) (*empty.Empty, error) ApplyIOChaos(ctx context.Context, in *ApplyIOChaosRequest, opts ...grpc.CallOption) (*ApplyIOChaosResponse, error) ApplyHttpChaos(ctx context.Context, in *ApplyHttpChaosRequest, opts ...grpc.CallOption) (*ApplyHttpChaosResponse, error) SetDNSServer(ctx context.Context, in *SetDNSServerRequest, opts ...grpc.CallOption) (*empty.Empty, error)}

Network failure

From the initial #41 PR, it can be clearly understood that the network error injection of Chaos Mesh® is processed by calling the pbClient.SetNetem method, encapsulating the parameters into a request, and giving it to Chaos Daemon on Node for processing.
(Note: This is the code at the beginning of 2019. With the development of the project, the functions in the code have been scattered into different files)

func (r *Reconciler) applyPod(ctx context.Context, pod *v1.Pod, networkchaos *v1alpha1.NetworkChaos) error { ... pbClient := pb.NewChaosDaemonClient(c) containerId := pod.Status.ContainerStatuses[0].ContainerID netem, err := spec.ToNetem() if err != nil {  return err } _, err = pbClient.SetNetem(ctx, &pb.NetemRequest{  ContainerId: containerId,  Netem:       netem, }) return err}

At the same time, in the pkg/chaosdaemon package, we can see how Chaos Daemon processes requests.

func (s *Server) SetNetem(ctx context.Context, in *pb.NetemRequest) (*empty.Empty, error) { log.Info("Set netem", "Request", in) pid, err := s.crClient.GetPidFromContainerID(ctx, in.ContainerId) if err != nil {  return nil, status.Errorf(codes.Internal, "get pid from containerID error: %v", err) } if err := Apply(in.Netem, pid); err != nil {  return nil, status.Errorf(codes.Internal, "netem apply error: %v", err) } return &empty.Empty{}, nil}// Apply applies a netem on eth0 in pid related namespacefunc Apply(netem *pb.Netem, pid uint32) error { log.Info("Apply netem on PID", "pid", pid) ns, err := netns.GetFromPath(GenNetnsPath(pid)) if err != nil {  log.Error(err, "failed to find network namespace", "pid", pid)  return errors.Trace(err) } defer ns.Close() handle, err := netlink.NewHandleAt(ns) if err != nil {  log.Error(err, "failed to get handle at network namespace", "network namespace", ns)  return err } link, err := handle.LinkByName("eth0") // TODO: check whether interface name is eth0 if err != nil {  log.Error(err, "failed to find eth0 interface")  return errors.Trace(err) } netemQdisc := netlink.NewNetem(netlink.QdiscAttrs{  LinkIndex: link.Attrs().Index,  Handle:    netlink.MakeHandle(1, 0),  Parent:    netlink.HANDLE_ROOT, }, ToNetlinkNetemAttrs(netem)) if err = handle.QdiscAdd(netemQdisc); err != nil {  if !strings.Contains(err.Error(), "file exists") {   log.Error(err, "failed to add Qdisc")   return errors.Trace(err)  } } return nil}

Finally, use the vishvananda/netlink library to operate the Linux network interface to complete the work.
It can be known here that NetworkChaos is a chaos type, which manipulates the Linux host network to create chaos, including tools such as iptables and ipset.
In Chaos Daemon's Dockerfile, you can see the Linux tool chain it depends on:

RUN apt-get update && \     apt-get install -y tzdata iptables ipset stress-ng iproute2 fuse util-linux procps curl && \    rm -rf /var/lib/apt/lists/*

pressure test

Chaos of the StressChaos type is also implemented by Chaos Daemon. After the Controller Manager calculates the rules, it sends tasks to the specific Daemon. The assembled parameters are as follows, these parameters will be combined into command execution parameters, appended to the stress-ng command and executed [3].

// Normalize the stressors to comply with stress-ngfunc (in *Stressors) Normalize() (string, error) { stressors := "" if in.MemoryStressor != nil && in.MemoryStressor.Workers != 0 {  stressors += fmt.Sprintf(" --vm %d --vm-keep", in.MemoryStressor.Workers)  if len(in.MemoryStressor.Size) != 0 {   if in.MemoryStressor.Size[len(in.MemoryStressor.Size)-1] != '%' {    size, err := units.FromHumanSize(string(in.MemoryStressor.Size))    if err != nil {     return "", err    }    stressors += fmt.Sprintf(" --vm-bytes %d", size)   } else {    stressors += fmt.Sprintf(" --vm-bytes %s",     in.MemoryStressor.Size)   }  }  if in.MemoryStressor.Options != nil {   for _, v := range in.MemoryStressor.Options {    stressors += fmt.Sprintf(" %v ", v)   }  } } if in.CPUStressor != nil && in.CPUStressor.Workers != 0 {  stressors += fmt.Sprintf(" --cpu %d", in.CPUStressor.Workers)  if in.CPUStressor.Load != nil {   stressors += fmt.Sprintf(" --cpu-load %d",    *in.CPUStressor.Load)  }  if in.CPUStressor.Options != nil {   for _, v := range in.CPUStressor.Options {    stressors += fmt.Sprintf(" %v ", v)   }  } } return stressors, nil}

In the Chaos Daemon server processing function, the official Go package os/exec is called to execute the command, and it is meaningless to paste a large piece of code. For details, you can read the pkg/chaosdaemon/stress_server_linux.go file. The file with the same name also ends with darwin, presumably for the convenience of development and debugging on macOS.
The code uses the shirou/gopsutil package to obtain the PID process status, and reads the standard output such as stdout and stderr. I have seen this processing mode in hashicorp/go-plugin, and go-plugin has done a better job in this regard. It is mentioned in my other article Dkron source code analysis [4].

IO injection

At the beginning, it was mentioned that Chaos Mesh® uses a privileged container to mount the FUSE device /dev/fuse on the host.
Seeing this, I definitely think that Chaos Mesh® uses Mutating access controller to inject Sidecar containers, mount FUSE devices, or modify Pod Volumes Mount and other configurations, and then its implementation is different from intuitively.
Look carefully at #826 PR, this PR introduces a new implementation of IOChaos, avoiding the use of Sidecar injection, and using Chaos Daemon to directly manipulate the Linux namespace through the underlying commands of the runc container, and run the chaos-mesh/toda FUSE developed by Rust Program (using JSON-RPC 2.0 protocol communication) for container IO chaos injection.
Pay attention to the new IOChaos implementation, it will not modify the Pod resources. When the IOChaos chaos experiment definition is created, for each Pod filtered by the selector (selector field), a corresponding PodIoChaos resource will be created, and the owner of PodIoChaos will be referenced ( Owner Reference) is the Pod. PodIoChaos will also be added with a group of Finalizers to release PodIoChaos resources before being deleted.

// Apply implements the reconciler.InnerReconciler.Applyfunc (r *Reconciler) Apply(ctx context.Context, req ctrl.Request, chaos v1alpha1.InnerObject) error { iochaos, ok := chaos.(*v1alpha1.IoChaos) if !ok {  err := errors.New("chaos is not IoChaos")  r.Log.Error(err, "chaos is not IoChaos", "chaos", chaos)  return err } source := iochaos.Namespace + "/" + iochaos.Name m := podiochaosmanager.New(source, r.Log, r.Client) pods, err := utils.SelectAndFilterPods(ctx, r.Client, r.Reader, &iochaos.Spec) if err != nil {  r.Log.Error(err, "failed to select and filter pods")  return err } r.Log.Info("applying iochaos", "iochaos", iochaos) for _, pod := range pods {  t := m.WithInit(types.NamespacedName{   Name:      pod.Name,   Namespace: pod.Namespace,  })  // TODO: support chaos on multiple volume  t.SetVolumePath(iochaos.Spec.VolumePath)  t.Append(v1alpha1.IoChaosAction{   Type: iochaos.Spec.Action,   Filter: v1alpha1.Filter{    Path:    iochaos.Spec.Path,    Percent: iochaos.Spec.Percent,    Methods: iochaos.Spec.Methods,   },   Faults: []v1alpha1.IoFault{    {     Errno:  iochaos.Spec.Errno,     Weight: 1,    },   },   Latency:          iochaos.Spec.Delay,   AttrOverrideSpec: iochaos.Spec.Attr,   Source:           m.Source,  })  key, err := cache.MetaNamespaceKeyFunc(&pod)  if err != nil {   return err  }  iochaos.Finalizers = utils.InsertFinalizer(iochaos.Finalizers, key) } r.Log.Info("commiting updates of podiochaos") err = m.Commit(ctx) if err != nil {  r.Log.Error(err, "fail to commit")  return err } r.Event(iochaos, v1.EventTypeNormal, utils.EventChaosInjected, "") return nil}

In the controller of the PodIoChaos resource, the Controller Manager encapsulates the resource into parameters and calls the Chaos Daemon interface for actual processing.

// Apply flushes io configuration on podfunc (h *Handler) Apply(ctx context.Context, chaos *v1alpha1.PodIoChaos) error { h.Log.Info("updating io chaos", "pod", chaos.Namespace+"/"+chaos.Name, "spec", chaos.Spec)    ... res, err := pbClient.ApplyIoChaos(ctx, &pb.ApplyIoChaosRequest{  Actions:     input,  Volume:      chaos.Spec.VolumeMountPath,  ContainerId: containerID,  Instance:  chaos.Spec.Pid,  StartTime: chaos.Spec.StartTime, }) if err != nil {  return err } chaos.Spec.Pid = res.Instance chaos.Spec.StartTime = res.StartTime chaos.OwnerReferences = []metav1.OwnerReference{  {   APIVersion: pod.APIVersion,   Kind:       pod.Kind,   Name:       pod.Name,   UID:        pod.UID,  }, } return nil}

In the code file pkg/chaosdaemon/iochaos_server.go that processes IOChaos in Chaos Daemon, the container needs to be injected into a FUSE program. Through #2305 Issue, you can know that /usr/local/bin/nsexec -l -p proc/ 119186/ns/pid -m proc/119186/ns/mnt - usr/local/bin/toda --path tmp --verbose info command to run the toda program under a specific Linux namespace (Namespace), that is, with Pods are in the same namespace.

func (s *DaemonServer) ApplyIOChaos(ctx context.Context, in *pb.ApplyIOChaosRequest) (*pb.ApplyIOChaosResponse, error) { ... pid, err := s.crClient.GetPidFromContainerID(ctx, in.ContainerId) if err != nil {  log.Error(err, "error while getting PID")  return nil, err } args := fmt.Sprintf("--path %s --verbose info", in.Volume) log.Info("executing", "cmd", todaBin+" "+args) processBuilder := bpm.DefaultProcessBuilder(todaBin, strings.Split(args, " ")...).  EnableLocalMnt().  SetIdentifier(in.ContainerId) if in.EnterNS {  processBuilder = processBuilder.SetNS(pid, bpm.MountNS).SetNS(pid, bpm.PidNS) }    ...    // JSON RPC 调用 client, err := jrpc.DialIO(ctx, receiver, caller) if err != nil {  return nil, err } cmd := processBuilder.Build()    procState, err := s.backgroundProcessManager.StartProcess(cmd) if err != nil {  return nil, err } ...}

The following piece of code finally builds the running commands, and these commands are the namespace isolation implementation at the bottom of runc [5]:

// GetNsPath returns corresponding namespace pathfunc GetNsPath(pid uint32, typ NsType) string { return fmt.Sprintf("%s/%d/ns/%s", DefaultProcPrefix, pid, string(typ))}// SetNS sets the namespace of the processfunc (b *ProcessBuilder) SetNS(pid uint32, typ NsType) *ProcessBuilder { return b.SetNSOpt([]nsOption{{  Typ:  typ,  Path: GetNsPath(pid, typ), }})}// Build builds the processfunc (b *ProcessBuilder) Build() *ManagedProcess { args := b.args cmd := b.cmd if len(b.nsOptions) > 0 {  args = append([]string{"--", cmd}, args...)  for _, option := range b.nsOptions {   args = append([]string{"-" + nsArgMap[option.Typ], option.Path}, args...)  }  if b.localMnt {   args = append([]string{"-l"}, args...)  }  cmd = nsexecPath }    ...}

Control plane

Chaos Mesh® is an open source chaos engineering system that is open sourced under the Apache 2.0 protocol. After the above analysis, it is known that its capabilities are rich and its ecology is good. The maintenance team developed a user-mode file system (FUSE) chaos-mesh/ around the chaos system. toda, CoreDNS chaos plug-in chaos-mesh/k8s_dns_chaos, BPF-based kernel error injection chaos-mesh/bpfki, etc.
If I want to build an end-user-oriented chaos engineering platform, the following code should be implemented on the server side. The example is only a practice, not a best practice. If you want to see the development practice of the Real World platform, you can refer to the official Dashboard of Chaos Mesh®, which uses the uber-go/fx dependency injection framework and the manager of the controller runtime. model.

Division of functions

Although the title here is the control plane, but looking at the above Chaos Mesh® work flow chart, in fact, all we need to do is to implement a server that sends YAML to the Kubernetes API. The behavior of complex rule verification and rule delivery to Chaos Daemon is Completed by Chaos Controller Manager. If you want to use it with your own platform, you only need to connect to the process of creating CRD resources.
Let's take a look at the official example given by PingCAP:

import (    "context"    "github.com/pingcap/chaos-mesh/api/v1alpha1"    "sigs.k8s.io/controller-runtime/pkg/client")func main() {    ...    delay := &chaosv1alpha1.NetworkChaos{        Spec: chaosv1alpha1.NetworkChaosSpec{...},    }    k8sClient := client.New(conf, client.Options{ Scheme: scheme.Scheme })    k8sClient.Create(context.TODO(), delay)    k8sClient.Delete(context.TODO(), delay)}

Chaos Mesh® has provided APIs corresponding to all CRD resource definitions. We use the controller-runtime developed by Kubernetes API Machinery SIG to simplify the interaction with the Kubernetes API.

Implement chaos

For example, we want to create a PodKill resource through program call. After the resource is sent to the Kubernetes API Server, it will be admitted to the controller through the Validating of Chaos Controller Manager for data verification. If the data format verification fails, it will be created An error is returned when. For specific parameters, please refer to the official documentation to create experiments using YAML configuration files.
NewClient has created a K8s API Client, you can refer to the client creation example.

package mainimport ( "context" "controlpanel" "log" "github.com/chaos-mesh/chaos-mesh/api/v1alpha1" "github.com/pkg/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1")func applyPodKill(name, namespace string, labels map[string]string) error { cli, err := controlpanel.NewClient() if err != nil {  return errors.Wrap(err, "create client") } cr := &v1alpha1.PodChaos{  ObjectMeta: metav1.ObjectMeta{   GenerateName: name,   Namespace:    namespace,  },  Spec: v1alpha1.PodChaosSpec{   Action: v1alpha1.PodKillAction,   ContainerSelector: v1alpha1.ContainerSelector{    PodSelector: v1alpha1.PodSelector{     Mode: v1alpha1.OnePodMode,     Selector: v1alpha1.PodSelectorSpec{      Namespaces:     []string{namespace},      LabelSelectors: labels,     },    },   },  }, } if err := cli.Create(context.Background(), cr); err != nil {  return errors.Wrap(err, "create podkill") } return nil}

The log output of the running program is:

I1021 00:51:55.225502   23781 request.go:665] Waited for 1.033116256s due to client-side throttling, not priority and fairness, request: GET:https://***2021/10/21 00:51:56 apply podkill

View the status of PodKill resources through kubectl:

$ k describe podchaos.chaos-mesh.org -n dev podkillvjn77Name:         podkillvjn77Namespace:    devLabels:       <none>Annotations:  <none>API Version:  chaos-mesh.org/v1alpha1Kind:         PodChaosMetadata:  Creation Timestamp:  2021-10-20T16:51:56Z  Finalizers:    chaos-mesh/records  Generate Name:     podkill  Generation:        7  Resource Version:  938921488  Self Link:         /apis/chaos-mesh.org/v1alpha1/namespaces/dev/podchaos/podkillvjn77  UID:               afbb40b3-ade8-48ba-89db-04918d89fd0bSpec:  Action:        pod-kill  Grace Period:  0  Mode:          one  Selector:    Label Selectors:      app:  nginx    Namespaces:      devStatus:  Conditions:    Reason:      Status:  False    Type:    Paused    Reason:      Status:  True    Type:    Selected    Reason:      Status:  True    Type:    AllInjected    Reason:      Status:  False    Type:    AllRecovered  Experiment:    Container Records:      Id:            dev/nginx      Phase:         Injected      Selector Key:  .    Desired Phase:   RunEvents:  Type    Reason           Age    From          Message  ----    ------           ----   ----          -------  Normal  FinalizerInited  6m35s  finalizer     Finalizer has been inited  Normal  Updated          6m35s  finalizer     Successfully update finalizer of resource  Normal  Updated          6m35s  records       Successfully update records of resource  Normal  Updated          6m35s  desiredphase  Successfully update desiredPhase of resource  Normal  Applied          6m35s  records       Successfully apply chaos for dev/nginx  Normal  Updated          6m35s  records       Successfully update records of resource

The control plane also naturally has the function of querying and acquiring Chaos resources, so that platform users can view the implementation status of all chaos experiments and manage them. Of course, here is nothing more than calling the REST API to send Get/List requests, but in practice, we need to pay attention to the details. Our company has happened that the Controller requests the full amount of resource data each time, causing the load of the K8s API Server to increase.
It is highly recommended to read クライアントの使い方, this controller runtime usage tutorial mentions very details. For example, the controller runtime will read kubeconfig, flags, environment variables, and the Service Account automatically mounted in the Pod from multiple locations by default. Armosec/kubescape #21 PR also takes advantage of this feature. This tutorial also includes common operations such as how to paging, update, and overwrite objects. I haven't seen any Chinese or English tutorials that are so detailed.
Example of Get/List request:

package controlpanelimport ( "context" "github.com/chaos-mesh/chaos-mesh/api/v1alpha1" "github.com/pkg/errors" "sigs.k8s.io/controller-runtime/pkg/client")func GetPodChaos(name, namespace string) (*v1alpha1.PodChaos, error) { cli := mgr.GetClient()

 item := new(v1alpha1.PodChaos)
 if err := cli.Get(context.Background(), client.ObjectKey{Name: name, Namespace: namespace}, item); err != nil {
  return nil, errors.Wrap(err, "get cr")
 }

 return item, nil
}

func ListPodChaos(namespace string, labels map[string]string) ([]v1alpha1.PodChaos, error) {
 cli := mgr.GetClient()

 list := new(v1alpha1.PodChaosList)
 if err := cli.List(context.Background(), list, client.InNamespace(namespace), client.MatchingLabels(labels)); err != nil {
  return nil, err
 }

 return list.Items, nil
}

In the example, the manager is used. In this mode, the cache mechanism is enabled to avoid repetitive fetching of large amounts of data.

• Get Pod
• Get full data for the first time (List)
• Update cache when Watch data changes

Chaos Choreography

Just as the CRI container runtime provides powerful underlying isolation capabilities that can support the stable operation of the container, and container orchestration is required for larger-scale and more complex scenarios, Chaos Mesh® provides Schedule and Workflow functions. Schedule can trigger faults regularly and at intervals according to the set cron time. Workflow can schedule multiple fault tests like Argo Workflow.

Of course, Chaos Controller Manager does most of the work for us. What the control plane needs is to manage these YAML resources. The only thing that needs to be considered is what functions should be provided to users.

Platform function

With reference to Chaos Mesh® Dashboard, we need to consider what functions the platform should provide to end users.

Possible platform function points:

• Chaos injection
• Pod crash
• Network failure
• Load test
• IO failure
• Event tracking
• Associated alarm
• Timing telemetry

see information
This article is not only a trial for the company to introduce new technologies, but also a record of self-learning. The more excellent parts of the learning materials previously contacted and the materials reviewed when writing this article are listed here for quick reference.

• Controller-runtime source code analysis
• つくって学ぶKubebuilder (Japanese Course)
• Kubebuilder Book / Chinese version
• kube-controller-manager source code analysis (3) Informer mechanism
• kubebuilder2.0 study notes-advanced use
• Skills in client-go and golang source code
• Chaos Mesh-Let applications dance with chaos on Kubernetes
• Self-made file system-02 Gospel for developers, FUSE file system
• System stress test tool-stress-ng
• Dkron source code analysis
• NameSpace of RunC Source Code Reading Guide