kubebuilder combat five: operator coding

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: Classification and summary of all original articles and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

Links to series of articles

1. kubebuilder combat one: preparation
2. kubebuilder actual combat 2: first experience with kubebuilder
3. kubebuilder combat 3: basic knowledge at a
4. kubebuilder combat four: operator requirement description and design
5. kubebuilder combat five: operator code
6. kubebuilder combat six: build deployment and run
7. kubebuilder combat seven: webhook
8. kubebuilder combat eight: a small note of knowledge

Overview of this article

• This article is the fifth article in the "kubebuilder actual combat" series. All previous efforts (environment preparation, knowledge reserve, demand analysis, data structure and business logic design) are all to implement the previous design with coding;
• Now that we have fully prepared and don't need too much words now, let's get started!

Source download

• The complete source code in this actual combat can be downloaded on GitHub, the address and link information are shown in the following table ( https://github.com/zq2599/blog_demos):

name	link	Remarks
Project homepage	https://github.com/zq2599/blog_demos	The project's homepage on GitHub
git warehouse address (https)	https://github.com/zq2599/blog_demos.git	The warehouse address of the source code of the project, https protocol
git warehouse address (ssh)	[e-mail]:zq2599/blog_demos.git	The warehouse address of the source code of the project, ssh protocol

• There are multiple folders in this git project, and kubebuilder-related applications are under the <font color="blue">kubebuilder</font> folder, as shown in the red box below:

• There are multiple subfolders under the kubebuilder folder. The corresponding source code of this article is in the <font color="blue">elasticweb</font> directory, as shown in the red box in the following figure:

New project elasticweb

• Create a new folder named <font color="blue">elasticweb</font>, execute the following command in it to create a project named <font color="blue">elasticweb</font>, the domain is <font color="blue">com.bolingcavalry</font>:

go mod init elasticweb
kubebuilder init --domain com.bolingcavalry

• Then CRD, execute the following command to create related resources:

kubebuilder create api \
--group elasticweb \
--version v1 \
--kind ElasticWeb

• Then use the IDE to open the entire project, here is goland:

CRD encoding

• Open the file <font color="blue">api/v1/elasticweb_types.go</font> and make the following changes:
• Modify the data structure ElasticWebSpec, adding the four fields designed in the previous section;
• Modify the data structure ElasticWebStatus to add a field designed in the previous article;
• Add the String method, so that we can view it when printing the log. Note that the RealQPS field is a pointer, so it may be empty and needs to be nulled;
• The complete <font color="blue">elasticweb_types.go</font> looks like this:

package v1

import (
    "fmt"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "strconv"
)

// 期望状态
type ElasticWebSpec struct {
    // 业务服务对应的镜像，包括名称:tag
    Image string `json:"image"`
    // service占用的宿主机端口，外部请求通过此端口访问pod的服务
    Port *int32 `json:"port"`

    // 单个pod的QPS上限
    SinglePodQPS *int32 `json:"singlePodQPS"`
    // 当前整个业务的总QPS
    TotalQPS *int32 `json:"totalQPS"`
}

// 实际状态，该数据结构中的值都是业务代码计算出来的
type ElasticWebStatus struct {
    // 当前kubernetes中实际支持的总QPS
    RealQPS *int32 `json:"realQPS"`
}

// +kubebuilder:object:root=true

// ElasticWeb is the Schema for the elasticwebs API
type ElasticWeb struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   ElasticWebSpec   `json:"spec,omitempty"`
    Status ElasticWebStatus `json:"status,omitempty"`
}

func (in *ElasticWeb) String() string {
    var realQPS string

    if nil == in.Status.RealQPS {
        realQPS = "nil"
    } else {
        realQPS = strconv.Itoa(int(*(in.Status.RealQPS)))
    }

    return fmt.Sprintf("Image [%s], Port [%d], SinglePodQPS [%d], TotalQPS [%d], RealQPS [%s]",
        in.Spec.Image,
        *(in.Spec.Port),
        *(in.Spec.SinglePodQPS),
        *(in.Spec.TotalQPS),
        realQPS)
}

// +kubebuilder:object:root=true

// ElasticWebList contains a list of ElasticWeb
type ElasticWebList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []ElasticWeb `json:"items"`
}

func init() {
    SchemeBuilder.Register(&ElasticWeb{}, &ElasticWebList{})
}

• Execute <font color="blue">make install</font> in the elasticweb directory to deploy CRD to kubernetes:

zhaoqin@zhaoqindeMBP-2 elasticweb % make install
/Users/zhaoqin/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/elasticwebs.elasticweb.com.bolingcavalry created

• After the deployment is successful, use the <font color="blue">api-versions</font> command to find the GV:

Review business logic

• After the core data structure design and coding is completed, it is time to write the business logic code. Everyone remember the business process designed in the previous article. Let's briefly review it, as shown in the following figure:
  
• Open the file <font color="blue">elasticweb_controller.go</font>, and then we will gradually add content;

Add resource access

• Our elasticweb will query, add, modify and other operations on the two resources of service and deployment. Therefore, we need the operation authority of these resources. Add the two lines of comments in the red box in the figure below, so that the code generation tool will be configured in RBAC Increase the corresponding permissions in:

Constant definition

• Prepare the constants first, it can be seen that the CPU and memory used by each pod are fixed here, you can also change to define in the Spec, so that it can be passed in from the outside, and here is only allocated for each pod 0.1 CPU, mainly because I can't afford a good CPU because I am poor, you can adjust this value as appropriate:

const (
    // deployment中的APP标签名
    APP_NAME = "elastic-app"
    // tomcat容器的端口号
    CONTAINER_PORT = 8080
    // 单个POD的CPU资源申请
    CPU_REQUEST = "100m"
    // 单个POD的CPU资源上限
    CPU_LIMIT = "100m"
    // 单个POD的内存资源申请
    MEM_REQUEST = "512Mi"
    // 单个POD的内存资源上限
    MEM_LIMIT = "512Mi"
)

Method getExpectReplicas

• There is a very important logic: According to the QPS and total QPS of a single pod, calculate how many pods are needed. Let's encapsulate this logic into a method for use:

/ 根据单个QPS和总QPS计算pod数量
func getExpectReplicas(elasticWeb *elasticwebv1.ElasticWeb) int32 {
    // 单个pod的QPS
    singlePodQPS := *(elasticWeb.Spec.SinglePodQPS)

    // 期望的总QPS
    totalQPS := *(elasticWeb.Spec.TotalQPS)

    // Replicas就是要创建的副本数
    replicas := totalQPS / singlePodQPS

    if totalQPS%singlePodQPS > 0 {
        replicas++
    }

    return replicas
}

Method createServiceIfNotExists

• Encapsulate the operation of creating a service into a method, so that the logic of the main code is clearer and more readable;
• When creating a service, there are a few things to pay attention to:
• First check whether the service exists, and then create it if it does not exist;
• Establish an association between service and CRD instance elasticWeb (controllerutil.SetControllerReference method), so that when elasticWeb is deleted, the service will be deleted automatically without our intervention;
• The <font color="blue">client-go</font> tool was used when creating the service. It is recommended that you read the "client-go combat series". The more proficient the tool, the more enjoyable the coding;
• The complete method to create a service is as follows:

// 新建service
func createServiceIfNotExists(ctx context.Context, r *ElasticWebReconciler, elasticWeb *elasticwebv1.ElasticWeb, req ctrl.Request) error {
    log := r.Log.WithValues("func", "createService")

    service := &corev1.Service{}

    err := r.Get(ctx, req.NamespacedName, service)

    // 如果查询结果没有错误，证明service正常，就不做任何操作
    if err == nil {
        log.Info("service exists")
        return nil
    }

    // 如果错误不是NotFound，就返回错误
    if !errors.IsNotFound(err) {
        log.Error(err, "query service error")
        return err
    }

    // 实例化一个数据结构
    service = &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Namespace: elasticWeb.Namespace,
            Name:      elasticWeb.Name,
        },
        Spec: corev1.ServiceSpec{
            Ports: []corev1.ServicePort{{
                Name:     "http",
                Port:     8080,
                NodePort: *elasticWeb.Spec.Port,
            },
            },
            Selector: map[string]string{
                "app": APP_NAME,
            },
            Type: corev1.ServiceTypeNodePort,
        },
    }

    // 这一步非常关键！
    // 建立关联后，删除elasticweb资源时就会将deployment也删除掉
    log.Info("set reference")
    if err := controllerutil.SetControllerReference(elasticWeb, service, r.Scheme); err != nil {
        log.Error(err, "SetControllerReference error")
        return err
    }

    // 创建service
    log.Info("start create service")
    if err := r.Create(ctx, service); err != nil {
        log.Error(err, "create service error")
        return err
    }

    log.Info("create service success")

    return nil
}

Method createDeployment

• Encapsulate the operation of creating deployment in a method, also to keep the main logic simple;
• There are also several points to note in the method of creating deployment:
• Call the getExpectReplicas method to get the number of pods to be created, which is an important parameter when creating a deployment;
• The CPU and memory resources required by each pod are also deployment parameters;
• Establish an association between deployment and elasticweb, so that the deployment will be automatically deleted when elasticweb is deleted;
• The same is to use the client-go client tool to create deployment resources;

// 新建deployment
func createDeployment(ctx context.Context, r *ElasticWebReconciler, elasticWeb *elasticwebv1.ElasticWeb) error {
    log := r.Log.WithValues("func", "createDeployment")

    // 计算期望的pod数量
    expectReplicas := getExpectReplicas(elasticWeb)

    log.Info(fmt.Sprintf("expectReplicas [%d]", expectReplicas))

    // 实例化一个数据结构
    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Namespace: elasticWeb.Namespace,
            Name:      elasticWeb.Name,
        },
        Spec: appsv1.DeploymentSpec{
            // 副本数是计算出来的
            Replicas: pointer.Int32Ptr(expectReplicas),
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{
                    "app": APP_NAME,
                },
            },

            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{
                        "app": APP_NAME,
                    },
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{
                        {
                            Name: APP_NAME,
                            // 用指定的镜像
                            Image:           elasticWeb.Spec.Image,
                            ImagePullPolicy: "IfNotPresent",
                            Ports: []corev1.ContainerPort{
                                {
                                    Name:          "http",
                                    Protocol:      corev1.ProtocolSCTP,
                                    ContainerPort: CONTAINER_PORT,
                                },
                            },
                            Resources: corev1.ResourceRequirements{
                                Requests: corev1.ResourceList{
                                    "cpu":    resource.MustParse(CPU_REQUEST),
                                    "memory": resource.MustParse(MEM_REQUEST),
                                },
                                Limits: corev1.ResourceList{
                                    "cpu":    resource.MustParse(CPU_LIMIT),
                                    "memory": resource.MustParse(MEM_LIMIT),
                                },
                            },
                        },
                    },
                },
            },
        },
    }

    // 这一步非常关键！
    // 建立关联后，删除elasticweb资源时就会将deployment也删除掉
    log.Info("set reference")
    if err := controllerutil.SetControllerReference(elasticWeb, deployment, r.Scheme); err != nil {
        log.Error(err, "SetControllerReference error")
        return err
    }

    // 创建deployment
    log.Info("start create deployment")
    if err := r.Create(ctx, deployment); err != nil {
        log.Error(err, "create deployment error")
        return err
    }

    log.Info("create deployment success")

    return nil
}

Method updateStatus

• Whether you are creating a deployment resource object or adjusting the number of pods of an existing deployment, you must modify the Status after these operations are completed, the actual status, so that the outside can know how much QPS the current elasticweb supports at any time and anywhere, so you need to change The operation of modifying the Status is encapsulated in a method for use in multiple scenarios. The calculation logic of Status is very simple: the number of pods multiplied by the QPS of each pod is the total QPS. The code is as follows:

// 完成了pod的处理后，更新最新状态
func updateStatus(ctx context.Context, r *ElasticWebReconciler, elasticWeb *elasticwebv1.ElasticWeb) error {
    log := r.Log.WithValues("func", "updateStatus")

    // 单个pod的QPS
    singlePodQPS := *(elasticWeb.Spec.SinglePodQPS)

    // pod总数
    replicas := getExpectReplicas(elasticWeb)

    // 当pod创建完毕后，当前系统实际的QPS：单个pod的QPS * pod总数
    // 如果该字段还没有初始化，就先做初始化
    if nil == elasticWeb.Status.RealQPS {
        elasticWeb.Status.RealQPS = new(int32)
    }

    *(elasticWeb.Status.RealQPS) = singlePodQPS * replicas

    log.Info(fmt.Sprintf("singlePodQPS [%d], replicas [%d], realQPS[%d]", singlePodQPS, replicas, *(elasticWeb.Status.RealQPS)))

    if err := r.Update(ctx, elasticWeb); err != nil {
        log.Error(err, "update instance error")
        return err
    }

    return nil
}

Backbone code

• The previous details have been processed, and the main process can be started. With the assignment of the previous flowchart, the code of the main process is easy to write. As shown below, enough comments have been added, so I won't repeat it:

func (r *ElasticWebReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    // 会用到context
    ctx := context.Background()
    log := r.Log.WithValues("elasticweb", req.NamespacedName)

    // your logic here

    log.Info("1. start reconcile logic")

    // 实例化数据结构
    instance := &elasticwebv1.ElasticWeb{}

    // 通过客户端工具查询，查询条件是
    err := r.Get(ctx, req.NamespacedName, instance)

    if err != nil {

        // 如果没有实例，就返回空结果，这样外部就不再立即调用Reconcile方法了
        if errors.IsNotFound(err) {
            log.Info("2.1. instance not found, maybe removed")
            return reconcile.Result{}, nil
        }

        log.Error(err, "2.2 error")
        // 返回错误信息给外部
        return ctrl.Result{}, err
    }

    log.Info("3. instance : " + instance.String())

    // 查找deployment
    deployment := &appsv1.Deployment{}

    // 用客户端工具查询
    err = r.Get(ctx, req.NamespacedName, deployment)

    // 查找时发生异常，以及查出来没有结果的处理逻辑
    if err != nil {
        // 如果没有实例就要创建了
        if errors.IsNotFound(err) {
            log.Info("4. deployment not exists")

            // 如果对QPS没有需求，此时又没有deployment，就啥事都不做了
            if *(instance.Spec.TotalQPS) < 1 {
                log.Info("5.1 not need deployment")
                // 返回
                return ctrl.Result{}, nil
            }

            // 先要创建service
            if err = createServiceIfNotExists(ctx, r, instance, req); err != nil {
                log.Error(err, "5.2 error")
                // 返回错误信息给外部
                return ctrl.Result{}, err
            }

            // 立即创建deployment
            if err = createDeployment(ctx, r, instance); err != nil {
                log.Error(err, "5.3 error")
                // 返回错误信息给外部
                return ctrl.Result{}, err
            }

            // 如果创建成功就更新状态
            if err = updateStatus(ctx, r, instance); err != nil {
                log.Error(err, "5.4. error")
                // 返回错误信息给外部
                return ctrl.Result{}, err
            }

            // 创建成功就可以返回了
            return ctrl.Result{}, nil
        } else {
            log.Error(err, "7. error")
            // 返回错误信息给外部
            return ctrl.Result{}, err
        }
    }

    // 如果查到了deployment，并且没有返回错误，就走下面的逻辑

    // 根据单QPS和总QPS计算期望的副本数
    expectReplicas := getExpectReplicas(instance)

    // 当前deployment的期望副本数
    realReplicas := *deployment.Spec.Replicas

    log.Info(fmt.Sprintf("9. expectReplicas [%d], realReplicas [%d]", expectReplicas, realReplicas))

    // 如果相等，就直接返回了
    if expectReplicas == realReplicas {
        log.Info("10. return now")
        return ctrl.Result{}, nil
    }

    // 如果不等，就要调整
    *(deployment.Spec.Replicas) = expectReplicas

    log.Info("11. update deployment's Replicas")
    // 通过客户端更新deployment
    if err = r.Update(ctx, deployment); err != nil {
        log.Error(err, "12. update deployment replicas error")
        // 返回错误信息给外部
        return ctrl.Result{}, err
    }

    log.Info("13. update status")

    // 如果更新deployment的Replicas成功，就更新状态
    if err = updateStatus(ctx, r, instance); err != nil {
        log.Error(err, "14. update status error")
        // 返回错误信息给外部
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

• At this point, the entire elasticweb operator coding is completed. Due to space limitations, let's put the deployment, operation, mirroring and other operations in the next article;

You are not alone, Xinchen and original are with you all the way

1. Java series
2. Spring series
3. Docker series
4. kubernetes series
5. database + middleware series
6. DevOps series

Welcome to pay attention to the public account: programmer Xin Chen

Search "Programmer Xin Chen" on WeChat, I am Xin Chen, and I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos