kubebuilder actual combat two: experience kubebuilder for the first time

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: Classification and summary of all original articles and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

Links to series of articles

1. kubebuilder combat one: preparation
2. kubebuilder actual combat 2: first experience with kubebuilder
3. kubebuilder combat three: a
4. kubebuilder combat four: operator requirement description and design
5. kubebuilder combat five: operator coding
6. kubebuilder combat six: build deployment operation
7. kubebuilder combat seven: webhook
8. kubebuilder combat eight: a small note of knowledge

Overview of this article

This article is the second in the "kubebuilder combat" series. The kubebuilder environment is prepared in the previous article. Today we create CRD and Controller in this environment, then deploy to the kubernetes environment and verify whether it works. The entire article consists of the following content:

1. Create API (CRD and Controller)
2. Build and deploy CRD
3. Compile and run the controller
4. Create an instance corresponding to CRD
5. Delete the instance and stop the controller
6. Make the controller into a docker image
7. Uninstall and clean up

Create the helloworld project

1. Execute the following command to create the helloworld project:

mkdir -p $GOPATH/src/helloworld
cd $GOPATH/src/helloworld
kubebuilder init --domain com.bolingcavalry

2. The console output is similar to the following:

[root@kubebuilder helloworld]# kubebuilder init --domain com.bolingcavalry
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.5.0
Update go.mod:
$ go mod tidy
Running make:
$ make
/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: define a resource with:
$ kubebuilder create api

3. After waiting for a few minutes to complete the creation, add the following content to the <font color="blue">$GOPATH/src/helloworld</font> directory. It can be seen that this is a standard go module project:

[root@kubebuilder ~]# tree $GOPATH/src/helloworld
/root/gopath/src/helloworld
├── bin
│   └── manager
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   └── role_binding.yaml
│   └── webhook
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       └── service.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

9 directories, 30 files

Create API (CRD and Controller)

1. The next step is to create resource-related content. These three parts <font color="blue">group/version/kind</font> can determine the unique identity of the resource. The command is as follows:

cd $GOPATH/src/helloworld
kubebuilder create api \
--group webapp \
--version v1 \
--kind Guestbook

2. The console will remind you whether to create a resource (Create Resource [y/n]), enter <font color="red">y</font>
3. Next, the console will remind whether to create a controller (Create Controller [y/n]), enter <font color="red">y</font>
4. kubebuilder will add multiple files according to the above command, as shown in the red box in the following figure:

Build and deploy CRD

1. The Makefile provided by kubebuilder greatly simplifies the construction and deployment work. Executing the following command will deploy the newly built CRD on kubernetes:

cd $GOPATH/src/helloworld
make install

2. The console outputs the following content, indicating that the deployment is successful:

[root@kubebuilder helloworld]# make install
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.com.bolingcavalry created

Compile and run the controller

1. The controller source code address automatically generated by kubebuilder is: $GOPATH/src/helloworld/controllers/guestbook_controller.go, and the content is as follows:

package controllers

import (
    "context"

    "github.com/go-logr/logr"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"

    webappv1 "helloworld/api/v1"
)

// GuestbookReconciler reconciles a Guestbook object
type GuestbookReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=webapp.com.bolingcavalry,resources=guestbooks,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=webapp.com.bolingcavalry,resources=guestbooks/status,verbs=get;update;patch

func (r *GuestbookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    _ = context.Background()
    _ = r.Log.WithValues("guestbook", req.NamespacedName)

    // your logic here

    return ctrl.Result{}, nil
}

func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&webappv1.Guestbook{}).
        Complete(r)
}

2. This article focuses on experiencing the basic process, and does not study the source code in depth, so only a few changes are made to the above code to verify whether it can take effect. The changes are shown in the red box in the following figure:

3. Executing the following command will compile and start the newly modified controller:

cd $GOPATH/src/helloworld
make run

4. At this time, the console outputs the following. It should be noted that the controller is running on the kubebuilder computer. Once the console is interrupted with <font color="blue">Ctrl+c</font>, the controller will stop:

[root@kubebuilder helloworld]# make run
/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2021-01-23T20:58:35.107+0800    INFO    controller-runtime.metrics    metrics server is starting to listen    {"addr": ":8080"}
2021-01-23T20:58:35.108+0800    INFO    setup    starting manager
2021-01-23T20:58:35.108+0800    INFO    controller-runtime.manager    starting metrics server    {"path": "/metrics"}
2021-01-23T20:58:35.108+0800    INFO    controller-runtime.controller    Starting EventSource    {"controller": "guestbook", "source": "kind source: /, Kind="}
2021-01-23T20:58:35.208+0800    INFO    controller-runtime.controller    Starting Controller    {"controller": "guestbook"}
2021-01-23T20:58:35.209+0800    INFO    controller-runtime.controller    Starting workers    {"controller": "guestbook", "worker count": 1}

Create an instance of Guestbook resource

1. Now kubernetes has deployed the Guestbook type CRD, and the corresponding controller is already running, you can try to create an instance of the Guestbook type (equivalent to having the definition of the pod, you can create the pod);
2. kubebuilder has automatically created a type of deployment file: $GOPATH/src/helloworld/config/samples/webapp_v1_guestbook.yaml, the content is as follows, very simple, then we will use this file to create a Guestbook instance:

apiVersion: webapp.com.bolingcavalry/v1
kind: Guestbook
metadata:
  name: guestbook-sample
spec:
  # Add fields here
  foo: bar

3. Reopen a console, log in to the kubebuilder computer, and execute the following command to create an instance of the Guestbook type:

cd $GOPATH/src/helloworld
kubectl apply -f config/samples/

4. As shown below, the console prompts that the resource is created successfully:

[root@kubebuilder helloworld]# kubectl apply -f config/samples/
guestbook.webapp.com.bolingcavalry/guestbook-sample created

5. You can see that the instance has been created with the kubectl get command:

[root@kubebuilder helloworld]# kubectl get Guestbook
NAME               AGE
guestbook-sample   112s

6. Use the command <font color="blue">kubectl edit Guestbook guestbook-sample</font> to edit the example, and the modified content is shown in the red box in the following figure:

7. At this point, go to the console where the controller is located, and you can see that both the new and modified operations have log output. Our new logs are all in it, and the code call stack is clear at a glance:

2021-01-24T09:51:50.418+0800    INFO    controllers.Guestbook    1. default/guestbook-sample
2021-01-24T09:51:50.418+0800    INFO    controllers.Guestbook    2. goroutine 188 [running]:
runtime/debug.Stack(0xc0002a1808, 0xc0002fc600, 0x1b)
    /root/go/src/runtime/debug/stack.go:24 +0x9f
helloworld/controllers.(*GuestbookReconciler).Reconcile(0xc0003c9dd0, 0xc0002d02f9, 0x7, 0xc0002d02e0, 0x10, 0x12f449647b, 0xc000456f30, 0xc000456ea8, 0xc000456ea0)
    /root/gopath/src/helloworld/controllers/guestbook_controller.go:49 +0x1a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00022a480, 0x1430e00, 0xc0003e7560, 0x0)
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00022a480, 0xc000469600)
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00022a480)
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000292980)
    /root/gopath/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000292980, 0x3b9aca00, 0x0, 0x1609101, 0xc000102480)
    /root/gopath/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000292980, 0x3b9aca00, 0xc000102480)
    /root/gopath/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:193 +0x32d

2021-01-24T09:51:50.418+0800    DEBUG    controller-runtime.controller    Successfully Reconciled    {"controller": "guestbook", "request": "default/guestbook-sample"}


2021-01-24T09:52:33.632+0800    INFO    controllers.Guestbook    1. default/guestbook-sample
2021-01-24T09:52:33.633+0800    INFO    controllers.Guestbook    2. goroutine 188 [running]:
runtime/debug.Stack(0xc0002a1808, 0xc0003fa5e0, 0x1b)
    /root/go/src/runtime/debug/stack.go:24 +0x9f
helloworld/controllers.(*GuestbookReconciler).Reconcile(0xc0003c9dd0, 0xc0002d02f9, 0x7, 0xc0002d02e0, 0x10, 0x1d0410fe42, 0xc000456f30, 0xc000456ea8, 0xc000456ea0)
    /root/gopath/src/helloworld/controllers/guestbook_controller.go:49 +0x1a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00022a480, 0x1430e00, 0xc0003d24c0, 0x0)
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00022a480, 0xc000469600)
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00022a480)
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000292980)
    /root/gopath/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000292980, 0x3b9aca00, 0x0, 0x1609101, 0xc000102480)
    /root/gopath/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000292980, 0x3b9aca00, 0xc000102480)
    /root/gopath/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /root/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:193 +0x32d

2021-01-24T09:52:33.633+0800    DEBUG    controller-runtime.controller    Successfully Reconciled    {"controller": "guestbook", "request": "default/guestbook-sample"}

Delete the instance and stop the controller

1. When the Guestbook instance is no longer needed, execute the following command to delete it:

cd $GOPATH/src/helloworld
kubectl delete -f config/samples/

2. When the controller is no longer needed, go to its console and use <font color="blue">Ctrl+c</font> to interrupt;

Make the controller into a docker image

1. So far, we have experienced the basic functions of kubebuilder, but in the actual production environment, the controller will generally run in the kubernetes environment. The above method of running outside of kubernetes is not suitable, let's try to make it The docker image is then run in the kubernetes environment;
2. There is a requirement here, that is, you must have a mirror repository that kubernetes can access, such as Harbor in the local area network, or public hub.docker.com. I chose hub.docker.com for the convenience of operation. The premise of using it is Have a registered account of hub.docker.com;
3. On the kubebuilder computer, open a console, execute the <font color="blue">docker login</font> command to log in, and enter the account and password of hub.docker.com as prompted, so that you can execute it on the current console The docker push command pushes the image to hub.docker.com (the network of this website is very poor, and it may take several logins to succeed);
4. Execute the following command to build a docker image and push it to hub.docker.com, the image name is <font color="blue">bolingcavalry/guestbook:002</font>:

cd $GOPATH/src/helloworld
make docker-build docker-push IMG=bolingcavalry/guestbook:002

5. The network condition of hub.docker.com is not generally bad. The docker on the kubebuilder computer must be set to mirror acceleration. If the above command fails over time, please try again several times. In addition, many go module dependencies will be downloaded during the build process. , You also need to wait patiently, and it is easy to encounter network problems, which require multiple retries. Therefore, it is best to use the Habor service built in the local area network;
6. Finally, the output after the command is successfully executed is as follows:

[root@kubebuilder helloworld]# make docker-build docker-push IMG=bolingcavalry/guestbook:002
/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out
?       helloworld    [no test files]
?       helloworld/api/v1    [no test files]
ok      helloworld/controllers    8.604s    coverage: 0.0% of statements
docker build . -t bolingcavalry/guestbook:002
Sending build context to Docker daemon  40.27MB
Step 1/14 : FROM golang:1.13 as builder
 ---> d6f3656320fe
Step 2/14 : WORKDIR /workspace
 ---> Using cache
 ---> 83d05ead1041
Step 3/14 : COPY go.mod go.mod
 ---> Using cache
 ---> ae3e15a529f4
Step 4/14 : COPY go.sum go.sum
 ---> Using cache
 ---> 082223532ccc
Step 5/14 : RUN go mod download
 ---> Using cache
 ---> bcdcfa1d65ca
Step 6/14 : COPY main.go main.go
 ---> Using cache
 ---> 81d6a629ca98
Step 7/14 : COPY api/ api/
 ---> Using cache
 ---> 75f99b174e97
Step 8/14 : COPY controllers/ controllers/
 ---> b130d9f47903
Step 9/14 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go
 ---> Running in 768880aca19f
Removing intermediate container 768880aca19f
 ---> bb4a494d3b43
Step 10/14 : FROM gcr.io/distroless/static:nonroot
 ---> 947e6f3ed7c1
Step 11/14 : WORKDIR /
 ---> Using cache
 ---> 22cc43cef8fb
Step 12/14 : COPY --from=builder /workspace/manager .
 ---> 2137778f22c0
Step 13/14 : USER nonroot:nonroot
 ---> Running in 18295673073d
Removing intermediate container 18295673073d
 ---> f7545379ab1f
Step 14/14 : ENTRYPOINT ["/manager"]
 ---> Running in 550c47dd61dc
Removing intermediate container 550c47dd61dc
 ---> 31cb31a6b03f
Successfully built 31cb31a6b03f
Successfully tagged bolingcavalry/guestbook:002
docker push bolingcavalry/guestbook:002
The push refers to repository [docker.io/bolingcavalry/guestbook]
99035107a955: Pushed 
728501c5607d: Layer already exists 
002: digest: sha256:54f8ec88511cce5b04c5d65cc15e0f7a7b4a8afb6b235904a638bff79e3c5784 size: 739

7. Go to hub.docker.com website to see, as shown below, the new image has been uploaded, so as long as any machine can access the Internet, you can pull this image to the local for use:

8. After the image is ready, execute the following command to deploy the controller in the kubernetes environment:

cd $GOPATH/src/helloworld
make deploy IMG=bolingcavalry/guestbook:002

9. The console will prompt that various resources have been created (mostly RBAC):

[root@kubebuilder ~]# cd $GOPATH/src/helloworld
[root@kubebuilder helloworld]# make deploy IMG=bolingcavalry/guestbook:002
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && kustomize edit set image controller=bolingcavalry/guestbook:002
kustomize build config/default | kubectl apply -f -
namespace/helloworld-system created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.com.bolingcavalry configured
role.rbac.authorization.k8s.io/helloworld-leader-election-role created
clusterrole.rbac.authorization.k8s.io/helloworld-manager-role created
clusterrole.rbac.authorization.k8s.io/helloworld-proxy-role created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/helloworld-metrics-reader created
rolebinding.rbac.authorization.k8s.io/helloworld-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/helloworld-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/helloworld-proxy-rolebinding created
service/helloworld-controller-manager-metrics-service created
deployment.apps/helloworld-controller-manager created

10. At this time, I went to look at the pod of the kubernetes environment and found that the controller has indeed been added, as shown in the red box as shown below:

11. Be careful, you should find that the pod in the yellow box above actually has two containers. Use the <font color="blue">kubectl describe</font> command to take a closer look. They are kube-rbac-proxy and manager, as shown below:

11. Since there are two containers, one of them must be specified when viewing the log. Our controller corresponds to the manager container, so the command to view the log is:

kubectl logs -f \
helloworld-controller-manager-689d4b6f5b-h9pzg \
-n helloworld-system \
-c manager

12. Create an instance of the Guestbook resource again, still using the <font color="blue">kubectl apply -f config/samples/</font> command, and then look at the manager container log, we can see that the content we modified has been printed out:

Uninstall and clean up

• After the experience, if you want to clean up all the previously created resources and CRD, you can execute the following command:

cd $GOPATH/src/helloworld
make uninstall

• At this point, we have experienced the basic process of creating Operator-related resources through kubebuilder. This article focuses on familiar tools and processes, and has not experienced the substantial and powerful functions of Operator. These are left to the following chapters, let’s step by step In-depth study and practice;

You are not lonely, Xinchen is with you all the way

1. Java series
2. Spring series
3. Docker series
4. kubernetes series
5. database + middleware series
6. DevOps series

Welcome to pay attention to the public account: programmer Xin Chen

Search "Programmer Xin Chen" on WeChat, I am Xin Chen, and I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos