Client-go actual combat 4: dynamicClient

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: Classification and summary of all original articles and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

Links to series of articles

1. client-go actual combat one: preparations
2. client-go actual combat 2: RESTClient
3. client-go actual combat three: Clientset
4. client-go combat four: dynamicClient
5. client-go actual combat five: DiscoveryClient

Overview of this article

• This article is the fourth in the "client-go combat" series. In the previous article, we studied the Clientset client and found that Clientset is very convenient when deploying and service kubernetes built-in resources. Each resource has its own method to cooperate. Official API documents and data structure definitions are more efficient than Restclient in development;
• But what if you are not dealing with the built-in resources of kubernetes? For example, CRD, there are no user-defined things in the code of Clientset, so Clientset is obviously not needed. At this time, the protagonist of this article, dynamicClient, is about to appear!

Relevant knowledge reserve

• Before formally learning dynamicClient, there are two important knowledge points to understand: <font color="blue">Object.runtime</font> and <font color="blue">Unstructured</font>, for the entire kubernetes Say they are all very important;

Object.runtime

• Before talking about Object.runtime, we must clarify two concepts: resource and resource object. Everyone is familiar with resources. Aren’t these pods and deployments all resources? My personal understanding is that resources are more like a strict definition. After a deployment is created in kubernetes, the newly created deployment instance is a resource object;
• In the code world of kubernetes, resource objects correspond to specific data structures. These data structures all implement the same interface, named <font color="red">Object.runtime</font>, and the source code location is <font color ="blue">staging/src/k8s.io/apimachinery/pkg/runtime/interfaces.go</font>, defined as follows:

type Object interface {
    GetObjectKind() schema.ObjectKind
    DeepCopyObject() Object
}

• The DeepCopyObject method, as the name implies, is a deep copy, that is, a new object is cloned from an object in memory;
• As for the function of the GetObjectKind method, I believe that the clever you have guessed: when dealing with Object.runtime type variables, as long as you call the GetObjectKind method to know its specific identity (such as deployment, service, etc.);
• Finally, I emphasize again: <font color="blue">resource objects are the realization of Object.runtime</font>;

Unstructured

• Before talking about Unstructured, let's look at a simple JSON string:

{
    "id": 101,
    "name": "Tom"
}

• The field names and field value types of the above JSON are fixed, so you can write a data structure to deal with it:

type Person struct {
    ID int
    Name String
}

• For the above JSON string is structured data (Structured Data), this should be easy to understand;
• The opposite of structured data is Unstructured Data. In the actual kubernetes environment, you may encounter some data with unpredictable structure. For example, there is a third field in the previous JSON string, the field The specific content and type of the value are not known when coding, but only when it is actually running. So how to deal with it when coding? I believe you will think of using <font color="blue">interface{}</font> to represent it. In fact, client-go does the same. Looking at the source code of the Unstructured data structure, the path is <font color="blue" >staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructured.go</font>:

type Unstructured struct {
    // Object is a JSON compatible map with string, float, int, bool, []interface{}, or
    // map[string]interface{}
    // children.
    Object map[string]interface{}
}

• Obviously, the above data structure definition does not play any role. What is really important is the associated method. As shown in the figure below, it can be seen that client-go has prepared a wealth of methods for Unstructured. With these methods, unstructured data can be flexibly processed:

Important knowledge points: the mutual conversion between Unstructured and resource objects

• There is also a very important point of knowledge: You can use Unstructured instances to generate resource objects, you can also use resource objects to generate Unstructured instances. This magical ability is implemented by the FromUnstructured and ToUnstructured methods of unstructuredConverter. The following code snippet shows how to Convert Unstructured instance to PodList instance:

// 实例化一个PodList数据结构，用于接收从unstructObj转换后的结果
podList := &apiv1.PodList{}

// unstructObj
err = runtime.DefaultUnstructuredConverter.FromUnstructured(unstructObj.UnstructuredContent(), podList)

• You may be curious about how the above FromUnstructured method implements the conversion. Let's take a look at the internal implementation of this method, as shown in the following figure. In fact, there is no suspense. You can get the field information of podList through reflection:

• So far, is the analysis of Unstructured over? No, it is strongly recommended that you enter the fromUnstructured method in the red box 2 above to see the details. This is very exciting. Take podList as an example. This is a data structure, and fromUnstructured only handles primitive types. For data structures, the structFromUnstructured method will be called. Processing, in the structFromUnstructured method
  When processing each field of the data structure, fromUnstructured is called again. This is a process of mutual iteration. In the end, no matter how many nested data structures are in the podList, it will be processed. Because of space limitations, I won’t expand and believe the analysis. The figure below is Part of the key code:

• Summary: The routine of turning Unstructured into a resource object is not mysterious. It is nothing more than using reflection to obtain the field type of the resource object, and then go to the Unstructured map to obtain the original data according to the field name, and then use reflection to set it in the field of the resource object;
• After finishing the preparatory work, it is time to return to the topic of this article: dynamicClient client

About dynamicClient

• The data structure of resources such as deployment and pod is clear and fixed, which can accurately correspond to the data structure and method in Clientset, but for CRD (user-defined resources), Clientset client can do nothing. At this time, a kind of data is needed. The structure carries the data of the resource object, and there must be corresponding methods to process the data;
• At this moment, the aforementioned Unstructured can be launched. That's right, the resource objects not supported by Clientset are handed over to Unstructured to carry them. Next, let's look at the relationship between dynamicClient and Unstructured:
• First look at the data structure definition, there is no difference from the clientset, there is only a restClient field:

type dynamicClient struct {
    client *rest.RESTClient
}

• This data structure has only one associated method Resource, the input parameter is GVR, and it returns another data structure, dynamicResourceClient:

func (c *dynamicClient) Resource(resource schema.GroupVersionResource) NamespaceableResourceInterface {
    return &dynamicResourceClient{client: c, resource: resource}
}

• From the above code, we can see that the key to dynamicClient is the data structure dynamicResourceClient and its associated methods. Take a look at this dynamicResourceClient, as shown in the figure below. Sure enough, all resource-related operations of dynamicClient are done by dynamicResourceClient (agent mode?), and the create method is selected. Take a closer look, serialization and deserialization are handed over to the unstructured UnstructuredJSONScheme, and the interaction with kubernetes is handed over to Restclient:

• summary:
• Unlike Clientset, dynamicClient provides a unified operation API for various types of resources, and the resources need to be packaged as an Unstructured data structure;
• Internally uses Restclient to interact with kubernetes;
• That's all for the introduction and analysis of dynamicClient, you can start the actual combat;

Demand confirmation

• The actual coding requirements for this time are very simple: query all pods under the specified namespace, and then print them out on the console, requiring dynamicClient to be implemented;
• You may ask: Pod is a built-in resource of kubernetes, which is more suitable for Clientset to operate, and dynamicClient is more suitable for handling CRD, isn't it? ---You are right. Pod is used here because it is too troublesome to toss CRD. After the definition, it will be published on kubernetes. So we simply use pod instead of CRD. Anyway, dynamicClient can handle it. Let's master dynamicClient through actual combat. The usage is enough, and all kinds of resources can be dealt with in the future;

Source download

• The source code in this actual combat can be downloaded on GitHub, the address and link information are shown in the following table ( https://github.com/zq2599/blog_demos):

name	link	Remarks
Project homepage	https://github.com/zq2599/blog_demos	The project's homepage on GitHub
git warehouse address (https)	https://github.com/zq2599/blog_demos.git	The warehouse address of the source code of the project, https protocol
git warehouse address (ssh)	[e-mail]:zq2599/blog_demos.git	The warehouse address of the source code of the project, ssh protocol

• There are multiple folders in this git project, and client-go related applications are under the <font color="blue">client-go-tutorials</font> folder, as shown in the red box in the following figure:

• There are multiple subfolders under the client-go-tutorials folder. The corresponding source code of this article is under the <font color="blue">dynamicclientdemo</font> directory, as shown in the red box below:

coding

• Create a new folder dynamicclientdemo, execute the following command in it to create a new module:

go mod init dynamicclientdemo

• Add the two dependencies of k8s.io/api and k8s.io/client-go. Note that the version must match the kubernetes environment:

go get k8s.io/api@v0.20.0
go get k8s.io/client-go@v0.20.0

• Create a new main.go, the content is as follows, I will talk about the important points to pay attention to later:

package main

import (
    "context"
    "flag"
    "fmt"
    apiv1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/runtime/schema"
    "k8s.io/client-go/dynamic"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/util/homedir"
    "path/filepath"
)

func main() {

    var kubeconfig *string

    // home是家目录，如果能取得家目录的值，就可以用来做默认值
    if home:=homedir.HomeDir(); home != "" {
        // 如果输入了kubeconfig参数，该参数的值就是kubeconfig文件的绝对路径，
        // 如果没有输入kubeconfig参数，就用默认路径~/.kube/config
        kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
    } else {
        // 如果取不到当前用户的家目录，就没办法设置kubeconfig的默认目录了，只能从入参中取
        kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
    }

    flag.Parse()

    // 从本机加载kubeconfig配置文件，因此第一个参数为空字符串
    config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)

    // kubeconfig加载失败就直接退出了
    if err != nil {
        panic(err.Error())
    }

    dynamicClient, err := dynamic.NewForConfig(config)

    if err != nil {
        panic(err.Error())
    }

    // dynamicClient的唯一关联方法所需的入参
    gvr := schema.GroupVersionResource{Version: "v1", Resource: "pods"}

    // 使用dynamicClient的查询列表方法，查询指定namespace下的所有pod，
    // 注意此方法返回的数据结构类型是UnstructuredList
    unstructObj, err := dynamicClient.
        Resource(gvr).
        Namespace("kube-system").
        List(context.TODO(), metav1.ListOptions{Limit: 100})

    if err != nil {
        panic(err.Error())
    }

    // 实例化一个PodList数据结构，用于接收从unstructObj转换后的结果
    podList := &apiv1.PodList{}

    // 转换
    err = runtime.DefaultUnstructuredConverter.FromUnstructured(unstructObj.UnstructuredContent(), podList)

    if err != nil {
        panic(err.Error())
    }

    // 表头
    fmt.Printf("namespace\t status\t\t name\n")

    // 每个pod都打印namespace、status.Phase、name三个字段
    for _, d := range podList.Items {
        fmt.Printf("%v\t %v\t %v\n",
            d.Namespace,
            d.Status.Phase,
            d.Name)
    }
}

• There are three important points to note in the above code:
• The Resource method specifies the resource type of this operation;
• The List method initiates a request to kubernetes;
• FromUnstructured converts the Unstructured data structure into PodList, the principle of which has been analyzed before;
• Execute <font color="blue">go run main.go</font>, as follows, you can get data from kubernetes and convert it to PodList as normal:

zhaoqin@zhaoqindeMBP-2 dynamicclientdemo % go run main.go
namespace        status          name
kube-system      Running         coredns-7f89b7bc75-5pdwc
kube-system      Running         coredns-7f89b7bc75-nvbvm
kube-system      Running         etcd-hedy
kube-system      Running         kube-apiserver-hedy
kube-system      Running         kube-controller-manager-hedy
kube-system      Running         kube-flannel-ds-v84vc
kube-system      Running         kube-proxy-hlppx
kube-system      Running         kube-scheduler-hedy

• At this point, the learning and actual combat of dynamicClient is complete. It is a veritable dynamic client tool. It uses a set of APIs to process all resources. In addition to breaking the built-in resource limit of Clientset, it also allows our business code to have greater flexibility. I hope this article can give you some references to help you write code that more closely matches the scene;

You are not lonely, Xinchen is with you all the way

1. Java series
2. Spring series
3. Docker series
4. kubernetes series
5. database + middleware series
6. DevOps series

Welcome to pay attention to the public account: programmer Xin Chen

Search "Programmer Xin Chen" on WeChat, I am Xin Chen, and I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos