Get Kubernetes cluster resource information using C#

Hello everyone, I'm Yan Zhenfan, a researcher at Microsoft's MVP Lab in this issue. Today, I will share with you how to use the Kubernetes API Server to write components and obtain the resource object information of the cluster from K8S through code examples.

Yan Zhenfan - Microsoft's most valuable expert, is currently learning about microservices, you can communicate more~

Preface

Some time ago, I wrote a project in C#, using Kubernetes API Server to obtain information and monitor Kubernetes resources, and then combined with Neting as an API gateway.

Experience address http://neting.whuanle.cn:30080/

Account admin, password admin123

This article mainly introduces how to develop Kubernetes-based applications through C#, obtain information of various resources in Kubernetes, and realize the prerequisite knowledge of Conroller.

Kubernetes API Server

kube-apiserver is one of the main processes of k8s. The apiserver component exposes the Kubernetes API (HTTP API). The apiserver is the front end of the Kubernetes control plane. We can write code in programming languages such as Go and C#, call Kubernetes remotely, and control the operation of the cluster. The endiont port exposed by the apiserver is 6443.

In order to control the operation of the cluster, Kubernetes officially provides a binary command line tool called kubectl. It is the apiserver that provides the interface service. After kubectl parses the instructions entered by the user, it initiates an HTTP request to the apiserver, and then feeds the result back to the user.

kubectl is a very powerful tool for controlling the cluster that comes with Kubernetes. It manages the entire cluster through command line operations.
Kubernetes has many visual panels, such as Dashboard, behind which is also an API that calls apiserver, which is equivalent to adjusting the front-end to the back-end.

In short, the various tools we use to manage the cluster are all back-end apiserver. Through the apiserver, we can also customize various tools for managing the cluster, such as the grid management tool istio. Cloud platforms such as Tencent Cloud and Alibaba Cloud provide online kubernetes services, as well as console visualization operations, which also use apiserver.

You can refer to the Kubernetes e-book written by the author to learn more: https://k8s.whuanle.cn/1.basic/5.k8s.html

In short, the Kubernetes API Server is the entry point for third parties to operate Kubernetes.

exposes Kubernetes API Server

First look at the Kubernetes components running in kube-system, there is a kube-apiserver-master running.

root@master:~# kubectl get pods -o wide  -n kube-system
NAME    READY   STATUS    RESTARTS         AGE   IP          NODE     NOMINATED NODE   READINESS GATES
... ...
kube-apiserver-master            1/1     Running   2 (76d ago)      81d   10.0.0.4    master   <none>           <none>
... ...

Although these components are important, there will only be one instance, and they will run as Pods, not Deployments. These components can only be run on the master node.

Then check the admin.conf file, which can be found in the /etc/kubernetes/admin.conf or $HOME/.kube/config path.

The admin.conf file is the credential for accessing the Kubernetes API Server. Through this file, we can programmatically access the Kubernetes API interface.

However, admin.conf is a very important file. If it is a development environment development cluster, you can create it casually. If it is a production environment, do not use it. You can restrict API access authorization through role binding and other methods.

Then download the admin.conf or config file locally.

You can use the kubectl edit pods kube-apiserver-master -n kube-system command to view some configuration information of the Kubernetes API Server.

Since the Kubernetes API Server is accessed through the cluster by default, if remote access is required, it needs to be exposed outside the cluster (it has nothing to do with whether it is on the intranet or not, it is related to whether it is within the cluster).

Expose the API Server outside the cluster:

kubectl expose pod  kube-apiserver-master --type=NodePort --port=6443 -n kube-system

View the port randomly assigned by the node:

root@master:~# kubectl get svc -n kube-system
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
kube-apiserver-master   NodePort    10.101.230.138   <none>        6443:32263/TCP           25s

Port 32263 is automatically assigned by Kubernetes, and everyone's is different.

Then you can test access through IP:32263.

If your cluster has CoreDNS installed, you can also access this service through the IP of other nodes.

Then download the admin.conf or config file (please rename it to admin.conf), and modify the server attribute inside, because we are accessing it remotely at this time.

connects to API Server

Create a new MyKubernetes console project, then copy the admin.conf file into the project, and generate output with the project.

Then search for the KubernetesClient package in Nuget, the author is currently using 7.0.1.

Then set the environment variables in the project:

This environment variable itself comes with ASP.NET Core, not in the console program.

Write a method below to instantiate and get the Kubernetes client:

private static Kubernetes GetClient()
    {
        KubernetesClientConfiguration config;
        if (Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") == "Development")
        {
            // 通过配置文件
            config = KubernetesClientConfiguration.BuildConfigFromConfigFile("./admin.conf");
        }
        else
        {
            // 通过默认的 Service Account 访问，必须在 kubernetes 中运行时才能使用
            config = KubernetesClientConfiguration.BuildDefaultConfig(); 
        }
        return new Kubernetes(config);
    }

The logic is very simple. If it is a development environment, use the admin.conf file to access it. If it is a non-development environment, BuildDefaultConfig() automatically obtains access credentials. This method is only valid when running in a Pod, using Service Account authentication.

Let's test it to get all namespaces:

static async Task Main()
    {
        var client = GetClient();
        var namespaces  = await client.ListNamespaceAsync();
        foreach (var item in namespaces.Items)
        {
            Console.WriteLine(item.Metadata.Name);
        }
    }

All right! You have already acquired Kubernetes resources, open the first step of getting started! Shower!

Client Tips

Although the first step of getting started is opened, don't rush to use various APIs. Here we will learn about the definition of various Kubernetes resources in the client and how to parse the structure.

First, in the code of Kubernetes Client C#, the model classes of all Kubernetes resources are recorded in k8s.Models.

If we were to look at the definition of an object in Kubernetes, such as in the kube-systtem namespace:

kubectl get namespace kube-system -o yaml

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-11-03T13:57:10Z"
  labels:
    kubernetes.io/metadata.name: kube-system
  name: kube-system
  resourceVersion: "33"
  uid: f0c1f00d-2ee4-40fb-b772-665ac2a282d7
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

In C#, the structure of the model is exactly the same:

In the client, the model's name is prefixed with the apiVersion version, and a list of such objects is obtained via V1NamespaceList.

If you want to get a certain type of resource, its interface starts with List, such as client.ListNamespaceAsync(),

client.ListAPIServiceAsync(), client.ListPodForAllNamespacesAsync(), etc.

It seems that learning is on the right track, let's experiment and practice!

Practice 1: How to parse a Service

Here the author has prepared some exercises for readers. The first exercise is to parse the information of a Service.
Check out the Servicie created earlier:

kubectl get svc  kube-apiserver-master -n kube-system -o yaml

The corresponding structure is as follows:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-01-24T12:51:32Z"
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver-master
  namespace: kube-system
  resourceVersion: "24215604"
  uid: ede0e3df-8ef6-45c6-9a8d-2a2048c6cb12
spec:
  clusterIP: 10.101.230.138
  clusterIPs:
  - 10.101.230.138
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 32263
    port: 6443
    protocol: TCP
    targetPort: 6443
  selector:
    component: kube-apiserver
    tier: control-plane
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

We define a model class like this in C#:

  public class ServiceInfo
    {
        /// <summary>
        /// SVC 名称
        /// </summary>
        public string Name { get; set; } = null!;

        /// <summary>
        /// 三种类型之一 <see cref="ServiceType"/>
        /// </summary>
        public string? ServiceType { get; set; }
        /// <summary>
        /// 命名空间
        /// </summary>
        public string Namespace { get; set; } = null!;

        /// <summary>
        /// 有些 Service 没有此选项
        /// </summary>
        public string ClusterIP { get; set; } = null!;

        /// <summary>
        /// 外网访问 IP
        /// </summary>
        public string[]? ExternalAddress { get; set; }

        public IDictionary<string, string>? Labels { get; set; }

        public IDictionary<string, string>? Selector { get; set; }

        /// <summary>
        /// name,port
        /// </summary>
        public List<string>? Ports { get; set; }

        public string[]? Endpoints { get; set; }

        public DateTime? CreationTime { get; set; }

        // 关联的 Pod 以及 pod 的 ip
    }

Next, specify which namespace's Service to obtain and its associated Endpoint information.

 static async Task Main()
    {
        var result = await GetServiceAsync("kube-apiserver-master","kube-system");
        Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result));
    }
    public static async Task<ServiceInfo> GetServiceAsync(string svcName, string namespaceName)
    {
        var client = GetClient();
        var service = await client.ReadNamespacedServiceAsync(svcName, namespaceName);

        // 获取 service 本身的信息
        ServiceInfo info = new ServiceInfo
        {
            Name = service.Metadata.Name,
            Namespace = service.Metadata.NamespaceProperty,
            ServiceType = service.Spec.Type,
            Labels = service.Metadata.Labels,
            ClusterIP = service.Spec.ClusterIP,
            CreationTime = service.Metadata.CreationTimestamp,
            Selector = service.Spec.Selector.ToDictionary(x => x.Key, x => x.Value),
            ExternalAddress = service.Spec.ExternalIPs?.ToArray(),
        };

        // service -> endpoint 的信息
        var endpoint = await client.ReadNamespacedEndpointsAsync(svcName, namespaceName);
        List<string> address = new List<string>();
        foreach (var sub in endpoint.Subsets)
        {
            foreach (var addr in sub.Addresses)
            {
                foreach (var port in sub.Ports)
                {
                    address.Add($"{addr.Ip}:{port.Port}/{port.Protocol}");
                }
            }

        }
        info.Endpoints = address.ToArray();
        return info;
    }

The output is as follows:

Dear, if you are not very clear about the network knowledge of Kubernetes, please open https://k8s.whuanle.cn/4.network/1.network.html to find out.

Practice 2: Detailed Analysis of Service Attributes

We know that a Service can be associated with multiple Pods to provide load balancing and other functions for multiple Pods. At the same time, Service has attributes such as externalIP, clusterIP, etc. It is difficult to really parse out a Service. For example, Service can have only ports and no IP; it can also be accessed using only DNS domain name; it can also not bind any Pod, and can indirectly access B from Service A DNS -> Service B IP;

There are many situations in Service. Readers can refer to the following figure. Below, we obtain the IP and port information of a Service through the code, and then generate the corresponding IP+port structure.

Simply obtaining IP and port is useless, because they are separate, the IP you obtain may be from Cluter, Node, LoadBalancer, or it may be just DNS without IP, so how do you access this port? At this time, it is necessary to parse the information and filter invalid data according to certain rules in order to obtain a useful access address.
First define some enumerations and models:

 public enum ServiceType
    {
        ClusterIP,
        NodePort,
        LoadBalancer,

        ExternalName
    }

    /// <summary>
    /// Kubernetes Service 和 IP
    /// </summary>
    public class SvcPort
    {

        // LoadBalancer -> NodePort -> Port -> Target-Port

        /// <summary>
        /// 127.0.0.1:8080/tcp、127.0.0.1:8080/http
        /// </summary>
        public string Address { get; set; } = null!;

        /// <summary>
        /// LoadBalancer、NodePort、Cluster
        /// </summary>
        public string Type { get; set; } = null!;

        public string IP { get; set; } = null!;
        public int Port { get; set; }
    }
    public class SvcIpPort
    {
        public List<SvcPort>? LoadBalancers { get; set; }
        public List<SvcPort>? NodePorts { get; set; }
        public List<SvcPort>? Clusters { get; set; }
        public string? ExternalName { get; set; }
    }

Write parsing code:

static async Task Main()
    {
        var result = await GetSvcIpsAsync("kube-apiserver-master","kube-system");
        Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result));
    }

    public static async Task<SvcIpPort> GetSvcIpsAsync(string svcName, string namespaceName)
    {
        var client = GetClient();
        var service = await client.ReadNamespacedServiceAsync(svcName, namespaceName);

        SvcIpPort svc = new SvcIpPort();

        // LoadBalancer
        if (service.Spec.Type == nameof(ServiceType.LoadBalancer))
        {
            svc.LoadBalancers = new List<SvcPort>();
            var ips = svc.LoadBalancers;

            // 负载均衡器 IP
            var lbIP = service.Spec.LoadBalancerIP;
            var ports = service.Spec.Ports.Where(x => x.NodePort != null).ToArray();
            foreach (var port in ports)
            {
                ips.Add(new SvcPort
                {
                    Address = $"{lbIP}:{port.NodePort}/{port.Protocol}",
                    IP = lbIP,
                    Port = (int)port.NodePort!,
                    Type = nameof(ServiceType.LoadBalancer)
                });
            }
        }

        if (service.Spec.Type == nameof(ServiceType.LoadBalancer) || service.Spec.Type == nameof(ServiceType.NodePort))
        {
            svc.NodePorts = new List<SvcPort>();
            var ips = svc.NodePorts;

            // 负载均衡器 IP，有些情况可以设置 ClusterIP 为 None；也可以手动设置为 None，只要有公网 IP 就行
            var clusterIP = service.Spec.ClusterIP;
            var ports = service.Spec.Ports.Where(x => x.NodePort != null).ToArray();
            foreach (var port in ports)
            {
                ips.Add(new SvcPort
                {
                    Address = $"{clusterIP}:{port.NodePort}/{port.Protocol}",
                    IP = clusterIP,
                    Port = (int)port.NodePort!,
                    Type = nameof(ServiceType.NodePort)
                });
            }
        }
        
        // 下面这部分代码是正常的，使用 {} 可以隔离部分代码，避免变量重名
        // if (service.Spec.Type == nameof(ServiceType.ClusterIP))
        // 如果 Service 没有 Cluster IP，可能使用了无头模式，也有可能不想出现 ClusterIP
        //if(service.Spec.ClusterIP == "None")
        {
            svc.Clusters = new List<SvcPort>();
            var ips = svc.Clusters;
            var clusterIP = service.Spec.ClusterIP;

            var ports = service.Spec.Ports.ToArray();
            foreach (var port in ports)
            {
                ips.Add(new SvcPort
                {
                    Address = $"{clusterIP}:{port.Port}/{port.Protocol}",
                    IP = clusterIP,
                    Port = port.Port,
                    Type = nameof(ServiceType.ClusterIP)
                });
            }
        }

        if (!string.IsNullOrEmpty(service.Spec.ExternalName))
        {
            /* NAME            TYPE           CLUSTER-IP       EXTERNAL-IP          PORT(S)     AGE
               myapp-svcname   ExternalName   <none>           myapp.baidu.com      <none>      1m
               myapp-svcname ->  myapp-svc 
               访问 myapp-svc.default.svc.cluster.local，变成 myapp.baidu.com
             */
            svc.ExternalName = service.Spec.ExternalName;
        }
        return svc;
    }

The rule analysis is more complicated, so it will not be explained in detail here. If readers have any questions, they can contact the author for discussion.

Main rule: LoadBalancer -> NodePort -> Port -> Target-Port .
The final result is as follows:

Through this part of the code, you can parse out the list of addresses that the Service can really access in the case of External Name, LoadBalancer, NodePort, ClusterIP, etc.

Practice 3: Parsing Endpoint Lists

If you don't know much about Endpoint, please open https://k8s.whuanle.cn/4.network/2.endpoint.html to take a look at the relevant knowledge.

In Kubernetes, Service is not directly associated with Pod, but indirectly proxy Pod through Endpoint. Of course, in addition to Service -> Pod, through Endpoint, you can also access third-party services outside the cluster. For example, the database cluster is not in the Kubernetes cluster, but if you want to access it uniformly through Kubernetes Service, you can use Endpoint for decoupling. Not much to say here, readers can refer to https://k8s.whuanle.cn/4.network/2.endpoint.html .

In this section, the author will also explain how to obtain resources by paging in Kubernetes.

First define the following model:

 public class SvcInfoList
    {
        /// <summary>
        /// 分页属性，具有临时有效期，具体由 Kubernetes 确定
        /// </summary>
        public string? ContinueProperty { get; set; }

        /// <summary>
        /// 预计剩余数量
        /// </summary>
        public int RemainingItemCount { get; set; }

        /// <summary>
        /// SVC 列表
        /// </summary>
        public List<SvcInfo> Items { get; set; } = new List<SvcInfo>();
    }

    public class SvcInfo
    {
        /// <summary>
        /// SVC 名称
        /// </summary>
        public string Name { get; set; } = null!;

        /// <summary>
        /// 三种类型之一 <see cref="ServiceType"/>
        /// </summary>
        public string? ServiceType { get; set; }

        /// <summary>
        /// 有些 Service 没有 IP，值为 None
        /// </summary>
        public string ClusterIP { get; set; } = null!;

        public DateTime? CreationTime { get; set; }

        public IDictionary<string, string>? Labels { get; set; }

        public IDictionary<string, string>? Selector { get; set; }

        /// <summary>
        /// name,port
        /// </summary>
        public List<string> Ports { get; set; }

        public string[]? Endpoints { get; set; }
    }

Paging in Kubernetes, there is no PageNo, PageSize, Skip, Take, Limit, and paging may only be expected, not necessarily completely accurate.

The ContinueProperty property cannot be used for the first access to get a list of objects.

After accessing Kubernetes for the first time and getting 10 pieces of data, Kubernetes will return a ContinueProperty token and the remaining number RemainingItemCount.

Then we can calculate the approximate number of pagination through RemainingItemCount. Because Kubernetes cannot be paged directly, but through something like a cursor, it records the current access position, and then continues to fetch objects downwards. ContinueProperty holds the token for the current query cursor, but this token is valid for a few minutes.

Parsing method:

public static async Task<SvcInfoList> GetServicesAsync(string namespaceName, 
                                                           int pageSize = 1, 
                                                           string? continueProperty = null)
    {
        var client = GetClient();

        V1ServiceList services;
        if (string.IsNullOrEmpty(continueProperty))
        {
            services = await client.ListNamespacedServiceAsync(namespaceName, limit: pageSize);
        }
        else
        {
            try
            {
                services = await client.ListNamespacedServiceAsync(namespaceName, 
                                                                   continueParameter: continueProperty, 
                                                                   limit: pageSize);
            }
            catch (Microsoft.Rest.HttpOperationException ex)
            {
                throw ex;
            }
            catch
            {
                throw;
            }
        }

        SvcInfoList svcList = new SvcInfoList
        {
            ContinueProperty = services.Metadata.ContinueProperty,
            RemainingItemCount = (int)services.Metadata.RemainingItemCount.GetValueOrDefault(),
            Items = new List<SvcInfo>()
        };

        List<SvcInfo> svcInfos = svcList.Items;
        foreach (var item in services.Items)
        {
            SvcInfo service = new SvcInfo
            {
                Name = item.Metadata.Name,
                ServiceType = item.Spec.Type,
                ClusterIP = item.Spec.ClusterIP,
                Labels = item.Metadata.Labels,
                Selector = item.Spec.Selector,
                CreationTime = item.Metadata.CreationTimestamp
            };
            // 处理端口
            if (item.Spec.Type == nameof(ServiceType.LoadBalancer) || item.Spec.Type == nameof(ServiceType.NodePort))
            {
                service.Ports = new List<string>();
                foreach (var port in item.Spec.Ports)
                {
                    service.Ports.Add($"{port.Port}:{port.NodePort}/{port.Protocol}");
                }
            }
            else if (item.Spec.Type == nameof(ServiceType.ClusterIP))
            {
                service.Ports = new List<string>();
                foreach (var port in item.Spec.Ports)
                {
                    service.Ports.Add($"{port.Port}/{port.Protocol}");
                }
            }

            var endpoint = await client.ReadNamespacedEndpointsAsync(item.Metadata.Name, namespaceName);
            if (endpoint != null && endpoint.Subsets.Count != 0)
            {
                List<string> address = new List<string>();
                foreach (var sub in endpoint.Subsets)
                {
                    if (sub.Addresses == null) continue;
                    foreach (var addr in sub.Addresses)
                    {
                        foreach (var port in sub.Ports)
                        {
                            address.Add($"{addr.Ip}:{port.Port}/{port.Protocol}");
                        }
                    }

                }
                service.Endpoints = address.ToArray();
            }
            svcInfos.Add(service);
        }

        return svcList;
    }

The rule analysis is more complicated, so it will not be explained in detail here. If readers have any questions, they can contact the author for discussion.
Call method:

static async Task Main()
    {
        var result = await GetServicesAsync("default", 2);
        Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result.Items));

        if (result.RemainingItemCount != 0)
        {
            while (result.RemainingItemCount != 0)
            {
                Console.WriteLine($"剩余 {result.RemainingItemCount} 条数据，{result.RemainingItemCount / 3 + (result.RemainingItemCount % 3 == 0 ? 0 : 1)} 页，按下回车键继续获取！");
                Console.ReadKey();
                result = await GetServicesAsync("default", 2, result.ContinueProperty); 
                Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result.Items));
            }
        }
    }

In the above practice, there is a lot of code. It is recommended that the reader debug after startup, debug it step by step, slowly check the data, compare various objects in Kubernetes, and gradually deepen the understanding.

The next article will explain how to implement the Conroller and Kubernetes Operator. Stay tuned!

Microsoft Most Valuable Professional (MVP)

The Microsoft Most Valuable Professional is a global award given to third-party technology professionals by Microsoft Corporation. For 29 years, technology community leaders around the world have received this award for sharing their expertise and experience in technology communities both online and offline.

MVPs are a carefully selected team of experts who represent the most skilled and intelligent minds, passionate and helpful experts who are deeply invested in the community. MVP is committed to helping others and maximizing the use of Microsoft technologies by Microsoft technical community users by speaking, forum Q&A, creating websites, writing blogs, sharing videos, open source projects, organizing conferences, etc.
For more details, please visit the official website:
https://mvp.microsoft.com/zh-cn