Aggregated APIServer best practices for building cloud native applications

author

Zhang Peng, Tencent Cloud Container Product Engineer, has many years of experience in cloud native project development and implementation. Currently, he is mainly responsible for the development of Tencent Cloud TKE cloud native AI products.

Xie Yuandong, Tencent senior engineer, Kubeflow Member, Fluid (CNCF Sandbox) core developer, responsible for the research and development and support of Tencent Cloud TKE in AI scenarios.

Overview

With the maturity of Kubernetes, more and more companies and enterprises begin to use K8s to build their own cloud-native platforms. Based on kubernetes' good scalability and mature and stable architecture, you can quickly deploy and manage your own cloud-native applications.

Currently we are building a cloud-based kubernetes native AI platform (we call it: SKAI ), the platform has extreme elasticity , cloudy compatibility , high ease , observability , The feature of aims to use cloud-native ideas and technologies to build a flexible and scalable system architecture for data processing, model training, model online reasoning and other requirements in AI scenarios, thereby improving resource utilization.

In order to make our platform more cloud-native, we did not choose commonly used web frameworks to build API services, but used kubernetes extensions to build the entire platform, so that our platform can better integrate with kubernetes and can be seamlessly adapted Any k8s-based multi-cloud hybrid cloud environment.

Why choose Aggregated APIServer?

Choose standalone API or Aggregated APIServer?

Although it is easy to build a stable API interface service using Go language web frameworks such as gin and go-restful, there are still many advantages to building API interface services in a native kubernetes way, such as:

• Can use kubernetes native authentication, authorization, access and other mechanisms to have higher development efficiency;
• It can better integrate with the K8s system, promote its products faster with the help of the K8s ecology, and facilitate users to get started;
• With the help of K8s mature API tools and specifications, the API interface constructed is more standardized and tidy;

However, in many scenarios, we are still not sure whether to use Aggregated APIServer or independent API to build our services. The official provides us with a comparison of two options; if you are not sure whether to use Aggregated APIServer or independent API, below The form may be helpful to you:

Consider the case of API aggregation	Prefer standalone API
You are developing a new API	You already have a program that provides API services and is working well
You hope you can use kubectl to read and write your new resource category	Does not require kubectl support
You want to view your new resource category in the Kubernetes UI (such as dashboard) along with other built-in categories	No need for Kubernetes UI support
You want to reuse Kubernetes API support feature	You don't need this kind of feature
You are willing to accept the format restrictions imposed by Kubernetes on REST resource paths, such as API groups and namespaces. (Refer to API overview )	You need to use some special REST paths in order to be compatible with the defined REST API
Your API is declarative	Your API does not conform to the declarative model
Your resources can naturally be defined as the cluster scope or a namespace scope in the cluster	The dichotomy of cluster scope or namespace scope is not appropriate; you need to control the details of the resource path

First of all, we hope that our SKAI platform can better integrate with k8s, and it is a declarative API that reuses the characteristics of the Kubernets API as much as possible. Obviously, the aggregation API is more suitable for us.

Choose CRDs or Aggregated APIServer?

In addition to the aggregation API, the official also provides another way to implement the extension of the standard kubernetes API interface: CRD (Custom Resource Definition), which can achieve basically the same functions as the aggregation API, and is easier to use and has a lower development cost, but In comparison, the aggregation API is more flexible. Regarding how to choose between these two expansion methods, the official also provides a corresponding reference.

Generally, CRD may be more appropriate if the following conditions exist:

• There are not many fields for custom resources;
• You use the resource within your organization or use it in a small-scale open source project, not in a commercial product;
  Converged API can provide more advanced API features, and can also customize other features; for example, customize the storage layer, support the protobuf protocol, and support operations such as logs and patches.

The core difference between the two methods is the way to define api-resource. In the Aggregated APIServer method, api-resource registers the resource type with the API through code, and Custom Resource registers the resource type with the API directly through the yaml file.

Simply put, CRD is to let kube-apiserver know more object categories (Kind), and Aggregated APIServer is to build its own APIServer service. Although CRD is simpler, it lacks more flexibility. For a more detailed comparison of CRDs and Aggregated API, please refer to official document .

For us, we hope to use more advanced API features, such as "logs" or "exec", not just CRUD, so we finally chose Aggregated APIServer.

The basic principle of APIServer extension

kube-apiserver, as the only entry for etcd operation of the entire Kubernetes cluster, is responsible for the authentication & authentication, verification and CRUD operations of Kubernetes resources, and provides RESTful APIs for other components to call:

kube-apiserver actually contains three APIServers:

• AggregatorServer : Responsible for processing apiregistration.k8s.io resource requests under the 0619cd0c6b44f4 group, and at the same time intercepting and forwarding requests from users to Aggregated APIServer (AA);
• KubeAPIServer : Responsible for some general processing of requests, including: authentication, authentication, and REST services of various built-in resources (pod, deployment, service), etc.;
• ApiExtensionsServer : Responsible for the registration of CustomResourceDefinition (CRD) apiResources and apiVersions, while processing CRD and the corresponding CustomResource (CR) REST request (return 404 if the corresponding CR cannot be processed), which is also the last link of apiserver Delegation;

The three APIServers are related through the delegation relationship. During the initial creation of kube-apiserver, the APIExtensionsServer is created first, and its delegationTarget is an empty delegate, that is, it does nothing. Then the GenericAPIServer of the APIExtensionsServer is passed as the delegationTarget to KubeAPIServer was created, KubeAPIServer was created, and then GenericAPIServer of kubeAPIServer was passed to AggregatorServer as delegationTarget, and AggregatorServer was created, so the delegation relationship between them is: Aggregator -> KubeAPIServer -> APIExtensions, as shown in the following figure:

How to quickly build Aggregated APIServer?

Although the official sample-apiserver , we can refer to the implementation of our own Aggregated APIServer. However, it is too complicated to write by hand, and it is not convenient for later maintenance. We finally chose the officially recommended tool apiserver-builder , apiserver-builder can help us quickly create the project skeleton, and compare the project directory structure built with apiserver-builder Clear, more conducive to later maintenance.

Install the apiserver-builder tool

Install via Go Get

$ GO111MODULE=on go get sigs.k8s.io/apiserver-builder-alpha/cmd/apiserver-boot

Install via installation package

• download the latest version of
• Unzip to /usr/local/apiserver-builder/
• If this directory does not exist, create this directory
• Add /usr/local/apiserver-builder/bin to your path export PATH=$PATH:/usr/local/apiserver-builder/bin
• Run apiserver-boot -h

Initialize the project

After completing the apiserver-boot installation, you can initialize an Aggregated APIServer project with the following command:

$ mkdir skai-demo
$ cd skai-demo 
$ apiserver-boot init repo --domain skai.io

The following directory will be generated after execution:

.
├── BUILD.bazel
├── Dockerfile
├── Makefile
├── PROJECT
├── WORKSPACE
├── bin
├── cmd
│   ├── apiserver
│   │   └── main.go
│   └── manager
│       └── main.go -> ../../main.go
├── go.mod
├── hack
│   └── boilerplate.go.txt
├── main.go
└── pkg
    └── apis
        └── doc.go

• The hack directory stores automated scripts
• cmd/apiserver is the start entry of aggregated server
• cmd/manager is the start entry of the controller
• pkg/apis stores CR-related structure definitions, which will be automatically generated in the next step

Generate custom resources

$ apiserver-boot create group version resource --group animal --version v1alpha1 --kind Cat --non-namespaced=false
Create Resource [y/n]
y
Create Controller [y/n]
n

You can choose whether to generate a Controller according to your needs. We temporarily choose not to generate it here. For resources that need to be isolated by namespace, you need to add the parameter --non-namespaced=false. The default is true.

The code structure after execution is as follows:

└── pkg
    └── apis
        ├── animal
        │   ├── doc.go
        │   └── v1alpha1
        │       ├── cat_types.go
        │       ├── doc.go
        │       └── register.go
        └── doc.go

You can see that the animal group is generated under pkg/apis and the cat_types.go file has been added under the v1alpha1 version. This file contains the basic definition of our resources. We add field definitions to the spec, and in the Validate method that has been implemented Complete the verification of the basic fields.

// Cat
// +k8s:openapi-gen=true
type Cat struct {
        metav1.TypeMeta   `json:",inline"`
        metav1.ObjectMeta `json:"metadata,omitempty"`
        Spec   CatSpec   `json:"spec,omitempty"`
        Status CatStatus `json:"status,omitempty"`
}
// CatSpec defines the desired state of Cat
type CatSpec struct {
    Name string `json:"name"`
}
func (in *Cat) Validate(ctx context.Context) field.ErrorList {
    allErrs := field.ErrorList{}
    if len(in.Spec.Name) == 0 {
        allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "name"), in.Spec.Name, "must be specify"))
    }
    return allErrs
}

Deploy and run

After completing the above steps, you actually have a complete Aggregated APIServer, then we try to run it; apiserver-boot itself provides two operating modes: in-cluster, local; local mode only serves as a separate API service Deploying locally is convenient for debugging. It’s too simple and I won’t introduce too much here. I mainly focus on the in-cluster mode; in-cluster can deploy your Aggregated APIServer in any K8s cluster, such as minikube, Tencent TKE, EKS, etc., We use the EKS cluster as a demonstration here.

Create EKS cluster & configure local kubeconfig ;

Execute deployment commands;

$ apiserver-boot run in-cluster --image=xxx/skai.io/skai-demo:0.0.1 --name=skai-demo --namespace=default

In the process of executing the deployment command, apiserver-boot mainly helps us to do the following things:

• Automatically generate APIServer Dockerfile file;
• Build the service image through APIServer Dockerfile and push the image to the designated warehouse;
• Generate the certificate files needed for CA and other APIServer deployment in the config directory;
• Generate Deployment, Service, APIService, ServiceAccount and other yaml files required for APIServer deployment in the config directory;
• Deploy the yaml file generated in the previous step to the cluster;

Functional Verification

Confirm that the Resource registration is successful

$ kubectl api-versions |grep animal
animal.skai.io/v1alpha1

Confirm that Aggregated APIServer can work normally

$ kubectl get apiservice v1alpha1.animal.skai.io 
NAME                      SERVICE             AVAILABLE   AGE
v1alpha1.animal.skai.io   default/skai-demo   True        19h

Create and view the newly added Resource

create

$ cat lucky.yaml
apiVersion: animal.skai.io/v1alpha1
kind: Cat
metadata:
  name: mycat
  namespace: default
spec:
  name: lucky
# 创建自定义 resource
$ kubectl apply -f lucky.yaml

Find

# 查找自定义 resource 列表
$ kubectl get cat
NAME    CREATED AT
mycat   2021-11-17T09:08:10Z
# 查找自定义资源详情
$ kubectl get cat mycat -oyaml
apiVersion: animal.skai.io/v1alpha1
kind: Cat
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"animal.skai.io/v1alpha1","kind":"Cat","metadata":{"annotations":{},"name":"mycat"},"spec":{"name":"lucky"}}
  creationTimestamp: "2021-11-17T09:08:10Z"
  name: mycat
  resourceVersion: "17"
  uid: 98af0905-f01d-4042-bad3-71b96c0919f4
spec:
  name: lucky
status: {}

Summarize

This article introduces the reasons why we chose Aggregated API during the development of the SKAI platform and the principle of kube-apisever expansion from a practical perspective. Finally, it introduces the apiserver-builder tool and demonstrates how to build our own Aggregated API step by step and deploy it. To the EKS cluster. I hope this Aggregated APIServer best practice can help developers who are about to use K8s API extensions to build cloud-native applications.

about us

For more cases and knowledge about cloud native, please follow the public account of the same name [Tencent Cloud Native]~

Welfare:

①Respond to the backstage of the official account [Manual] to get "Tencent Cloud Native Roadmap Manual" & "Tencent Cloud Native Best Practices"~

②The public account backstage reply [series], you can get the "15 series of 100+ super practical cloud native original dry goods collection", including Kubernetes cost reduction and efficiency, K8s performance optimization practices, best practices and other series.

[Tencent Cloud Native] Yunshuo new products, Yunyan new technology, Yunyou Xinhuo, Yunxiang information, scan the QR code to follow the public account of the same name, and get more dry goods in time! !