Lightweight installation of TKEStack: Let the existing K8s cluster have the ability of enterprise-level container cloud platform

about us

For more cases and knowledge about cloud native, you can pay attention to the public account of the same name [Tencent Cloud Native]~

Welfare:

① Reply to the [Manual] in the background of the official account, you can get the "Tencent Cloud Native Roadmap Manual" & "Tencent Cloud Native Best Practices"~

②The official account will reply to [series] in the background, and you can get "15 series of 100+ super practical cloud native original dry goods collection", including Kubernetes cost reduction and efficiency enhancement, K8s performance optimization practices, best practices and other series.

③If you reply to the [White Paper] in the background of the official account, you can get the "Tencent Cloud Container Security White Paper" & "The Source of Cost Reduction - Cloud Native Cost Management White Paper v1.0"

④ Reply to [Introduction to the Speed of Light] in the background of the official account, you can get a 50,000-word essence tutorial of Tencent Cloud experts, Prometheus and Grafana of the speed of light.

author

Wang Long, Tencent Cloud backend development engineer, is responsible for the design, development and maintenance of TKEStack and related work on hybrid cloud projects.

foreword

TKEStack is an open source enterprise-level container cloud platform that combines the most advanced technologies to provide highly scalable, high-performance container management services that allow you to easily run Kubernetes anywhere, meet IT needs, and empower DevOps teams.

TKEStack provides a wealth of components to implement user management, access policy management, cluster management, business management, monitoring, logging, private image library and other functions, which accordingly increases the time to install TKEStack locally. For some users, they may only need core functions such as user management, cluster management, and console access of TKEStack, but they also need to pay the same time cost. In addition, during the installation of TKEStack, a global cluster will be deployed on bare metal, and the lifecycle of independent clusters and imported clusters will be managed through the global cluster. Therefore, users cannot install and integrate TKEStack on an existing K8s cluster.

Based on the above background, TKEStack supports lightweight installation scenarios in the latest v1.9.0 release. The lightweight installation breaks the limitation of creating a new global cluster, and realizes the direct integration of TKESatck's core functions such as user management, cluster management, and console access on the user's existing K8s cluster. From this, users can view and manage the basic information of the cluster, worker nodes, namespaces, workloads and other resources through the front-end visualization page. In addition, by setting custom access policies, the limitation of "single tenant" is broken, and users can realize flexible permission control over the cluster.

Prerequisites

The content introduced in this article is based on an existing K8s cluster that is running normally, and the following operations need to be performed on the master node. If there is no K8s cluster, you can create a local cluster through kind and perform the following operations.

The content introduced in this article requires the core components of TKEStack to be installed through helm . You can refer to the installation manual to install helm.

Lightweight installation of TKEStack

Prepare installation files

Pull the latest code of TKEStack with the following command:

 git clone https://github.com/tkestack/tke.git

Create a local cluster (if you already have a local cluster, you can skip this step)

If the user does not have a K8s cluster locally, it can be created through kind. Since the cluster created by kind actually runs in the image container, in order to ensure that the ports of the core components of TKEStack can be accessed normally from outside the container, switch to the hack/lightweight-install/ directory of the TKEStack code, and execute the following command to create the cluster:

 kind create cluster --config kindConfig.yaml --name {your cluster name} 

Create the specified namespace

The three core components of tke-auth, tke-platform, and tke-gateway need to run under the specified namespace, and execute the following commands:

 kubectl create namespace tke

install chart

TKEStack provides chart files to install the three core components of tke-auth, tke-platform, and tke-gateway.

Switch to the hack/lightweight-install/ directory of the TKEStack code, and compile the binary executable file to generate the yaml file required to install the chart file:

 go build -o gen

The yaml file customConfig.yaml is placed in the hack/lightweight-install/ directory of the TKEStack code. customConfig.yaml Some "required" parameters in the file need to be filled in, and the rest of the parameters can be filled in as needed. If the optional part is empty, the default value will be filled automatically.

customConfig.yaml The content is as follows:

 # 必填，etcd 访问地址，形式如 https://172.19.0.2:2379
etcd:
  host: https://172.18.0.2:2379 
# 必填，服务器内网 IP，数组形式
serverIPs:
  - 172.18.0.2
# 必填，公网可访问的 IP 地址以及要使用的域名，数组形式
dnsNames:
  - tke.gateway
# 必填，集群 front-proxy-ca.crt 文件地址，默认位置为 /etc/kubernetes/pki/front-proxy-ca.crt
frontProxyCaCrtAbsPath: /etc/kubernetes/pki/front-proxy-ca.crt
# 必填，集群 etcd 的 ca.crt 文件地址，默认位置为 /etc/kubernetes/pki/etcd/ca.crt
etcdCrtAbsPath: /etc/kubernetes/pki/etcd/ca.crt
# 必填，集群 etcd 的 ca.key文件地址，默认位置为 /etc/kubernetes/pki/etcd/ca.key
etcdKeyAbsPath: /etc/kubernetes/pki/etcd/ca.key
tke-auth:
  api:
    # 必填
    replicas: 1
    # 必填
    image: tkestack/tke-auth-api-amd64:74592a3bceb5bebca602bea21aaebf78007a3bb2
    # 必填，数组形式，auth 的重定向访问地址，包括集群服务器 IP 地址（必填）、tke-gateway 的域名（可选）、集群高可用的 VIP 地址（可选，有的话需要填写）和集群的公共可访问域名（可选，有的话需要填写）
    redirectHosts: 
      - 172.18.0.2
    enableAudit: 
    # tke-auth-api 组件在 node 上的对外暴露端口，默认31138
    nodePort: 
    # 集群的租户 id，默认 default
    tenantID: 
    # OIDC 认证方式的 secret，默认自动生成
    oIDCClientSecret: 
    # authentication 用户名，默认为 admin
    adminUsername: 
  controller:
    # 必填
    replicas: 1
    # 必填
    image: tkestack/tke-auth-controller-amd64:74592a3bceb5bebca602bea21aaebf78007a3bb2
    # 控制台登陆的用户名，默认为 admin
    adminUsername: 
    # 控制台登陆的密码，默认自动生成
    adminPassword: 
tke-platform:
  # 必填，VIP 或者公网可访问的集群 IP
  publicIP:
  metricsServerImage: metrics-server:v0.3.6
  addonResizerImage: addon-resizer:1.8.11
  api:
    # 必填
    replicas: 1
    # 必填
    image: tkestack/tke-platform-api-amd64:bc48bed59bff2022d87db5e1484481715357ee7c
    enableAuth: true
    enableAudit: 
    # OIDC 认证方式客户端 id，默认为 default
    oIDCClientID: 
    # OIDC 认证方式的 issuer_url，默认为 https://tke-auth-api/oidc
    oIDCIssuerURL: 
    # 是否开启 OIDC 认证，默认不开启，值为空
    useOIDCCA:
  controller:
    # 必填
    replicas: 1
    # 必填
    providerResImage: tkestack/provider-res-amd64:v1.21.4-1
    # 必填
    image: tkestack/tke-platform-controller-amd64:bc48bed59bff2022d87db5e1484481715357ee7c
    # 默认为 docker.io
    registryDomain:
    # 默认为 tkestack
    registryNamespace:
    # 监控存储类型，默认为 influxdb
    monitorStorageType: 
    # 监控存储地址，默认为集群 master IP 地址加8086端口
    monitorStorageAddresses:
tke-gateway:
  # 必填
  image: tkestack/tke-gateway-amd64:bc48bed59bff2022d87db5e1484481715357ee7c
  # 默认为 docker.io
  registryDomainSuffix:
  # 集群的租户 id，默认为 default
  tenantID:
  # OIDC 认证方式的 secret，默认自动生成
  oIDCClientSecret:
  # 是否开启自签名，默认为 true
  selfSigned: true
  # 第三方 cert 证书，在 selfSigned 为 false 时需要填值
  serverCrt:
  # 第三方 certKey 密钥，在 selfSigned 为 false 时需要填值
  serverKey:
  enableAuth: true
  enableBusiness:
  enableMonitor:
  enableRegistry:
  enableLogagent:
  enableAudit:
  enableApplication:
  enableMesh:

customConfig.yaml After filling in the parameters in the file, execute gen in the current hack/lightweight-install/ directory:

 ./gen

It will generate three yaml files auth-chart-values.yaml , platform-chart-values.yaml , gateway-chart-values.yaml in the same level directory, corresponding to three chart (tke-auth, tke-platform) gateway) values.yaml file required during installation.

Install the three core components in the current hack/lightweight-install/ directory:

 # tke-auth 的安装
helm install -f auth-chart-values.yaml tke-auth tke-auth/

 # tke-platform 的安装
helm install -f platform-chart-values.yaml tke-platform tke-platform/

 # tke-gateway 的安装
helm install -f gateway-chart-values.yaml tke-gateway tke-gateway/

If the installation status of the three charts can be queried by the following commands: deployed , it means that the core components are installed successfully:

 helm list

After the chart is installed, you can query the following information, as shown in the figure:

Modify the cluster apiserver configuration

Create a new file under the directory corresponding to the cluster /etc/kubernetes/pki/ tke-authz-webhook.yaml , the content of the file is as follows (the IP address in the cluster.server parameter needs to be changed to the IP address of the master):

 apiVersion: v1
kind: Config
clusters:
  - name: tke
    cluster:
      server: https://172.19.0.2:31138/auth/authz
      insecure-skip-tls-verify: true
users:
  - name: admin-cert
    user:
      client-certificate: /etc/kubernetes/pki/webhook.crt
      client-key: /etc/kubernetes/pki/webhook.key
current-context: tke
contexts:
- context:
    cluster: tke
    user: admin-cert
  name: tke

The binaries gen generated webhook.crt and webhook.key (position at TKEStack code hack/lightweight-install/data/ directory) to the corresponding directory cluster /etc/kubernetes/pki/ under.

Modify the content of /etc/kubernetes/mainfest/kube-apiserver.yaml in the K8s cluster, and add the following two parameters to the spec.containers.command field:

 # 如果已有这两个参数，则将其按照以下内容修改
- --authorization-mode=Node,RBAC,Webhook
- --authorization-webhook-config-file=/etc/kubernetes/pki/tke-authz-webhook.yaml

Create a standalone cluster

Access address http://{公网可访问ip}/tkestack , the following login screen appears, enter the user name set before adminUsername and password adminPassword , if not set, the default user name admin , the password is YWRtaW4= .

After logging in, click on the new independent cluster of cluster management:

For specific cluster creation information, refer to the document Cluster Creation .

After the cluster is created, you can see the following status on the page:

And you can query the information of the independent cluster on the master node:

If the Kubernetes version shows no data when creating an independent cluster, you can solve it by adding the following fields to the configmap named cluster-info (namespace is kube-public) (for specific field values, please refer to hack/lightweight-install/ under the patch.yaml file):

 data:
  k8sValidVersions: '["1.21.4-tke.1","1.20.4-tke.1"]'

Summarize

This article introduces how to install TKEStack lightweight on an existing K8s cluster based on the latest v1.9.0 release version of TKEStack, and integrate TKEStack's core functions such as user management, cluster management, and console access.

[Tencent Cloud Native] New products of Yunshuo, new techniques of Yunyan, new activities of Yunyou, and information of cloud appreciation, scan the code to follow the public account of the same name, and get more dry goods in time! !