GitOps delivery based on KubeVela

Author | Dong Tianxin (Fog)
review & proofreading: Xiyang, Haizhu
Editing & Typesetting: Wen Yan

KubeVela is a simple, easy-to-use, and highly scalable cloud-native application management and delivery platform that allows developers to easily and quickly define and deliver modern microservice applications on Kubernetes without knowing any Kubernetes infrastructure-related details.

The OAM model behind KubeVela naturally solves the management problems of complex resource combination and orchestration in the application construction process. At the same time, it also models the later operation and maintenance strategy. This means that KubeVela can combine GitOps to manage complex large-scale applications and converge due to the team The system complexity problem caused by the growth of the system scale.

What is GitOps

Its core idea is to store the declarative description of the infrastructure and application configuration required by the application system in the Git warehouse, and cooperate with an automated process, so that every time the warehouse is updated, the automated process can gradually update the environment to the latest configuration .

This method allows developers to automatically deploy applications by directly changing the code and configuration in the Git repository. Using GitOps can bring many values to application development, such as:

• Increase productivity. Automatic continuous deployment can speed up the average deployment time and increase development efficiency.
• Lower the barriers for developers to deploy. By pushing code instead of container configuration, developers can deploy easily without knowing the internal implementation of Kubernetes.
• Make the change log traceable. Git is used to manage the cluster, so that every change can be tracked, and the audit trail is strengthened.
• The cluster can be restored through Git's rollback/branch function.

KubeVela and GitOps

As a declarative application delivery control plane, KubeVela naturally supports the use of GitOps, and makes users more clearly feel the benefits brought by GitOps, and the end-to-end application delivery and management experience, including:

• Application delivery workflow (CD pipeline): namely: KubeVela supports the description of procedural application delivery in the GitOps model, rather than simply declaring the final state;
• Deal with various dependencies and topological structures in the deployment process;
• Provide a unified upper-level abstraction on top of the semantics of various existing GitOps tools to simplify the application delivery and management process;
• Unified declaration, deployment and service binding of cloud services;
• Provide out-of-the-box delivery strategies (canary, blue-green release, etc.);
• Provide out-of-the-box hybrid cloud/multi-cloud deployment strategies (placement rules, cluster filtering rules, etc.);
• Provide Kustomize-style Patches in multi-environment delivery to describe deployment differences without having to learn any details of Kustomize itself;
• ……

In this article, we mainly explain the steps of directly using KubeVela to deliver in GitOps mode.

GitOps workflow

The GitOps workflow is divided into two parts: CI and CD:

• CI (Continuous Integration): Continuous integration performs code construction on business applications, builds images, and pushes them to the image warehouse. There are many mature CI tools, such as GitHub Action and Travis, which are commonly used in open source projects, and Jenkins and Tekton, which are commonly used in enterprises. In this article, we use GitHub Action to complete the CI step. Of course, you can also use other CI tools instead. KubeVela revolves around GitOps to connect to the CI process under any tool.
• CD (Continuous Delivery): Continuous deployment will automatically update the configuration in the cluster, such as updating the latest image in the mirror warehouse to the cluster. There are currently two main types of CDs:
1) Push-Based: The CD in Push mode is mainly completed by configuring the CI pipeline. In this way, the access key of the cluster needs to be shared with the CI, so that the CI pipeline can push the changes to the cluster through commands. You can refer to Our previous blog: Use Jenkins + KubeVela to complete continuous delivery of applications (see related links at the end of the article for details).
2) Pull-Based: The CD in the pull mode will monitor changes in the warehouse (code warehouse or configuration warehouse) in the cluster, and synchronize these changes to the cluster. Compared with Push mode, Pull-Based is actively pulled by the cluster for updates, thus avoiding the problem of secret key exposure. This is also the core content of this article.

There are two types of delivery personnel, which we will introduce separately:

1. For the delivery of infrastructure for platform administrators/operations and maintenance personnel, users can directly update the configuration files in the warehouse to update the infrastructure configuration in the cluster, such as system dependent software, security policies, storage, network and other infrastructure configurations.
2. For delivery to terminal developers, once the user's code is merged into the application code warehouse, it will automatically trigger the update of the application in the cluster, which can complete the iteration of the application more efficiently, and KubeVela's gray-scale release, traffic allocation, multi-cluster deployment and other functions The combination can form a more powerful automated publishing capability.

Delivery to platform administrators/operations and maintenance personnel

As shown in the figure, for platform administrators/operations and maintenance personnel, they do not need to care about the application code, so they only need to prepare a Git configuration repository and deploy the KubeVela configuration file. Follow-up changes to the application and infrastructure configuration This can be done by directly updating the Git configuration repository, so that every configuration change can be tracked.

Prepare to configure the warehouse

For specific configuration, please refer to example warehouse 1 (see related links at the end of the article for details).

In this example, we will deploy a MySQL database software as the project's infrastructure, while deploying a business application to use this database. The directory structure of the configuration warehouse is as follows:

• The clusters/ contains the KubeVela GitOps configuration in the cluster, and the user needs to manually deploy the files in clusters/ to the cluster. This is a one-time control operation. After execution, KubeVela can automatically monitor file changes in the configuration warehouse and automatically update the configuration in the cluster. Among them, clusters/apps.yaml will monitor the changes of all applications under apps/, and clusters/infra.yaml will monitor the changes of all infrastructures under infrastructure/.
• The apps/ directory contains all the configuration of the business application, in this case a business application that uses a database.
• Infrastructure/ contains some infrastructure-related configurations and strategies, in this case, the MySQL database.

├── apps
│   └── my-app.yaml
├── clusters
│   ├── apps.yaml
│   └── infra.yaml
└── infrastructure
    └── mysql.yaml

KubeVela recommends using the above directory structure to manage your GitOps repository. The related KubeVela GitOps configuration is stored in clusters/ and needs to be manually deployed to the cluster. Apps/ and infrastructure/ store your application and infrastructure configuration respectively. By separating the application from the basic configuration, you can manage your deployment environment more reasonably and isolate the impact of application changes.

clusters/ directory

First, let's take a look at the clusters directory, which is also the initial configuration directory for KubeVela to connect to GitOps.

Take clusters/infra.yaml as an example:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: infra
spec:
  components:
  - name: database-config
    type: kustomize
    properties:
      repoType: git
      # 将此处替换成你需要监听的 git 配置仓库地址
      url: https://github.com/FogDong/KubeVela-GitOps-Infra-Demo
      # 如果是私有仓库，还需要关联 git secret
      # secretRef: git-secret
      # 自动拉取配置的时间间隔，由于基础设施的变动性较小，此处设置为十分钟
      pullInterval: 10m
      git:
        # 监听变动的分支
        branch: main
      # 监听变动的路径，指向仓库中 infrastructure 目录下的文件
      path: ./infrastructure

apps.yaml and infra.yaml are almost the same, except that the file directory to be monitored is different. In apps.yaml, the value of properties.path will be changed to ./apps, indicating that the file changes under the apps/ directory will be monitored.

The GitOps control configuration file in the cluster folder needs to be manually deployed to the cluster at one time during initialization. After that, KubeVela will automatically monitor the configuration files in the apps/ and infrastructure/ directories and update them regularly.

apps/ directory

The apps/ directory stores application configuration files, which is a simple application configured with database information and Ingress. The application will connect to a MySQL database and simply start the service. Under the default service path, the current version number will be displayed. Under the /db path, the information in the current database will be listed.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: my-app
  namespace: default
spec:
  components:
    - name: my-server
      type: webservice
      properties:
        image: ghcr.io/fogdong/test-fog:master-cba5605f-1632714412
        port: 8088
        env:
          - name: DB_HOST
            value: mysql-cluster-mysql.default.svc.cluster.local:3306
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-secret
                key: ROOT_PASSWORD
      traits:
        - type: ingress
          properties:
            domain: testsvc.example.com
            http:
              /: 8088

This is an application that uses KubeVela's built-in component type webservice, which is bound to Ingress operation and maintenance features. By declaring the operation and maintenance capabilities in the application, only one file is required to gather the underlying Deployment, Service, and Ingress, so as to manage the application more conveniently.

infrastructure/ directory

Some infrastructure configurations are stored in the infrastructure/ directory. Here we use the mysql controller (see related links at the end of the article for details) to deploy a MySQL cluster.

Note, please make sure that there is a secret in your cluster and the MySQL password is declared through ROOT_PASSWORD.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: mysql
  namespace: default
spec:
  components:
    - name: mysql-controller
      type: helm
      properties:
        repoType: helm
        url: https://presslabs.github.io/charts
        chart: mysql-operator
        version: "0.4.0"
    - name: mysql-cluster
      type: raw
      dependsOn:
        - mysql-controller
      properties:
        apiVersion: mysql.presslabs.org/v1alpha1
        kind: MysqlCluster
        metadata:
          name: mysql-cluster
        spec:
          replicas: 1
          # 关联 secret 名称
          secretName: mysql-secret

In this MySQL application, we use the KubeVela workflow capabilities. The workflow is divided into two steps, the first step is to deploy the MySQL controller. After the controller is successfully deployed and running correctly, the second step will begin to deploy the MySQL cluster.

Deploy files in the clusters/ directory

After configuring the above files and storing them in the Git configuration repository, we need to manually deploy the KubeVela GitOps configuration file under the clusters/ directory in the cluster.

First, deploy clusters/infra.yaml in the cluster. You can see that it automatically pulls up the MySQL deployment files in the infrastructure/ directory in the cluster:

kubectl apply -f clusters/infra.yaml

$ vela ls
APP     COMPONENT           TYPE        TRAITS  PHASE   HEALTHY STATUS  CREATED-TIME
infra   database-config     kustomize           running healthy         2021-09-26 20:48:09 +0800 CST
mysql   mysql-controller    helm                running healthy         2021-09-26 20:48:11 +0800 CST
└─      mysql-cluster       raw                 running healthy         2021-09-26 20:48:11 +0800 CST

Then, deploy in the cluster

clusters/apps.yaml, you can see that it automatically pulls up the application deployment files in the apps/ directory:

kubectl apply -f clusters/apps.yaml

APP     COMPONENT           TYPE        TRAITS  PHASE   HEALTHY STATUS  CREATED-TIME
apps    apps                kustomize           running healthy         2021-09-27 16:55:53 +0800 CST
infra   database-config     kustomize           running healthy         2021-09-26 20:48:09 +0800 CST
my-app  my-server           webservice  ingress running healthy         2021-09-27 16:55:55 +0800 CST
mysql   mysql-controller    helm                running healthy         2021-09-26 20:48:11 +0800 CST
└─      mysql-cluster       raw                 running healthy         2021-09-26 20:48:11 +0800 CST

So far, we deployed the KubeVela GitOps configuration file to automatically pull up the application and database in the cluster.

Through the Ingress of the curl application, you can see that the current version is 0.1.5 and successfully connected to the database:

$ kubectl get ingress
NAME        CLASS    HOSTS                 ADDRESS         PORTS   AGE
my-server   <none>   testsvc.example.com   <ingress-ip>    80      162m
$ curl -H "Host:testsvc.example.com" http://<ingress-ip>
Version: 0.1.5
$ curl -H "Host:testsvc.example.com" http://<ingress-ip>/db
User: KubeVela
Description: It's a test user

Change setting

After completing the first deployment, we can complete the configuration update of the application in the cluster by modifying the configuration in the configuration warehouse.

Modify the domain of the application Ingress:

...
      traits:
        - type: ingress
          properties:
            domain: kubevela.example.com
            http:
              /: 8089

After a period of time, check the Ingress in the cluster again:

NAME        CLASS    HOSTS                 ADDRESS         PORTS   AGE
my-server   <none>   kubevela.example.com  <ingress-ip>    80      162m

As you can see, the Host address of Ingress has been successfully updated.

In this way, we can easily update the files in the Git configuration repository to automatically update the configuration in the cluster.

Delivery for terminal developers

For terminal developers, in addition to the KubeVela Git configuration warehouse, an application code warehouse is also required. As shown in the figure, after the user updates the code in the application code warehouse, a CI needs to be configured to automatically build the image and push it to the image warehouse. KubeVela will monitor the latest mirror in the mirror warehouse, and automatically update the mirror configuration in the configuration warehouse, and finally update the application configuration in the cluster. Allow users to achieve the effect that the configuration in the cluster is automatically updated after the code is updated.

Prepare code warehouse

Prepare a code repository, which contains some source code and the corresponding Dockerfile.

These codes will connect to a MySQL database and simply start the service. Under the default service path, the current version number will be displayed. Under the /db path, the information in the current database will be listed.

http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        _, _ = fmt.Fprintf(w, "Version: %s\n", VERSION)
    })
    http.HandleFunc("/db", func(w http.ResponseWriter, r *http.Request) {
        rows, err := db.Query("select * from userinfo;")
        if err != nil {
            _, _ = fmt.Fprintf(w, "Error: %v\n", err)
        }
        for rows.Next() {
            var username string
            var desc string
            err = rows.Scan(&username, &desc)
            if err != nil {
                _, _ = fmt.Fprintf(w, "Scan Error: %v\n", err)
            }
            _, _ = fmt.Fprintf(w, "User: %s \nDescription: %s\n\n", username, desc)
        }
    })
    if err := http.ListenAndServe(":8088", nil); err != nil {
        panic(err.Error())
    }

We hope that after the user changes the code and submits it, the latest mirror will be automatically constructed and pushed to the mirror warehouse. This step of CI can be achieved by integrating GitHub Actions, Jenkins or other CI tools. In this example, we use GitHub Actions to complete continuous integration. For specific code files and configurations, please refer to Sample Warehouse 2 (see related links at the end of the article for details).

Configure key information

After the new image is pushed to the mirror warehouse, KubeVela will recognize the new image and update the Application configuration files in the warehouse and cluster. Therefore, we need a Secret containing Git information to enable KubeVela to submit to the Git repository. Deploy the following files and replace the username and password with your Git username and password (or Token):

apiVersion: v1
kind: Secret
metadata:
  name: git-secret
type: kubernetes.io/basic-auth
stringData:
  username: <your username>
  password: <your password>

Prepare to configure the warehouse

The configuration warehouse is similar to the previous configuration for operation and maintenance personnel, only the configuration related to the mirror warehouse needs to be added. For specific configuration, please refer to Sample Warehouse 1 (see related links at the end of the article for details).

Modify apps.yaml in clusters/. The GitOps configuration will monitor changes in the application files in apps/ in the warehouse and mirror updates in the mirror warehouse:

...
  imageRepository:
    # 镜像地址
    image: <your image>
    # 如果这是一个私有的镜像仓库，可以通过 `kubectl create secret docker-registry` 创建对应的镜像秘钥并相关联
    # secretRef: imagesecret
    filterTags:
      # 可对镜像 tag 进行过滤
      pattern: '^master-[a-f0-9]+-(?P<ts>[0-9]+)'
      extract: '$ts'
    # 通过 policy 筛选出最新的镜像 Tag 并用于更新
    policy:
      numerical:
        order: asc
    # 追加提交信息
    commitMessage: "Image: {{range .Updated.Images}}{{println .}}{{end}}"

Modify the image field in apps/my-app.yaml and add the comment # {"$imagepolicy": "default:apps"} after it. KubeVela will use this comment to update the corresponding mirror field. default:apps is the namespace and name corresponding to the above GitOps configuration.

spec:
  components:
    - name: my-server
      type: webservice
      properties:
        image: ghcr.io/fogdong/test-fog:master-cba5605f-1632714412 # {"$imagepolicy": "default:apps"}

After updating the file containing the mirror warehouse configuration in clusters/ to the cluster, we can update the application by modifying the code.

Modify the code

Change the Version in the code file to 0.1.6, and modify the data in the database:

const VERSION = "0.1.6"
...
func InsertInitData(db *sql.DB) {
    stmt, err := db.Prepare(insertInitData)
    if err != nil {
        panic(err)
    }
    defer stmt.Close()
    _, err = stmt.Exec("KubeVela2", "It's another test user")
    if err != nil {
        panic(err)
    }
}

Submit the change to the code repository, and you can see that the CI pipeline we configured starts to build the mirror and push it to the mirror repository.

And KubeVela will monitor the mirror repository and update the application my-app under apps/ in the configuration repository according to the latest mirror Tag.

At this point, you can see that there is a submission from kubevelabot in the configuration warehouse, and the submission information is prefixed with Update image automatically. You can also add the information you need in the commitMessage field by {{range .Updated.Images}}{{println .}}{{end}}.

It is worth noting that if you want to put the code and configuration in the same warehouse, you need to filter out submissions from kubevelabot to prevent repeated builds in the pipeline. You can filter through the following configuration in CI:

jobs:
publish:
  if: "!contains(github.event.head_commit.message, 'Update image automatically')"

Check the applications in the cluster again, you can see that after a period of time, the image of the application my-app has been updated.

KubeVela will use the interval you configured to obtain the latest information from the configuration warehouse and the mirror warehouse at regular intervals:

• When the configuration file in the Git repository is updated, KubeVela will update the applications in the cluster according to the latest configuration.
• When a new tag is added to the mirror repository, KubeVela will filter out the latest mirror tag according to the policy rules you configured, and update it to the Git repository. When the files in the code warehouse are updated, KubeVela will repeat the first step to update the files in the cluster, thus achieving the effect of automatic deployment.

View the current version and database information through the Ingress corresponding to curl:

$ kubectl get ingress
NAME        CLASS    HOSTS                 ADDRESS         PORTS   AGE
my-server   <none>   kubevela.example.com  <ingress-ip>    80      162m
$ curl -H "Host:kubevela.example.com" http://<ingress-ip>
Version: 0.1.6
$ curl -H "Host:kubevela.example.com" http://<ingress-ip>/db
User: KubeVela
Description: It's a test user
User: KubeVela2
Description: It's another test user

The version has been successfully updated! So far, we have completed all operations from changing the code to automatically deploying to the cluster.

Summarize

On the operation and maintenance side, if you need to update the configuration of the infrastructure (such as the database) or the configuration items of the application, you only need to modify the files in the configuration warehouse, and KubeVela will automatically synchronize the configuration to the cluster, simplifying the deployment process.

On the R&D side, after users modify the code in the code warehouse, KubeVela will automatically update the mirror in the configuration warehouse. In order to update the version of the application.

By combining with GitOps, KubeVela accelerates the entire process from application development to deployment.

Click on the original text to view the official homepage and documentation of the KubeVela project! !

Related Links
1) Use Jenkins + KubeVela to complete the continuous delivery of the application:
https://kubevela.io/zh/blog/2021/09/02/kubevela-jenkins-cicd
2) Example warehouse 1:
https://github.com/oam-dev/samples/tree/master/9.GitOps_Demo/for-SREs
3）mysql controller：
https://github.com/bitpoke/mysql-operator
4) Example warehouse 2:
https://github.com/oam-dev/samples/tree/master/9.GitOps_Demo/for-developers/app-code

For more related information, please scan the QR code below or search for the Dingding group number (35922870) to join the Alibaba Cloud native information exchange group! Get more information!