GitOps Application Practice Series-Flux CD and its core components

Hi everyone, this is Zhang Jintao.

After the previous three articles, I not only introduced what GitOps is, but also how to implement GitOps using Argo CD. In this article, I will introduce you another tool that can be used to implement GitOps: Flux CD.

Flux CD

Flux is a set of tools that can support the implementation of GitOps. It is used to Kubernetes cluster with the configuration source (such as the Git repository), and automatically synchronize the configuration after the code is updated. The -oriented 161ca7dbc4fcbd continuous and progressive delivery solution .

The history of Flux CD

• On October 28, 2016, the Flux single-user service version was released.

It laid the two keynotes of flux:

• • Centralized service
  • Run in the k8s cluster in automatic mode as a daemon process

• On December 15, 2016, "Continuous Delivery with Weave Flux" was released, building a Flux that links CI and continuous deployment (CD).

​

• On August 22, 2017, the v1.0.0 version was officially released.

Since v1.0.0, Flux is committed to synchronizing the cluster with the configuration stored in Git, and automatically upgrade the image when the new version is ready for deployment. (Proposed: Configuration as code)

​

• Helm Operator was integrated in the alpha version released on May 1, 2018. This is the first alpha label version of Flux Helm Operator.
• On August 15, 2019, Flux announced to join the CNCF Sandbox. As developers and companies begin to implement GitOps, the number of Flux users continues to grow. At that time, there were more than 2500 GitHub stars, and they were continuously integrated: Helm Operator, Kustomize, Weave Flagger, OpenFaaS, Fluxcloud, Flux Web UI, etc.
• The version released on October 2, 2019 officially supports kustomize.
• On November 14, 2019, in "Introduction to Argo Flux-Weaveworks-Intuit-AWS Collaboration", Weaveworks announced the creation of Argo Flux in cooperation with Intuit. The project is called GitOps-Engine and is now in the Argo project organization, driven by Intuit, Red Hat, and GitLab. (For details, please refer to: https://www.weave.works/blog/argo-flux-join-forces)
• On October 5, 2020, Flux v1 (and Helm Operator v1) announced that it was in maintenance mode (only 6 months of support), and announced that it would work on the development of Flux v2.

• • Flux v1 is in maintenance mode #3320 https://github.com/fluxcd/flux/issues/3320
  • Helm Operator (v1) is in maintenance mode #546 https://github.com/fluxcd/helm-operator/issues/546
• • V1 in maintenance #25 https://github.com/fluxcd/website/pull/25

• The new version was released on February 17, 2021, officially pointing issue and PR to v2.
• On March 11, 2021, CNCF Technical Oversight Committee (TOC) voted to upgrade Flux from Sandbox to an incubation project (Incubation). Since joining CNCF Sandbox, Flux's end user base has increased by 2.75 times and its community has expanded by 2 to 4 times, including Slack users, mailing list subscribers and contributors. More than 80 organizations use it in production, including Babylon Health, Fidelity Investments, MyFitnessPal, Starbucks, etc. CNCF End User Community incorporates Flux into the Technology Radar of continuous delivery.

​

• On August 13, 2021, "Flux v2 helps GitOps become better" was officially released, announcing the transition from Flux to Flux v2. Based on the existing functions of Flux, Flux v2 is more suitable for Kubernetes operation and observability , and has more powerful functions. The foundation of Flux v2 is the GitOps Toolkit, which is a set of components used to build GitOps-based delivery and automation. (Also realized the decoupling of important components, please refer to: https://www.weave.works/blog/gitops-with-flux-v2)

​

​ Note: The Flux team decided to move on without using GitOps Engine and built GitOps Toolkit . Flux v2 refers to GitOps Engine, but it does not use GitOps Engine.

• The Flagger project that started in October 2018 will also be integrated into Flux v2 in 2021.

​

​

Flux CD v2 and GitOps Toolkit

GitOps Toolkit

The GitOps Toolkit is a set of APIs and controllers that make up the Flux runtime. You can extend Flux through the GitOps Toolkit and build the required CD system.

Source Controller

The main function of Source Controller is to provide a common interface for artifact acquisition. The Source API defines a set of Kubernetes objects that cluster admin and various automation operators can interact with to offload Source (such as Git and Helm repo) registration, authentication, verification, and resource acquisition to a dedicated controller.

Each operator of Flux and Helm uses the code (mirror) in Git repositories in the same way, but other components may need to access different mirrors. Flux and Helm operators can use standard Kubernetes mechanisms to build Sources (usually Git repo, but also Helm charts, etc.) as Kubernetes resources, independently of the components that use them for management.

Git Repository

It defines a source for artifacts from Git, and exposes the latest state from Git as artifacts as a gzip compressed TAR file (<commit hash>.tar.gz).

Git files (.git/, .gitignore, .gitmodules, .gitattributes) are excluded by default; file extensions (.jpg, .jpeg, .gif, .png, .wmv, .flv, .tar.gz,. zip); CI configuration (.github/, .circleci/, .travis.yml, .gitlab-ci.yml, appveyor.yml, .drone.yml, cloudbuild.yaml, codeship-services.yml, codeship-steps.yml ); CLI configuration (.goreleaser.yml, .sops.yaml); Flux v1 configuration (.flux.yaml).

If you want to customize exclusion, you can consider the following two methods:

• Add files to the root directory of the repository through .sourceignore (Follow the .gitignore format, see https://git-scm.com/docs/gitignore#_pattern_format )
• Use the spec.ignore field.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 5m
  url: https://github.com/stefanprodan/podinfo
  ignore: |
    ## exclude all
    /*
    ## include deploy dir
    !/deploy
    ## exclude file extensions from deploy dir
    /deploy/**/*.md
    /deploy/**/*.txt  

Helm Charts

It defines a source for Helm chart artifacts from Helm source, which will be published as the latest pulled or packaged chart of artifacts.

Helm Repositories

It defines a source for Helm repositories that will be exposed as the latest synchronized repository index of artifacts.

Object storage buckets

It defines a source for S3-compatible storage (Minio, Amazon S3, Google Cloud Storage, Alibaba Cloud OSS, etc.), and exposes the latest synchronization status from S3 as an artifact as a gzip-compressed TAR file (<bucket checksum>. tar.gz).

For default and custom excluded files, please refer to the related content of excluded files in Git Repositories.

Kustomize Controller

Kustomize-controller is a Kubernetes operator dedicated to running a continuous delivery pipeline defined by using Kustomize to assemble Kubernetes lists and workloads. It provides an automated operator that can guide and continuously coordinate the cluster state from multiple sources (such as infrastructure and application repositories).

When configuring a new cluster, you can install workloads in a specific order. When multiple teams operate the cluster at the same time, the cluster administrator can assign roles and service accounts to each team. The manifests owned by the team will be applied to the cluster using the team account to ensure isolation between the teams.

When processing events, you can choose not to stop the coordinator to affect the entire cluster, and use the method of suspending the coordination of certain workloads and fixing the coordination of other workloads to the specified Git version.

When operating a cluster, different teams can receive related CD pipeline status notifications.

The Kustomization API defines a pipeline that can obtain, decrypt, build, verify and apply Kubernetes manifests.

spec.sourceRef is a reference to an object managed by the Source Controller. When the Source (supported types: GitRepository, Bucket) version changes, it will generate a Kubernetes event to trigger the construction and application of kustomize.

If the repository contains the original Kubernetes manifest, it will provide spec.path the manifest automatically generate all Kubernetes and subdirectories kustomization.yaml . (It is recommended that kustomization.yaml be generated and stored in the Git repository)

spec.interval tells the Controller at which time interval to obtain the Kubernetes manifest for the Source, build Kustomization and apply it to the cluster. The unit of interval time is s (the minimum value should exceed 60 seconds, interval: 60s).

spec.healthChecks , spec.wait and spec.timeout can set a series of health checks.

• Kubernetes built-in (Deployment, DaemonSet, StatefulSet, PersistentVolumeClaim, Pod, PodDisruptionBudget, Job, CronJob, Service, Secret, ConfigMap, CustomResourceDefinition)
• Toolkit (HelmRelease, HelmRepository, GitRepository, etc.)
• Custom resources compatible with kstatus ( https://github.com/kubernetes-sigs/cli-utils/tree/master/pkg/kstatus)

spec.dependsOn can customize dependencies. In the case of multiple Kustomizations, the order and dependencies can be specified. This introduces two things that must be paid attention to:

• Avoid loops.
• When combined with a health check, Kustomization will run after all its dependencies pass the health check.

Kustomization also has many flexible settings for covering custom configuration, variable substitution, encryption and decryption, etc. Interested friends can read the official documents to understand.

Helm Controller

Helm Controller is a Kubernetes operator that allows the use of Kubernetes manifests to manage Helm chart Release in a declarative manner. It provides an automated operator that can perform Helm operations (such as installation, upgrade, uninstall, rollback, testing) and continuously coordinate the status of Helm Release.

When configuring a new cluster, you can specify the order in which Helm Release is installed. When processing an event, you don't need to stop the coordinator to affect the entire cluster, just suspend some Helm Release coordination. When operating a cluster, different teams can receive notifications of Helm Release status related to them.

The HelmRelease object defines a Helm Release coordination driven by a controller resource through Helm operations (such as install, upgrade, test, uninstall, and rollback). Including release location (namespace/name), release content (chart/values ​​overrides), operation trigger configuration, individual operation configuration and status.

The default deployment and storage Helm Release namespace/name can be spec.targetNamespace by 061ca7dbc50511, spec.storageNamespace and spec.releaseName

spec.chart.spec is a template for creating a new HelmChart resource with the specified spec. spec.chart.spec.sourceRef is a reference to an object managed by the Source Controller. When the source (supported types: HelmRepository, GitRepository, Bucket) version changes, a Kubernetes event that triggers the new version will be generated.

If Helm Release with matching namespace/name is not found, Helm Release in the configuration will be installed.

Helm Release will be upgraded when the following status updates occur:

• spec (and metadata.generation)
• The latest HelmChart revision is available
• ConfigMap and Secret value override (in order to avoid a large number of upgrades when updating multiple resources, it will not be performed immediately, but will be performed after the interval of spec.interval)

Helm supports the installation of CRD. Due to the risk of data loss, currently it does not support the use of Helm to upgrade or delete CRD.

For more details about Helm Controller, please refer to the official documentation.

Notification Controller

Notification Controller is a Kubernetes operator, specializing in handling entrance and exit events.

It provides a notification service that can receive events via HTTP and dispatch them to external systems (Slack, Microsoft Teams, Discord, Rocker) based on the severity of the event and the objects involved.

GitOps Controller is essentially pull-based. In order to notify Controllers about changes in Git or Helm libraries, it can support setting webhooks to trigger cluster coordination every time the source changes.

Image reflector and automation controllers

image-reflector-controller and image-automation-controller work together to update the Git repository when new container images are available.

• image-reflector-controller scans the mirror warehouse and compares it with the metadata in the K8S resource.
• The image-automation-controller updates the YAML file and submits the changes to the specified Git repository according to the latest image scanned.

Flux CD usage requirements

Kubernetes version

To use Flux CD, the best Kubernetes cluster version is v 1.19 or higher. The old version is also supported, but it is officially not recommended to run the EOL Kubernetes version in production.

Note: If you are using APIService (such as metrics-server ), the Kubernetes cluster version needs at least v1.18.18, v1.19.10, v1.20.6 or v1.21.0.

Git Repo permissions

Configure the target cluster to synchronize with the Git repository by setting up SSH deployment keys or using token-based authentication.

• GitLab usage example

//将 GitLab personal access token 导出为环境变量：
export GITLAB_TOKEN=<your-token>
//在 GitLab 账号上运行 git 库的引导程序：
flux bootstrap gitlab \
  --owner=my-gitlab-username \
  --repository=my-repository \
  --branch=master \
  --path=clusters/my-cluster \
  --token-auth \
  --personal
//使用部署密钥进行身份验证运行 git 库的引导程序，必须指定 SSH hostname：
flux bootstrap gitlab \
  --ssh-hostname=gitlab.com \
  --owner=my-gitlab-username \
  --repository=my-repository \
  --branch=master \
  --path=clusters/my-cluster
//为 GitLab 团队的 git 库运行引导程序：
flux bootstrap gitlab \
  --owner=my-gitlab-group/my-gitlab-subgroup \
  --repository=my-repository \
  --branch=master \
  --path=clusters/my-cluster
//为托管在本地或企业 GitLab 上的 git 库运行引导程序，必须指定 GitLab hostname：
flux bootstrap gitlab \
  --hostname=my-gitlab.com \
  --token-auth \
  --owner=my-gitlab-group \
  --repository=my-repository \
  --branch=master \
  --path=clusters/my-cluster

• GitHub usage example

//将 GitHub personal access token 导出为环境变量：
export GITHUB_TOKEN=<your-token>
//flux bootstrap 命令会创建一个 SSH 密钥，并存储在 Kubernetes 集群中。
//该密钥还用于在 GitHub 库中创建部署密钥。
//在 GitHub 账号上运行 git 库的引导程序：
flux bootstrap github \
  --owner=my-github-username \
  --repository=my-repository \
  --path=clusters/my-cluster \
  --personal
//当指定团队列表时，这些团队将被授予对 git 库的维护者访问权限。
//为 GitHub 中团队拥有的 git 库运行引导程序：
flux bootstrap github \
  --owner=my-github-organization \
  --repository=my-repository \
  --team=team1-slug \
  --team=team2-slug \
  --path=clusters/my-cluster

Summarize

This article mainly introduces the development history of Flux CD and its core components and functions. Follow-up articles will introduce how to use Flux CD to practice GitOps, so stay tuned.