KubeSphere + Argo CD, realize real GitOps!

Sharing from community user willqy

Introduction to Argo CD

Argo CD is a declarative GitOps continuous delivery tool for Kubernetes. The application definition, configuration, and environment should be declarative and subject to version control. Application deployment and lifecycle management should be automated, auditable, and easy to understand.

Argo CD follows the GitOps model, which uses Git repositories as the source of truth for defining the desired application state.

Argo CD can automatically deploy the required application state in the specified target environment. Application deployment can track the branch, tag update, or pin to a specific version of the list when Git is submitted.

Official website: https://argoproj.github.io/

Argo CD architecture diagram:

Argo CD is implemented as a Kubernetes controller that continuously monitors the running application and compares the current activity state with the desired target state (specified in the Git repository). When the running state of the deployed application deviates from the target state, it will be regarded as OutOfSync by Argo CD.

Argo CD reports and visualizes the differences, while providing the ability to automatically or manually synchronize the real-time status back to the desired target status. Any modifications made to the desired target state in the Git repository can be automatically applied and synchronized to the specified target environment.

The Kubernetes configuration list supported by Argo CD includes helm charts, kustomize or pure YAML/json files.

This article involves content:

• Use KubeSphere DevOps to implement the CI part, and the CD part is completed by Argo CD;
• Argo CD continuously monitors the changes of the yaml file in a directory of the Git warehouse, and automatically deploys the yaml file to the K8s cluster;
• Argo CD continuously monitors the changes of a certain mirror tag in the Harbor mirror warehouse, and automatically deploys the latest mirror to the K8s cluster.

Basic schematic diagram:

Prepare Git code repository

Prepare 2 Git repositories, a source code repository, and a yaml file repository. The source code and the yaml file are separated.

The source code repository can refer to the following link, for offline environment reasons, here is the second example spring-demo:

• https://github.com/KubeSphere/DevOps-java-sample
• https://github.com/willzhang/spring-demo

The yaml file warehouse can refer to the following link, which is named argocd-gitops:

• https://github.com/argoproj/argocd-example-apps

Create a javademo directory under the yaml warehouse, and create 2 simple yaml files:

[root@jenkins git]# tree argocd-gitops/
argocd-gitops/
├── javademo
│   ├── javademo-deployment.yaml
│   └── javademo-svc.yaml

javademo-deployment.yaml example, the current mirroring tag can be arbitrarily specified, and this parameter will be replaced in real time when CI is executed:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: javademo
spec:
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: javademo
  template:
    metadata:
      labels:
        app: javademo
    spec:
      containers:
      - image: 10.39.140.196:8081/apps/javademo:replace
        name: javademo
        ports:
        - containerPort: 8080

javademo-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: javademo
spec:
  type: NodePort
  ports:
  - port: 8012
    targetPort: 8080
  selector:
    app: javademo

Deploy Argo CD

Argo CD has multiple deployment methods, you can deploy yaml files directly.

kubectl create namespace Argo CD
kubectl apply -n Argo CD -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Here we use the helm method to deploy, you can directly specify the Argo CD server service type nodePort:

helm repo add argo https://argoproj.github.io/argo-helm

helm install Argo CD \
  --namespace=Argo CD --create-namespace \
  --set server.service.type=NodePort \
  argo/argo-cd

View the running Pod:

[root@master ~]# kubectl -n Argo CD get pods
NAME                                             READY   STATUS    RESTARTS   AGE
argocd-application-controller-5db8c6f8f9-qnmtr   1/1     Running   0          8h
argocd-dex-server-84b5cbfbc9-fc7rf               1/1     Running   0          8h
argocd-redis-7c7c79dcd9-hjhgr                    1/1     Running   0          8h
argocd-repo-server-5fb9cbb945-9xmc7              1/1     Running   0          8h
argocd-server-8d8cb6488-pjwt4                    1/1     Running   0          8h

If you use KubeSphere to deploy Argo CD, you first need to configure the Argo CD helm warehouse, enter the corporate space, select the application template to upload the offline helm chart package, or configure the public network helm repo address in the application warehouse.

After completion, enter the project, click to deploy a new application, and select Argo CD helm chart to deploy:

Install Argo CD CLI

To interact with Argo CD API Server, we need to install CLI commands:

wget https://github.com/argoproj/argo-cd/releases/download/v1.7.10/argocd-linux-amd64
cp argocd-linux-amd64 /usr/local/bin/Argo CD
chmod +x /usr/local/bin/Argo CD

Argo CD version

If the Argo CD above is deployed in yaml mode, change the serivce type to nodeport in order to access the Argo CD API Server

kubectl patch svc argocd-server -n Argo CD -p '{"spec": {"type": "NodePort"}}'

View Argo CD server service, record nodeport information:

[root@master ~ ]# kubectl -n Argo CD get svc
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
argocd-dex-server       ClusterIP   10.99.232.27     <none>        5556/TCP,5557/TCP,5558/TCP   5d
argocd-metrics          ClusterIP   10.107.37.4      <none>        8082/TCP                     5d
argocd-redis            ClusterIP   10.106.160.6     <none>        6379/TCP                     5d
argocd-repo-server      ClusterIP   10.100.101.100   <none>        8081/TCP,8084/TCP            5d
argocd-server           NodePort    10.106.141.243   <none>        80:31195/TCP,443:32079/TCP   5d
argocd-server-metrics   ClusterIP   10.109.81.234    <none>        8083/TCP                     5d

Argo CD default login user is admin, the initial password is argocd-server Pod name, get the Pod name

podName=`kubectl get pods -n Argo CD -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2`

Log in using Argo CD CLI, and use nodeIP and nodePort as the login address of Argo CD server:

Argo CD login 10.39.140.248:31195 --username admin --password $podName

Modify the default password:

Argo CD account update-password \
  --current-password $podName \
  --new-password Argo CD@123

Browser login Argo CD UI:

https://10.39.140.248:31195

Deploy the Argo CD application

After logging in to the Argo CD UI, select NEW APP to create an application, and select EDIT AS AYML:

Paste the following content, click CREATE on the upper left after SAVE, of course, you can also directly use the kubectl apply command to execute the following content, the effect is the same.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: javademo
  namespace: Argo CD
  finalizers:
    - resources-finalizer.Argo CD.argoproj.io
spec:
  project: default
  source:
    path: javademo
    repoURL: http://10.39.140.196:10080/gogs/argocd-gitops.git
    targetRevision: HEAD
  destination:
    namespace: apps
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
    - Validate=false
    - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Parameter Description:

• metadata field: The application name is specified, and the namespace must specify Argo CD. Adding the finalizers field can cascade and delete related k8s resources when deleting the application;
• Source field: Specify the URL of the git warehouse where the yaml file is located, and the directory where the yaml file to be monitored is stored. Any changes to the files in this directory will automatically be updated and deployed to the k8s cluster by Argo CD;
• Destination field: specify which k8s cluster and which namespace to deploy the monitored yaml file to;
• syncPolicy field: Specify the automatic synchronization policy and frequency. If you do not configure it, you need to manually trigger the synchronization.

In addition, if you use a private Git repository, you need to create a credential. The credential here is the credential for Argo CD to access the yaml file Git repository:

The equivalent Argo CD cli command:

Argo CD repo add http://10.39.140.196:10080/gogs/argocd-gitops --username gogs --password xxxxxx

After creation, Argo CD will automatically deploy the yaml file in the javademo directory of the Git warehouse to the K8s cluster. At this time, the application cannot start normally because the image tag in the yaml file does not exist yet, and the image pull will fail:

You can also use the Argo CD CLI to view the deployed applications:

[root@master ~]# Argo CD app get javademo
Name:               javademo
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          apps
URL:                https://10.39.140.248:31195/applications/javademo
Repo:               http://10.39.140.196:10080/gogs/argocd-gitops.git
Target:             HEAD
Path:               javademo
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to HEAD (1b96380)
Health Status:      Progressing

GROUP  KIND        NAMESPACE  NAME      STATUS  HEALTH       HOOK  MESSAGE
       Service     apps       javademo  Synced  Healthy            service/javademo unchanged
apps   Deployment  apps       javademo  Synced  Progressing        deployment.apps/javademo unchanged

Check the Pod status on the KubeSphere UI, and have been retrying to pull the image:

Use the kubectl command to view, the status is ImagePullBackOff:

[root@master ~]# kubectl -n apps get pods
NAME                       READY   STATUS             RESTARTS   AGE
javademo-64d46bff8-6dgjn   0/1     ImagePullBackOff   0          13m

[root@master ~]# kubectl -n apps get svc
NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
javademo   ClusterIP   10.111.56.180   <none>        8012/TCP   33m

KubeSphere creates a pipeline

Create a CI pipeline, use KubeSphere DevOps to complete source code compilation, image construction and push to the Harbor warehouse, and finally update the image field in the yaml warehouse by git commit.

Since Argo CD continues to monitor the changes in the yaml warehouse configuration file at this time, when the CI part executes git push, it will trigger Argo CD to update the yaml file to the K8s cluster.

Create an empty pipeline under the KubeSphere DevOps project, name it javademo, enter the pipeline, choose to edit the Jenkinsfile, and copy the following content:

pipeline {

    environment {
        GIT_URL='http://10.39.140.196:10080/gogs/spring-demo.git'
        GIT_CREDENTIAL_ID = 'git-id'
        GIT_BRANCH = 'master'
        REGISTRY = '10.39.140.196:8081/apps/javademo'
        REGISTRY_CREDENTIAL_ID = 'harbor-id'
    }

    agent {
        node {
            label 'maven'
        }
    }

    stages {

        stage('SCM Checkout') {
            steps {
                git branch: "${GIT_BRANCH}", credentialsId: "${GIT_CREDENTIAL_ID}", url: "${GIT_URL}"
            }
        }

        stage('source build') {
            steps {
                container('maven') {
                    sh 'mvn clean package'

                }
            }
        }

        stage('docker build & push') {
            steps {
                script {
                    env.COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
                    env.TIMESTRAP = sh(returnStdout: true, script: 'date +%Y%m%d%H%M%S').trim()
                    env.DOCKER_TAG = "dev_${TIMESTRAP}_${COMMIT_ID}_${BUILD_NUMBER}"
                }
                container('maven') {
                    withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$REGISTRY_CREDENTIAL_ID" ,)]) {
                        sh 'docker build -t $REGISTRY:$DOCKER_TAG .'
                        sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
                        sh 'docker push $REGISTRY:$DOCKER_TAG'
                    }
                }
            }
        }

        stage('update docker tag') {
            environment {
                BUILD_USER = 'admin'
                BUILD_USER_EMAIL = 'admin@Argo CD.com'
                YAML_REPO_URL='http://${username}:${password}@10.39.140.196:10080/gogs/argocd-gitops.git'
            }

            steps {
                withCredentials([usernamePassword(passwordVariable : 'password' ,usernameVariable : 'username' ,credentialsId : "$GIT_CREDENTIAL_ID" ,)]) {
                    sh """
                        git config --global user.name "$BUILD_USER"
                        git config --global user.email "$BUILD_USER_EMAIL"
                        git clone ${YAML_REPO_URL} && cd argocd-gitops
                        sed -i "s#$REGISTRY.*#${REGISTRY}:${DOCKER_TAG}#g" javademo/javademo-deployment.yaml
                        git add -A && git commit -m "update tag: ${DOCKER_TAG}" && git push ${YAML_REPO_URL}
                    """
                }
            }
        }
    }
}

Pay attention to modify the relevant parameters, there are 2 vouchers quoted in the pipeline:

• GIT_CREDENTIAL_ID is the account password of the gogs git warehouse in the intranet
• REGISTRY_CREDENTIAL_ID is the harbor warehouse account password

Before running the pipeline, you need to create relevant credentials in advance under the DevOps project, and then you need to reference them in the jenkinsfile.

The final pipeline is as follows, click Run, wait for the pipeline execution to complete, and check the status as successful:

Looking at the pipeline construction log, you can see that the following process has been performed. In the last update docker tag step, two key operations have been performed. The sed command replaces the mirror tag, and then git push is executed to update the yaml warehouse.

View the image pushed to the Harbor warehouse:

After Argo CD detects that the yaml file has changed, it is updated to the K8s cluster:

Argo CD UI View the image used:

Log in to KubeSphere UI to view the application status as running:

Modifying the yaml file configuration directly in the Git warehouse can also trigger Argo CD synchronization. For example, change the service type to nodePort:

Waiting for the Argo CD to automatically synchronize the configuration and update to the K8s cluster, and the browser will access the java web application in nodeport mode:

Deploy Argo CD Image Updater

The above demonstrates based on Git repository changes as the source of truth for application deployment, and the following demonstrates another way to use mirror tag changes as the source of truth for application deployment. Argo CD provides a Argo CD Image Updater for this operation.

Argo CD image updater is a that automatically updates Kubernetes workload container images managed by 160ebfd9ca78a0 Argo CD

The tool is currently under development and has the following features and limitations:

• Only the image of the application managed by Argo CD and by 160ebfd9ca78db Helm or Kustomize
• Default support for widely used container warehouses: dockerhub, harbor private mirror warehouse, etc.;
• Ability to use the matcher function to filter the list of tags returned by the mirror warehouse;
• The image pull secrets must exist in the same Kubernetes cluster where Argo CD Image Updater is running (or accessible). It is currently impossible to obtain these secrets from other clusters.
• In the current version, Argo CD Image Updater will not write any changes back to the Git repository.

Official documents:

https://argocd-image-updater.readthedocs.io/en/stable/

The deployment of Argo CD Image Updater is a bit cumbersome, the deployment operations are as follows:

1. Create a local user in Argo CD

To create an Argo CD image update program, you need to access the Argo CD API Server credentials, use an image-updater account with appropriate API permissions, and add the following user definitions to argocd-cm:

# kubectl -n Argo CD edit cm argocd-cm
data:
  accounts.image-updater: apiKey

Create an access token for the user, copy the value of the token somewhere, and you will need it later.

Argo CD account generate-token --account image-updater --id image-updater

2. Grant RBAC permissions in Argo CD

Configure the appropriate RBAC permissions for the image-updater user. Argo CD Image Updater requires update and get permissions for the application.

# kubectl -n Argo CD edit cm argocd-rbac-cm
data:
  policy.default: role:readonly
  policy.csv: |
    p, role:image-updater, applications, get, */*, allow
    p, role:image-updater, applications, update, */*, allow
    g, image-updater, role:image-updater

3. Install Argo CD Image Updater

yaml file download: https://github.com/argoproj-labs/argocd-image-updater/tree/master/manifests

kubectl create ns argocd-image-updater
kubectl apply -n argocd-image-updater -f manifests/install.yaml

4. Configure the mirror warehouse

Even if you do not plan to use a private mirror warehouse, you need to configure at least one empty registries.conf :

# kubectl -n argocd-image-updater edit cm argocd-image-updater-config
data:
  registries.conf: ""

Without this entry argocd-image-updater Pod will not start.

If you use a private mirror warehouse, you can refer to the following configuration, take Harbor mirror warehouse as an example:

data:
  Argo CD.insecure: "true"
  log.level: debug
  registries.conf: |
    registries:
    - name: harbor
      api_url: http://10.39.140.196:8081
      prefix: 10.39.140.196:8081
      ping: yes
      insecure: yes

5. Configure API access token key

When installing from the list to Kubernetes cluster, Argo CD Image Updater from the named Argo CD_TOKEN read environment variables required to access Argo CD API token, the environment variable is named from Argo CD.token set of secret field in argocd-image-updater-secret .

Argo CD.token should be set to the base64 encoded value of the access token you generated above. As a shortcut, you can use kubectl generate a key and apply it to an existing resource:

YOUR_TOKEN=xxx
kubectl create secret generic argocd-image-updater-secret \
  --from-literal Argo CD.token=$YOUR_TOKEN --dry-run -o yaml |
  kubectl -n argocd-image-updater apply -f -

After the change, the argocd-image-updater Pod must be restarted, that is, run

kubectl -n argocd-image-updater rollout restart deployment argocd-image-updater

Create a new yaml warehouse Kustomize file

Since the image updater only supports helm or Kustomize type yaml, here is a new yaml directory based on Kustomize, and the parameters in the yaml should not conflict with the previous ones:

[root@jenkins git]# tree argocd-gitops/kustomize-javademo/
argocd-gitops/kustomize-javademo/
├── javademo-deployment.yaml
├── javademo-svc.yaml
└── kustomization.yaml

javademo-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: javademo-tag
spec:
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: javademo-tag
  template:
    metadata:
      labels:
        app: javademo-tag
    spec:
      containers:
      - image: 10.39.140.196:8081/apps/javademo:replace
        name: javademo-tag
        ports:
        - containerPort: 8080

javademo-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: javademo-tag
spec:
  ports:
  - port: 8012
    targetPort: 8080
  selector:
    app: javademo-tag

kustomization.yaml

amePrefix: kustomize-

resources:
- javademo-deployment.yaml
- javademo-svc.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

Log in to the Argo CD UI to create a new Argo CD application. Compared with the previous one, the annotations parameter is added, the mirror address to be monitored is specified, the update strategy is latest, and the source path is modified:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: javademo=10.39.140.196:8081/apps/javademo
    argocd-image-updater.argoproj.io/javademo.update-strategy: latest
  name: javademo-tag
  namespace: Argo CD
  finalizers:
    - resources-finalizer.Argo CD.argoproj.io
spec:
  destination:
    namespace: apps
    server: https://kubernetes.default.svc
  project: default
  source:
    path: kustomize-javademo
    repoURL: http://10.39.140.196:10080/gogs/argocd-gitops.git
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
    - Validate=false
    - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Log in to the KubeSphere UI to re-create a CI pipeline and delete the update docker tag step. It is no longer necessary to trigger application deployment based on git push:

pipeline {

    environment {
        GIT_URL='http://10.39.140.196:10080/gogs/spring-demo.git'
        GIT_CREDENTIAL_ID = 'git-id'
        GIT_BRANCH = 'master'
        REGISTRY = '10.39.140.196:8081/apps/javademo'
        REGISTRY_CREDENTIAL_ID = 'harbor-id'
    }

    agent {
        node {
            label 'maven'
        }
    }

    stages {

        stage('SCM Checkout') {
            steps {
                git branch: "${GIT_BRANCH}", credentialsId: "${GIT_CREDENTIAL_ID}", url: "${GIT_URL}"
            }
        }

        stage('source build') {
            steps {
                container('maven') {
                    sh 'mvn clean package'

                }
            }
        }

        stage('docker build & push') {
            steps {
                script {
                    env.COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
                    env.TIMESTRAP = sh(returnStdout: true, script: 'date +%Y%m%d%H%M%S').trim()
                    env.DOCKER_TAG = "dev_${TIMESTRAP}_${COMMIT_ID}_${BUILD_NUMBER}"
                }
                container('maven') {
                    withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$REGISTRY_CREDENTIAL_ID" ,)]) {
                        sh 'docker build -t $REGISTRY:$DOCKER_TAG .'
                        sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
                        sh 'docker push $REGISTRY:$DOCKER_TAG'
                    }
                }
            }
        }
    }
}

Check the pipeline log, the image was successfully pushed to the Harbor warehouse:

Harbor warehouse mirror tag update, Argo CD image updater will automatically update the latest tag to K8s cluster.

View the mirror tag:

In the future, every time the Harbor warehouse generates the latest image, Argo CD will automatically update it to the K8s cluster.

This article is published by the blog one article multi-posting OpenWrite