Install jenkins in k8s and configure CI/CD

We misunderstood the world and said it deceived us. —— Tagore, "Asuka"

I. Overview

Before learning to use a tool, we need to know how to install it. This article records my own learning process. On the one hand, it consolidates the content of learning, and on the other hand, I hope to provide some help to small partners who have the same needs.

Open source tools	describe	Official document	Official installation document	docker installation
jenkins	devops continuous integration tool	jenkins official website	jenkins quick installation	docker install

The above table lists the official installation addresses. If you need a quick experience and use, it is recommended to install directly using docker, and you can start the application with one line of command:

docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts-jdk11

This article introduces the installation and use of jenkins in k8s, mainly through two ways of installation practice:

• write your own yaml file install
• use helm to install

Installation Environment

Here, minikube is used for installation, which is basically the same in k8s cluster

• minikube : v1.18.1
• helm : v3.5.3

2. Customize yaml file to install jenkins

Because jenkins need persistent data, so we need to create PVC , recommended storageClass dynamically created PVC , in minikube there is a default in storageClass , the name is: standard , you can use the following command to see:

# kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  50m

the use of storageClass 161121cf5dabd1, please check the official website: https://kubernetes.io/zh/docs/concepts/storage/storage-classes/

Create the jenkins-deploy.yaml file, the content of the file is as follows:

###############使用 storageClass 创建 pvc ###################
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-data-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  # 指定 storageClass 的名字，这里使用 minikube 默认的 standard
  storageClassName: "standard"
  resources:
    requests:
      storage: 10Gi

###############创建一个ServiceAccount 名称为：jenkins-admin###################
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: default
  labels:
    name: jenkins

###############绑定账户jenkins-admin 为集群管理员角色，为了控制权限建议绑定自定义角色###################
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins-admin
  labels:
    name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins-admin
    namespace: default
roleRef:
  kind: ClusterRole
  # cluster-admin 是 k8s 集群中默认的管理员角色
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io


############### 在 default 命名空间创建 deployment ###################
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      # 注意：k8s 1.21.x 中 serviceAccount 改名为 serviceAccountName
      # 这里填写上面创建的 serviceAccount 的 name
      serviceAccount: jenkins-admin
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts-jdk11
          imagePullPolicy: IfNotPresent
          env:
            - name: JAVA_OPTS
              value: -Duser.timezone=Asia/Shanghai
          ports:
            - containerPort: 8080
              name: web
              protocol: TCP
            - containerPort: 50000
              name: agent
              protocol: TCP
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          volumeMounts:
            - name: jenkinshome
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkinshome
          persistentVolumeClaim:
            claimName: jenkins-data-pvc

############### 在 default 命名空间创建 service ###################
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: default
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: ClusterIP
  ports:
    - name: web
      port: 8080
      targetPort: 8080


---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-agent
  namespace: default
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: ClusterIP
  ports:
    - name: agent
      port: 50000
      targetPort: 50000

Use the following command to deploy jenkins:

# kubectl apply -f jenkins-deploy.yaml

persistentvolumeclaim/jenkins-data-pvc created
serviceaccount/jenkins-admin created
clusterrolebinding.rbac.authorization.k8s.io/jenkins-admin created
deployment.apps/jenkins created
service/jenkins created
service/jenkins-agent created

Use the following command to temporarily expose the service port:

kubectl port-forward service/jenkins 8080:8080 -n default

In the production environment, it is recommended to use ingress to expose the service through the domain name

After exposing the service port, you can visit: http://localhost:8080

Use the following command to view the administrator password:

# kubectl get pod -n default
NAME                       READY   STATUS    RESTARTS   AGE
jenkins-68666b56fc-p8fvd   1/1     Running   0          8m28s

# kubectl exec jenkins-68666b56fc-p8fvd -- cat /var/jenkins_home/secrets/initialAdminPassword
b06be4420bcd4a02ab4968ab02838986

After successful login, you need to install the plug-in:

The reason why the recommended plug-ins are not installed here is that we do not need many plug-ins, and the default download plug-ins will be downloaded from abroad, which is relatively slow. The configuration of the domestic download address will be introduced later.

After clicking install, create the first administrator user:

According to the actual configuration, choose to use the admin account to continue.

3. Use helm to install jenkins

You can go to the helm official package management warehouse to find the applications that need to be installed.

helm package management address: https://artifacthub.io/

Search for jenkins in Artifact hub, as shown below:

Install according to the instructions of jenkins, and the installation steps are described in detail below:

Use the following command to add the repository where jenkins is installed

$  helm repo add jenkins https://charts.jenkins.io
"jenkins" has been added to your repositories

Use the following command to view the added helm warehouse:

$  helm repo list
NAME            URL
kong            https://charts.konghq.com
aliyun          https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
stable          https://charts.helm.sh/stable
kubeview        https://benc-uk.github.io/kubeview/charts
tscharts        https://technosophos.github.com/tscharts
bitnami         https://charts.bitnami.com/bitnami
apisix          https://charts.apiseven.com
jenkins         https://charts.jenkins.io

Use the following command to update the helm repository:

$  helm repo update

Use the following command to search for jenkins in the warehouse:

$  helm search repo jenkins

aliyun/jenkins  0.13.5          2.73            Open source continuous integration server. It s...
bitnami/jenkins 8.0.8           2.289.3         The leading open source automation server
jenkins/jenkins 3.5.9           2.289.3         Jenkins - Build great things at any scale! The ...
stable/jenkins  2.5.4           lts             DEPRECATED - Open source continuous integration...

Use the following command to view what can be configured:

$  helm show values jenkins/jenkins

Use the following command to download the chart package of helm to the local:

$  helm pull jenkins/jenkins

The downloaded package is a compressed package, which can be tar -zxvf command:

$  tar -zxvf jenkins-3.5.9.tgz

-rw-r--r-- 1  1049089 45006 Jul 28 23:36 CHANGELOG.md
-rw-r--r-- 1  1049089  1287 Jul 28 23:36 Chart.yaml
-rw-r--r-- 1  1049089 30809 Jul 28 23:36 README.md
-rw-r--r-- 1  1049089 37647 Jul 28 23:36 VALUES_SUMMARY.md
drwxr-xr-x 1  1049089     0 Aug  5 17:59 templates
-rw-r--r-- 1  1049089 36203 Jul 28 23:36 values.yaml

Modify the values.yaml file as needed to customize the configuration. For quick experience, no other configuration is required, just use the following command to install:

$  helm install jenkins ./jenkins

Use the following command to view the login user name and password:

# 查看登录的用户名
$  kubectl exec jenkins-0 -- cat /run/secrets/chart-admin-username
# 查看登录的密码
$  kubectl exec jenkins-0 -- cat /run/secrets/chart-admin-password

Use the following command to temporarily expose the service:

kubectl --namespace default port-forward svc/jenkins 8080:8080

In the production environment, it is recommended to use ingress to expose the service through the domain name

After exposing the service port, you can visit: http://localhost:8080, as shown in the figure below:

4. Configure jenkins to implement devops

Jenkins has been installed according to the previous steps. Next, I will introduce how to configure jenkins to implement devops.

1. Configure the plug-in update site as a domestic source

The default plug-in update site is: https://updates.jenkins.io/update-center.json

Modify the plug-in update site as: https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json

As shown below:

2. Download common plug-ins

Download commonly used plug-ins as shown in the figure below:

3. Configure kubernetes cluster

Click [System Management] —> [Node Management] —> [Configure Clouds]

Do not do any configuration here, just click [Connection Test] and you can find that the connection to k8s is successful, as shown in the following figure:

Configure k8s cluster related, as shown below:

Configure the pod template, as shown below:

Add the first container to the pod template: jenkins/inbound-agent as the slave node of jenkins, as shown in the following figure:

Add a second container to the pod template: docker , which is used to build and push the image, as shown in the following figure:

Add a third container to the pod template: maven:3.8.1-openjdk-11 , as shown in the following figure:

Add more containers as needed. For example, if you need to build a front-end project, you can add a node:16.6.1-slim container, as shown in the following figure:

4. Configure caching and mount settings.xml for the maven container

If you need to mount a custom Settings file for the Jenkins Slave Pod, you can create a Config Map Volume first, and then configure it on the Pod Template.

Execute the following command to create a custom Settings file:

kubectl -n default create configmap maven-config --from-file=settings.xml 

In volume click add volumes , select Config the Map Volume type of volume, and the other is disposed maven, persistent dependence to improve the construction of the velocity, the configuration as shown below:

5. Use docker to build and push container images

When using docker push the image, you need to set the access permissions of the image warehouse, and set it in the following way:

Execute the following command to log in to the mirror warehouse, and the config.json file will be generated when logging in to the mirror warehouse:

docker login -u <username> -p <password> registry.cn-hangzhou.aliyuncs.com

Use the generated config.json file to create a secret named my-secret under the jenkins

kubectl create secret generic jenkins-docker-cfg -n default --from-file=/root/.docker/config.json

Configure the mounted volume and environment variables in the Pod Template of the Jenkins system:

6. Configure kubeconfig to access k8s

Configure kubeconfig to access k8s, which will be used when kubernetesDeploy

• [Manage Credentials] -> [jenkins] -> [Global Credentials] -> [Add Credentials]

7. Set the user name and password for pulling the private warehouse image

• [Manage Credentials] -> [jenkins] -> [Global Credentials] -> [Add Credentials]

8. Test the devops build process

Create a pipeline project, as shown in the following figure:

Enter the following in the pipeline script:

pipeline {
     // 定义本次构建使用哪个标签的构建环境
    agent{
        node{
          label 'slave-pipeline'
        }
      }

    stages {
        stage('Hello') {
            steps {
                echo 'Hello World'
            }
        }
        // 拉取代码
        stage('git clone') {
            steps {
                git branch: "master", credentialsId: "", url: "https://gitee.com/peterwd/devops-demo.git"
            }
        }
        //  运行源码打包命令
        stage('Package'){
          steps{
              container("maven") {
                  sh "mvn --version"
                  sh "mvn clean package -DskipTests"
              }
          }
        }

        // 运行容器镜像构建和推送命令
        stage('Image Build And Publish'){
          steps{
              container('docker') {
                   sh 'docker version'
              }
          }
        }
        
        //  运行 node 构建命令
        stage('node'){
          steps{
              container('node') {
                   sh 'npm version'
              }
          }
        }
    }
}

As shown below:

Click Save, and build, as shown below: