Private cloud environment installed in a notebook: continuous integration (part 1)

This article is the fifth article in the series. Let's continue to talk about how to deploy a simplified private cloud environment in a notebook to meet the low-cost, low-power, and low-latency experimental environment. If you have an idle lightweight cloud server, you can also try it.

Write in front

As the first content of the "Continuous Integration" chapter, let's first talk about the use of CI on a stand-alone server.

Regarding the basic construction, it has been mentioned many times in the previous article, so I will not repeat it. This article will focus on some details of the process. If you are interested in Gitea and Drone or GitLab, you can read the previous content:

• "Lightweight Warehouse and CI Usage Plan in Container Mode: Gitea + Drone Basics"
• "Using Containers to Compile Drone CI Without Functional Limitations"
• "Lightweight and safe deployment plan"
• "Use Docker and Traefik v2 to build a lightweight code warehouse (Gitea)"
• Some GitLab related content

In order to lower maintenance costs and expand the use of multiple machines in the future, all programs in this article are used in a container environment.

Stand-alone CI design

Before embarking on the practical details, we have to talk about "design" first.

Architecture design

Participants in the CI process mainly include the following categories (the software warehouse part will not be discussed in this article): users, Git services, CI services, and CI executors.

Simply define the above participants: "Users" can be flesh and blood, automated scripts or BOT, creators of various data; "Git Service", used to store code data and provide basic permissions functions And interface management procedures; "CI Service", a program that provides continuous integration task scheduling and management; "CI executor", a program used to perform specific CI tasks.

Considering that in addition to the Git service and CI service, the stand-alone server will also run the programs we need to update and deploy. In order to make the use of resources more efficient, lower the maintenance cost, and prevent us from configuring HTTPS certificates for each Web program, we You can add an application gateway that supports service discovery.

Even if it is a stand-alone server, we still need to pay attention to the safety of SSH. In a multi-machine environment, we will use springboard and cloud server security policies for centralized security management. In a stand-alone scenario, I use the SSH service switch to complete the simple Security protection (when not in use, turn it off directly, which also saves some power for the sniffing robot on the Internet).

If the above "participants" are represented by a legend, a basic stand-alone CI usage model will be similar to the following:

I will label the digital serial numbers of the data interaction of different roles in the figure, and briefly explain the specific content of these serial numbers:

• "1" means that the user uses a specific domain name to access our Git service and CI service to perform warehouse management or configure CI tasks. This type of interaction uses HTTP, such as accessing https://gitea.lab.com , https://gitlab.lab.com , https://drone.lab.com in a browser.
• "2" means that the user or client uses SSH to access the Git repository, which needs to be used with RSA Key.
• "3" and "4" indicate the way that Traefik uses service discovery to aggregate Git services and CI services to provide users with access in the form of domain names. The proxy mode used here is also HTTP.
• "5" indicates the data interaction between the SSH switch and the Git SSH service, and the interaction form is TCP.
• "6" and "7" represent the data interaction between the CI service and the Git service and the CI executor, respectively. Obtain warehouse changes from Git, then create a CI task, and then continuously push the execution status of the CI task to the Git service, interacting The form is not limited. You can use HTTP API or various TCP-based RPC methods.
• "8" indicates how the CI executor obtains code from the code repository of the Git server, or updates some data back to the Git server. Normally, it uses HTTP. I recommend using Git Over SSH for interaction.

Deployment mode

In the stand-alone full container mode, we generally use two methods to complete the deployment.

One is based on file mounting methods, such as opening the file system in the CI JOB container with the host during the CI process, and then synchronizing the build product to the host, similar variants and using various network file protocols Mount the file system; the other type uses SSH, SCP, Rsync and other methods to access the host in the container to complete data exchange or service initialization or start and stop operations.

In addition, if we use software warehouses and container warehouses, we can also complete pure container delivery, making the interaction more pure and "clean". We will expand on this topic in a follow-up article.

Single-machine CI configuration practice

Next, let's talk about how to use CI in stand-alone mode using the application of "SSH switch" in the above article, using continuous integration and deployment practices in Gitea and Drone environments as an example.

Because this project type is a network program that does not support hot loading and needs to run continuously, the update of the program needs to restart the service. So we can just use the mount file method in the "deployment mode" to update the file, and use the SSH method to stop and restart the service. (If it is a static resource type project deployment, you only need to complete the resource replacement update)

Define CI profile

First, place and upload the projects that need to integrate CI to a warehouse in Gitea. Here is an example of the Git SSH switch mentioned in the above article. Create a CI configuration file .drone.yml in the project.

A relatively common CI configuration can be expressed in the following form:

---
kind: pipeline
name: default

steps:

- name: clone

- name: stop-previous-services
  depends_on: [ clone ]

- name: update-services
  depends_on: [ stop-previous-services ]

- name: start-new-services
  depends_on: [ update-services ]

The above configuration includes four processes: downloading the warehouse code, stopping the original service, updating the service program code, and restarting the service. In actual production, depending on the type of business, our execution order may change, and it may not even be executed in the above "serial" way.

After configuring CI according to the above configuration, when we push the code to the code warehouse to trigger the CI task. In the graphical interface, we will see a result similar to the image above.

Download code using SSH protocol

No matter which CI tool you use, I recommend that you use Git Over SSH to get the code instead of using Git Token or account password to interact. This can make your program less dependent on a certain CI or Git repository, and it is easier to switch to a more suitable tool at the right time and at low cost.

In Drone CI, if you want to use SSH to download code, you can use the following configuration: (same as in GitLab Runner)

---
kind: pipeline
name: default

clone:
  disable: true

steps:

- name: clone
  image: alpine/git
  pull: if-not-exists
  environment:
    KEY:
      from_secret: ssh_key
  commands:
    - GIT_HOST=$(echo $DRONE_GIT_SSH_URL | sed 's/git@/\1/' | sed 's/:.*/\1/') && mkdir "$HOME/.ssh" && echo "$KEY" > "$HOME/.ssh/id_rsa" && chmod 600 $HOME/.ssh/id_rsa && eval `ssh-agent -s` && ssh-add $HOME/.ssh/id_rsa && ssh-keyscan $GIT_HOST > ~/.ssh/known_hosts && chmod 400 "$HOME/.ssh/known_hosts";
    - git clone $DRONE_GIT_SSH_URL .
    - git -c advice.detachedHead=false checkout $DRONE_COMMIT

In the above code, in order to download the program code using SSH, the CI program will do two things:

1. Read our pre-configured ssh_key environment variables from the CI software, and then output the variables as rsa_key that can be used directly by the program, set the permissions, and use ssh-agent load the program.
2. The default warehouse HTTP protocol replaced Git protocol used to prepare the program.

Of course, if you want to download the code using SSH, we need to configure SSH Key in the account or warehouse of the Git software.

Use SSH to operate service start and stop

In this application, we docker-compose.yml , so the startup and shutdown of the service can be done with the familiar commands docker-compose up -d and docker-compose down .

Because CI is executed in the container, we cannot directly manipulate the host, so we need to use SSH or dind mode docker.sock to complete the service status change.

This article first talk about how to use SSH to solve basic deployment operations:

- name: stop-or-start-services
  image: deploy-tool
  depends_on: [ clone ]
  pull: if-not-exists
  environment:
    KEY:
      from_secret: ssh_key
    # 环境变量，除了私密的定义在 CI 软件的环境变量中，也可显式声明在 CI 配置中
    TARGET_HOST: user@host
    TARGET_PORT: 22
  commands:
    - mkdir "$HOME/.ssh" && echo "$KEY" > "$HOME/.ssh/id_rsa" && chmod 600 $HOME/.ssh/id_rsa && eval `ssh-agent -s` && ssh-add $HOME/.ssh/id_rsa";
    # 关闭服务
    - ssh -i "$HOME/.ssh/id_rsa" -p $TARGET_PORT -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $TARGET_HOST "bash -c \"cd /app-path/ && docker-compose down\""
    # 启动服务
    - ssh -i "$HOME/.ssh/id_rsa" -p $TARGET_PORT -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $TARGET_HOST "bash -c \"cd /app-path/ && docker-compose up -d\""

Similar to the download code, we initialize rsa key from environment variables, and then load the private key ssh-agent Then use the ssh client to connect to the host, switch the working directory, and execute the command to start and close the service.

Similarly, if we want to use SSH to operate the server, we need to configure the corresponding public key ~/.ssh/authorized_keys

Update the code using file mounting

There are two ways to update the code. One is to use the SSH method mentioned above to remotely execute scp and rsync to synchronize data, and the other is to use file mounting. Because our deployment is on the same machine, file mounting can be an efficient way.

Take Drone CI configuration as an example to demonstrate how to mount the host directory into the container:

- name: update-services
  image: deploy-tool
  depends_on: [ stop-previous-services ]
  pull: if-not-exists
  commands:
    - rm -rf /deploy/*
    - cp -r /drone/src/* /deploy/
    - cp -r /drone/src/.env /deploy/
  volumes:
    - name: host-dir
      path: /deploy

volumes:
  - name: host-dir
    host:
      path: /app-path

At last

In the next "Continuous Integration" related articles, I will talk about the use of CI in multi-machine and relatively complex scenarios, as well as the actual deployment details of other scenarios.

--EOF

---

We have a small tossing group, which gathers hundreds of friends who like to toss.

Without advertising, we will talk about software and hardware, HomeLab, programming issues, and share some technical salon information in the group from time to time.

Friends who like toss are welcome to scan the code to add friends. (Add a friend, please note the real name, indicate the source and purpose, otherwise it will not pass the review)

Those things about tossing into groups

---

If you think the content is practical, you are welcome to like it and share it with your friends. Thank you.

---

This article uses the "Attribution 4.0 International (CC BY 4.0)" license agreement, welcome to reprint, or re-modify the use, but need to indicate the source. Signature 4.0 International (CC BY 4.0)

Author of this article: Su Yang

Created: January 02, 2022
Count of words: 5438 words
Reading time: 11 minutes to read
Link to this article: https://soulteary.com/2022/01/02/private-cloud-environment-installed-in-a-notebook-cicd-part-1.html