Design and implement a simple automated deployment tool

Before I say it, put the address https://cycle.bucai.cc/ , the development environment is running, it is a bit slow, please wait patiently for the loading.

GitHub: https://github.com/notbucai/cb-cycle

This is a record of my attempt to use a process-based approach to project development and implementation.

As for why the existing and perfect framework on the market is not applicable, it is because I haven't written a complete and meaningful thing for a long time, and of course I will still "learn" from some mature projects.

design

source of demand

Recently, I have been intermittently proud to upgrade and optimize the blog function points. The project is divided into the backend nestjs project, the frontend nuxtjs, and the management system vue.js. The current deployment method uses Docker to compile the local image, push it to the private warehouse, and then enter the server Restart after pull. There is also multi-person collaborative development. Everyone needs to install the server environment (Docker, etc.) locally to ensure environment consistency.

It is not difficult to find from the above description:

1. The compilation methods cannot be unified due to differences in the templates, languages, and states (static, services) used by the project after deployment.
2. Repetitive issues with the current manual deployment process.
3. The server environment needs to be matched, and the local development computer configuration requirements are high.

target expectations

1. Project deployment templates, and different templates can be selected for packaging according to different types.
2. Transition manual deployments to automated deployments.
3. Depends on branch monitoring, push automatically triggers deployment, and solves high local configuration requirements.

research

Because most people need this tool in their daily work, the survey is simply explained.

Research objectives: Netlify (foreign), Tencent Cloud Webify (domestic).

what problems do they solve

1. Deploy the resource.
2. CI/CD workflow.

their disadvantages ( Forcibly increase the height of the current project )

1. Netlify abroad, dddd.
2. Webify cloud manufacturers conduct operation and maintenance, and start to cut leeks when they are not guaranteed.

demand analysis

Through the above content, it is very intuitive to obtain the required function points of the project.

core analysis

The core function is to operate "task" as a unit, create tasks and manage tasks.

The so-called "task" means that the state needs to be transferred. At present, tasks are divided into in-progress, failed, and stopped.

Each trigger should generate a subtask, and if there is an ongoing subtask at the time of triggering, it should stop and start a new task.

In-progress tasks contain sub-states: Compiling, Compilation Succeeded, Compilation Failed.

Please refer to the flow chart below for user interaction and core links.

feature design

1. Create a task

   It mainly registers the webhook for a branch of a warehouse, and writes the task information to the database.

2. Administrative tasks

   It mainly manages the created task information, deletes, pauses, and starts.

   1. Start

      Resume the suspended task.

   2. pause

      on an ongoing task.

   3. delete

      Delete this piece of data and delete the webhook.

3. task execution

   Trigger execution through webhook, find the current data for execution.

task execution design

How to compile/run differently with different configurations?

All compilations are compiled using docker in isolation, and different ways of running are selected according to different types.

There are currently two types of distinction, one is static and the other is dynamic:

• Static: The method of uploading cdn after compiling with docker is used, of course, it is only simulated and temporarily placed in the nginx static directory.
• Dynamic: Docker runs after compiling with docker.

web	service
No port required	port
nginx forwards resources	nginx forwarding port
may need to be compiled	may need to be compiled
Does not require a container to run	Requires container to run

action	spa project	normal web page	Ts node project	normal node project
Construct	Yes	no	Yes	Yes
compile	Yes	no	Yes	no
run	no	no	Yes	Yes

The above compares the resource type and the project type, and intuitively expresses the execution action of the task.

After the static resource (web) container is built, the dist data in the image is copied to the host through docker cp, and then transferred to nginx.

After the service resource (service) container is built and compiled, it is run, and then nginx is added to map the container to the machine port.

The ports are maintained later. The ports in the container are not managed, and the nginx container is used for mapping and maintenance.

The user's own task containers can communicate with each other and can be accessed through the task id. If it is a static resource, it needs to be accessed and then the interface will be forwarded through configuration. At the same time, the user can bind his own domain name to the current machine.

page design

The front end adopts naive-ui as the front-end layout style component.

Its interaction is mainly divided into three steps:

1. Choose a hosting platform

   For platforms such as github and gitee, the selected hosting platform must be authorized to log in.

2. Enter information

   Enter the data fields necessary for the task

3. Confirm information

   Confirm whether the input meets the requirements

Because the design is relatively simple, it is directly displayed in the form of pictures.

Task list fields: task name/ID, hosting platform, template, status, creator, creation time, update time, operation time.

Details page

Database Design

Only show the design of the core table

user table

Hosting Platform Table

Template table

task list

subtask table

interface design

Including crud, directly view the interface documentation:

https://www.apifox.cn/apidoc/project-741496/api-14138505

Technical selection

The criteria for the chosen scheme are 我会 , 简单 .

Currently choose egg + mysql + redis + vue architecture, and the encoding should be as ts as possible. Other frameworks have also been considered, such as nestjs and koa. Relatively speaking, the nest is too big, and the koa is too small. I just want to try the 3.x version when choosing vue.

eggjs : Built for enterprise frameworks and applications.

vue : I know everything.

Project Architecture

The architecture is very simple, that is, storage cooperates with dependencies to trigger.

Server operation and maintenance architecture

For independent current applications, users can share their own services and run them independently (isolation). In principle, each user can apply for some storage and other resources (to be determined), but it is still recommended to use remote/third-party resources.

Compile/deploy trigger process

Whether to compile: Some node projects do not require compilation steps

Whether to run: static resources do not need to run, just type directly into nginx

accomplish

Core function implementation (for the time being, only github-related business is implemented, and other platforms reserve interfaces).

Project progress view

Bind hosting platform

Quoting Ruan Yifeng's picture and article , the binding process is basically a browser redirection, then return to the code, get the code to exchange the token, and then save the token.

Get relevant information

Do it through the following api

User repository list

get branch list

Get the contents of the specified file

Create a task

Create a task to write to the webhook.

Create webhook

task execution

wait for webhook callback

https://docs.github.com/cn/developers/webhooks-and-events/webhooks/testing-webhooks

The template related to task execution is executed every time it is triggered, and ejs is used for configuration rendering and generation.

 // 注入内容
{
  user, // 数据库user model
  task, // 数据库task model
  taskChild, // 数据库task_child model
  template, // 数据库template
  path: {
    ...path, // 相关路径
    code: relativeCodePath // 代码与配置的相对位置
  }
}

 # dock er file
FROM node:12.18.2

LABEL maintainer="<%= user.id %><<%= user.email %>>"

ADD <%= path.code %> /app/

WORKDIR /app

RUN rm -rf node_modules

RUN rm -rf <%= path.build %>

RUN npm config set sharp_binary_host https://npm.taobao.org/mirrors/sharp

RUN npm config set sharp_libvips_binary_host https://npm.taobao.org/mirrors/sharp-libvips

RUN npm config set sass_binary_site https://npm.taobao.org/mirrors/node-sass

RUN npm install --registry https://registry.npm.taobao.org --max-old-space-size=4096

ENV NODE_ENV production

<% if (template.is_build) { %>
RUN npm run <%= task.build_script %>
<% } %>

<% if (template.is_run) { %>
EXPOSE <%= task.server_port %>

CMD  nohup sh -c 'npm run <%= task.run_script %>'
<% } %>

 # docker compose config
version: "3.7"

services:
  "<%= task.id %>":
    build: .
    image: <%= task.id %>:<%= taskChild.version || '0.0.1' %>
    container_name: <%= task.id %>
    restart: always
<% if (template.is_run) { %>
    networks:
      <%= user.id %>-network:
        ipv4_address: <%= task.ip %>

networks:
  <%= user.id %>-network:
    external: true
<% } %>

build/compile/run related

node child_process exec execute the command:

Use exec('docker-compose build') to build the image.

Use exec(docker cp $(docker create --rm ${imageName}):/app/${buildPath} ./dist) to copy the data in the container to the host.

Use docker-compose down to stop the existing mirror image.

Use 'docker-compose restart to restart the image container.

Use docker-compose up to start the compiled image.