How to build a build and deployment platform suitable for your team

This is the 108th original without water. If you want to get more original good articles, please search the public and follow us~ This article first appeared in the front-end blog of Zheng Caiyun: 160ffb6055af73 How to build a construction and deployment platform suitable for your team

The existing build and deployment schemes in the front-end industry should be commonly used, such as Jenkins, Docker, GitHub Actions, and it just so happens that our company now coexists the first two schemes. Now that we have a stable way to build and deploy, why do we have to do it ourselves? Do you build your own front-end platform? Of course it is not for fun, the reason is to let me analyze it slowly.

Various problems may be encountered when building and using the front-end, such as:

• Eslint skip verification-the front-end projects in the company, over time, at different stages, projects created through new and old scaffolding may have different styles, and the verification rules may not be uniform, although the project itself may have Various Eslint, Stylelint and other verification interception, but can not prevent developers from skipping these code verification.
• Npm version upgrade is not compatible-some compatibility checks are required for the dependent npm version. If some npm plugins suddenly upgrade some incompatible versions, an error will be reported after the code is online, typically all types of IE compatibility.
• I can’t add the functions I want freely-I want to optimize the process of front-end construction, or optimize the functions that are convenient for front-end use, but because I rely on the operation and maintenance platform to build applications, I want to add my own functions and need to wait for others to schedule.

And these problems, if you have your own build platform, this will not be a problem, so there will be the current-Yunchang.

Why is it called "Yunchang"? Of course, I hope that this platform can be like "Guan Yunchang". So what kind of capabilities can Yun Chang provide us?

Cloud growth ability

Build and deploy

This is of course a necessary basic ability. Yunchang provides the company's different types of front-end projects, such as Pampas, React, Vue, Uniapp, and so on. The whole process is actually not complicated. After the start of the construction, the cloud leader server obtains the project name, branch, deployment environment and other information to be built, and then starts the code update of the project, depends on the installation, and then the code is packaged. Finally, the generated code is packaged into a mirror file, and then the mirror is uploaded to the mirror warehouse, and some resource static files of the project can be uploaded to the CDN to facilitate the call after the front-end, and finally call the K8S mirror deployment service to perform The image is deployed according to the environment, and an online construction and deployment process is completed.

Pluggable build process

If you are using someone else's build platform, many front-end script functions you want to add will rely on other people's services to achieve. If you go to the cloud, you can provide an open interface so that the front-end can freely customize its own plug-in services.

For example, in the process of online construction and packaging, some of the problems and pain points mentioned above can be dealt with, such as:

• All kinds of Eslint, Tslint and other compliance verification of the code, no longer afraid of being skipped by the verification step.
• Before the project is built, you can also check the npm package version to prevent compatibility errors after the code is online.
• After the code is packaged, it can also do some global front-end resource injection, such as burying points, error monitoring, message push and other types.

Review release process

The company’s existing platform release process management and control relies on the maintenance of the operation and maintenance list. Each project will manage a list of publishable persons. Therefore, the release of the basic project requires the publisher to follow the release that night, and Yunchang aims to solve this problem. , Provides a concept of audit flow.

That is, after the project is tested in the pre-release environment, the code developer can file a real release request form, and then the releaseable person of the project will receive a request form that needs to be reviewed through DingTalk, which can be accessed through the web page. Or Ding the message directly to operate, agree or reject the release application. After the application is approved, the code developer can deploy the project release line by himself after the release time. After the release line, the follow-up will be created for the project A code Merge Request request to facilitate the filing and sorting of subsequent codes.

The advantage of this is that on the one hand, the front-end can control the permissions of project construction and release, so that the publishing permissions can be closed, and on the other hand, it can also liberate the project publisher and make it more convenient for developers to go online. The release of the project was opened again.

Ability to export

Yunchang can export some of the ability to build and update externally, which makes it possible for third-party plug-ins to access the build process. We intimately provide developers with the VsCode plug-in, allowing you to freely update the code during the development process. The time to open the web page for construction, stay at home, and update the code in the editor. The common environment also provides a shortcut for one-click update, which further saves the operation time in the middle. At this time, write two more lines Isn't the code happier?

Our VsCode plug-in not only provides some of Yunchang's building capabilities, but also small program construction, routing lookup, and other functions. If you look forward to sharing this plug-in, please look forward to our follow-up articles.

Cloud Chang Architecture

I mentioned the construction process of Yunchang. Yunchang relies on the ability to deploy images provided by K8S. Yunchang's client and server are all services running in Docker, so Yunchang adopts Docker In Docker. The design plan is to package a Docker image by the service in Docker.

For the construction of the code, the server part of the cloud leader introduces the processing of the process pool. Each project built in the cloud leader is an independent instance in the process pool and has an independent packaging process, and the progress of the packaging process follows The advancement is carried out by Redis's timing task query, which realizes the architecture of Yunchang's multi-instance parallel construction.

The interface communication between Yunchang client and server is normal HTTP request and Websocket request. After the client initiates the request, the server uses MySQL data to store some applications, users, construction information and other data.

The external resource interaction is that during the construction process, some static resources and packaged images will be uploaded to the cdn and mirror warehouse, and finally the deployment interface of K8S will be called to deploy the project.

0-1 front-end construction

I have seen some function introductions of "Yunchang" and the architecture design of "Yunchang" above. I believe many friends also want to build a front-end construction and release platform similar to "Yunchang". What do I need to do? Let's take a look at the design ideas of the main modules of the front-end construction platform.

Build process

The main core module of the front-end construction platform must be build and package. The build and deployment process can be divided into the following steps:

• After each build starts, you need to save some information and data of this build, so you need to create a build release record, which will store the release information of this release, such as the name of the release project, branch, commitId, commit information, and operator data , The release environment that needs to be updated, etc. At this time, we will need a build release record table, and if you need some data of the project and the operator, you need the application table and the user table to store the relevant data for association.

• After the build release record is created, the front-end build process starts. The build process can pipeline . The process can refer to the following example

  // 构建的流程
  async run() {
    const app = this.app;
    const processData = {};
    const pipeline = [{
      handler: context => app.fetchUpdate(context), // Git 更新代码
      name: 'codeUpdate',
      progress: 10 // 这里是当前构建的进度
    }, {
      handler: context => app.installDependency(context), // npm install 安装依赖
      name: 'dependency',
      progress: 30
    }, {
      handler: context => app.check(context), // 构建的前置校验（非必须）：代码检测，eslint，package.json 版本等
      name: 'check',
      progress: 40
    }, {
      handler: context => app.pack(context), // npm run build 的打包逻辑，如果有其他的项目类型，例如 gulp 之类，也可以在这一步进行处理
      name: 'pack', 
      progress: 70
    }, {
      handler: context => app.injectScript(context), // 构建的后置步骤（非必须）：打包后的资源注入
      name: 'injectRes',
      progress: 80
    }, { // docker image build
      handler: context => app.buildImage(context), // 生成 docker 镜像文件，镜像上传仓库，以及之后调用 K8S 能力进行部署
      name: 'buildImage',
      progress: 90
    }];
    // 循环执行每一步构建流程
    for (let i = 0; i < pipeline.length; i++) {
      const task = pipeline[i];
      const [ err, response ] = await to(this.execProcess({
        ...task,
        step: i
      }));
      if (response) {
        processData[task.name] = response;
      }
    }
    return Promise.resolve(processData);
  }
  // 执行构建中的 handler 操作
  async execProcess(task) {
    this.step(task.name, { status: 'start' });
    const result = await task.handler(this.buildContext);
    this.progress(task.progress);
    this.step(task.name, { status: 'end', taskMeta: result });
    return result;
  }

• Steps to build, a number of process building above, compared to we all want to know how to run the build process among some scripts on the server, in fact, the idea is through node of child_process shell script module, here are some sample code:

  import { spawn } from 'child_process';
  // git clone 
  execCmd(`git clone ${url} ${dir}`, {
  cwd: this.root,
  verbose: this.verbose
  });
  // npm run build
  const cmd = ['npm run build', cmdOption].filter(Boolean).join(' ');
  execCmd(cmd, options);
  // 执行 shell 命令
  function execCmd(cmd: string, options:any = {}): Promise<any> {
  const [ shell, ...args ] = cmd.split(' ').filter(Boolean);
  const { verbose, ...others } = options;
  return new Promise((resolve, reject) => {
    let child: any = spawn(shell, args, others);
    let stdout = '';
    let stderr = '';
    child.stdout && child.stdout.on('data', (buf: Buffer) => {
      stdout = `${stdout}${buf}`;
      if (verbose) {
        logger.info(`${buf}`);
      }
    });
    child.stderr && child.stderr.on('data', (buf: Buffer) => {
      stderr = `${stderr}${buf}`;
      if (verbose) {
        logger.error(`${buf}`);
      }
    });
    child.on('exit', (code: number) => {
      if (code !== 0) {
        const reason = stderr || 'some unknown error';
        reject(`exited with code ${code} due to ${reason}`);
      } else {
        resolve({stdout,  stderr});
      }
      child.kill();
      child = null;
    });
    child.on('error', err => {
      reject(err.message);
      child.kill();
      child = null;
    });
  });
  };

• For example, if we want to add Eslint verification operation before building, we can also add it in the build process, which means that we can add interception-type verification in the link of online construction to control the quality of the online build code.

  import { CLIEngine } from 'eslint';
  export function lintOnFiles(context) {
  const { root } = context;
  const [ err ] = createPluginSymLink(root);
  if (err) {
    return [ err ];
  }
  const linter = new CLIEngine({
    envs: [ 'browser' ],
    useEslintrc: true,
    cwd: root,
    configFile: path.join(__dirname, 'LintConfig.js'),
    ignorePattern: ['**/router-config.js']
  });
  let report = linter.executeOnFiles(['src']);
  const errorReport = CLIEngine.getErrorResults(report.results);
  const errorList = errorReport.map(item => {
    const file = path.relative(root, item.filePath);
    return {
      file,
      errorCount: item.errorCount,
      warningCount: item.warningCount,
      messages: item.messages
    };
  });
  const result = {
    errorList,
    errorCount: report.errorCount,
    warningCount: report.warningCount
  }
  return [ null, result ];
  };

• After the build deployment is completed, the update status information of this build record can be updated according to the build situation. The Docker image generated in this build, after uploading the image warehouse, also needs information records, so that the previously built image can be updated again later or Rollback operation, so you need to add a mirror table, the following are some example codes generated by Docker

  import Docker = require('dockerode');
  // 保证服务端中有一个基本的 dockerfile 镜像文件
  const docker = new Docker({ socketPath: '/var/run/docker.sock' });
  const image = '镜像打包名称'
  let buildStream;
  [ err, buildStream ] = await to(
  docker.buildImage({
    context: outputDir
  }, { t: image })
  );
  let pushStream;
  // authconfig 镜像仓库的一些验证信息
  const authconfig = {
  serveraddress: "镜像仓库地址"
  };
  // 向远端私有仓库推送镜像
  const dockerImage = docker.getImage(image);
  [ err, pushStream ] = await to(dockerImage.push({
  authconfig,
  tag
  }));
  // 3s 打印一次进度信息
  const progressLog = _.throttle((msg) => logger.info(msg), 3000); 
  const pushPromise = new Promise((resolve, reject) => {
  docker.modem.followProgress(pushStream, (err, res) => {
    err ? reject(err) : resolve(res);
  }, e => {
    if (e.error) {
      reject(e.error);
    } else {
      const { id, status, progressDetail } = e;
      if (progressDetail && !_.isEmpty(progressDetail)) {
        const { current, total } = progressDetail;
        const percent = Math.floor(current / total * 100);
        progressLog(`${id} : pushing progress ${percent}%`);
        if (percent === 100) { // 进度完成
          progressLog.flush();
        }
      } else if (id && status) {
        logger.info(`${id} : ${status}`);
      }
    }
  });
  });
  await to(pushPromise);

• Every time you build, you need to save some build progress, logs and other information, and you can add a log table to save the logs.

  Operation of multiple build instances

At this point, the build process of a project has successfully run through, but a build platform must not only be able to build and update one project at a time, so you can introduce a process pool at this time, so that your build platform can build multiple projects at the same time .

Node is a single-threaded model. When multiple independent and time-consuming tasks need to be executed, the tasks can only child_process to increase the processing speed. Therefore, it is also necessary to implement a process pool to control the running problems of multiple build processes. The pool idea is that the main process creates a task queue to control the number of child processes. When the child process completes the task, it continues to add new child processes through the task queue of the process to control the operation of concurrent processes. The process is implemented as follows.

ProcessPool.ts The following is part of the code of the process pool, mainly showing ideas.

import * as child_process from 'child_process';
import { cpus } from 'os';
import { EventEmitter } from 'events';
import TaskQueue from './TaskQueue';
import TaskMap from './TaskMap';
import { to } from '../util/tool';
export default class ProcessPool extends EventEmitter {
  private jobQueue: TaskQueue;
  private depth: number;
  private processorFile: string;
  private workerPath: string;
  private runningJobMap: TaskMap;
  private idlePool: Array<number>;
  private workPool: Map<any, any>;
  constructor(options: any = {}) {
    super();
    this.jobQueue = new TaskQueue('fap_pack_task_queue');
    this.runningJobMap = new TaskMap('fap_running_pack_task');
    this.depth = options.depth || cpus().length; // 最大的实例进程数量
    this.workerPath = options.workerPath;
    this.idlePool = []; // 工作进程  pid 数组
    this.workPool = new Map();  // 工作实例进程池
    this.init();
  }
  /**
   * @func init 初始化进程，
   */
  init() {
    while (this.workPool.size < this.depth) {
      this.forkProcess();
    }
  }
  /**
   * @func forkProcess fork 子进程，创建任务实例
   */
  forkProcess() {
    let worker: any = child_process.fork(this.workerPath);
    const pid = worker.pid;
    this.workPool.set(pid, worker);
    worker.on('message', async (data) => {
      const { cmd } = data;
      // 根据 cmd 状态 返回日志状态或者结束后清理掉任务队列
      if (cmd === 'log') {
      }
      if (cmd === 'finish' || cmd === 'fail') {
        this.killProcess();//结束后清除任务
      }
    });
    worker.on('exit', () => {
      // 结束后，清理实例队列，开启下一个任务
      this.workPool.delete(pid);
      worker = null;
      this.forkProcess();
      this.startNextJob();
    });
    return worker;
  }
  // 根据任务队列，获取下一个要进行的实例，开始任务
  async startNextJob() {
    this.run();
  }
  /**
   * @func add 添加构建任务
   * @param task 运行的构建程序
   */
  async add(task) {
    const inJobQueue = await this.jobQueue.isInQueue(task.appId); // 任务队列
    const isRunningTask = await this.runningJobMap.has(task.appId); // 正在运行的任务
    const existed = inJobQueue || isRunningTask;
    if (!existed) {
      const len = await this.jobQueue.enqueue(task, task.appId);
      // 执行任务
      const [err] = await to(this.run());
      if (err) {
        return Promise.reject(err);
      }
    } else {
      return Promise.reject(new Error('DuplicateTask'));
    }
  }
  /**
   * @func initChild 开始构建任务
   * @param child 子进程引用
   * @param processFile 运行的构建程序文件
   */
  initChild(child, processFile) {
    return new Promise(resolve => {
      child.send({ cmd: 'init', value: processFile }, resolve);
    });
  }
  /**
   * @func startChild 开始构建任务
   * @param child 子进程引用
   * @param task 构建任务
   */
  startChild(child, task) {
    child.send({ cmd: 'start', task });
  }
  /**
   * @func run 开始队列任务运行
   */
  async run() {
    const jobQueue = this.jobQueue;
    const isEmpty = await jobQueue.isEmpty();
    // 有空闲资源并且任务队列不为空
    if (this.idlePool.length > 0 && !isEmpty) {
      // 获取空闲构建子进程实例
      const taskProcess = this.getFreeProcess();
      await this.initChild(taskProcess, this.processorFile);
      const task = await jobQueue.dequeue();
      if (task) {
        await this.runningJobMap.set(task.appId, task);
        this.startChild(taskProcess, task);
        return task;
      }
    } else {
      return Promise.reject(new Error('NoIdleResource'));
    }
  }
  /**
   * @func getFreeProcess 获取空闲构建子进程
   */
  getFreeProcess() {
    if (this.idlePool.length) {
      const pid = this.idlePool.shift();
      return this.workPool.get(pid);
    }
    return null;
  }
  
  /**
   * @func killProcess 杀死某个子进程，原因：释放构建运行时占用的内存
   * @param pid 进程 pid
   */
  killProcess(pid) {
    let child = this.workPool.get(pid);
    child.disconnect();
    child && child.kill();
    this.workPool.delete(pid);
    child = null;
  }
}

Build.ts

import ProcessPool from './ProcessPool';
import TaskMap from './TaskMap';
import * as path from 'path';
// 日志存储
const runningPackTaskLog = new TaskMap('fap_running_pack_task_log');
//初始化进程池
const packQueue = new ProcessPool({
  workerPath: path.join(__dirname, '../../task/func/worker'),
  depth: 3
});
// 初始化构建文件
packQueue.process(path.join(__dirname, '../../task/func/server-build'));
let key: string;
packQueue.on('message', async data => {
  // 根据项目 id，部署记录 id，以及用户 id 来设定 redis 缓存的 key 值，之后进行日志存储
  key = `${appId}_${deployId}_${deployer.userId}`;
  const { cmd, value } = data;
  if(cmd === 'log') { // 构建任务日志
    runningPackTaskLog.set(key,value);
  } else if (cmd === 'finish') { // 构建完成
    runningPackTaskLog.delete(key);
    // 后续日志可以进行数据库存储
  } else if (cmd === 'fail') { // 构建失败
    runningPackTaskLog.delete(key);
    // 后续日志可以进行数据库存储
  }
  // 可以通过 websocket 将进度同步给前台展示
});
//添加新的构建任务
let [ err ] = await to(packQueue.add({
  ...appAttrs, // 构建所需信息
}));

After the process pool has processed the multi-process construction, how to record the construction progress of each process? I chose to use the Redis database to cache the construction progress status, and at the same time synchronize the progress display of the foreground through Websocket. After the construction is completed, perform Local storage of logs.
The above code briefly introduces the implementation and use of the process pool. Of course, the specific application depends on your own design ideas. With the help of the process pool, the rest of the idea is actually the implementation of the specific code.

The future of front-end construction

Finally, let’s talk about some of our thoughts on the future of front-end construction. First of all, the front-end construction must ensure a more stable construction. Under the premise of stability, it can achieve faster construction. For the CI/CD direction, such as a more complete construction Smoothly, after updating and generating the online environment, the code is archived automatically, and the latest Master code after the archive is reintegrated into each development branch, and then all the test environments are updated, etc.

Regarding the server performance, we have considered whether we can rely on each computer to complete the cloud construction ability to achieve local construction, cloud deployment offshore cloud construction, and distribute the server pressure to their respective computers. It can also alleviate the pressure of server-side construction, and the server-side can only do the final deployment service.

For example, our development classmates really want the project to be packaged and released according to the dimensions of the group. In a release version, select the project and version branch to be updated together, and release the update in a unified manner.

summary

Open source works

• Front-end tabloid of Zheng Caiyun

open source address www.zoo.team/openweekly/ (There is a WeChat exchange group on the homepage of the tabloid official website)

Recruitment

The ZooTeam front-end team (ZooTeam), a young, passionate and creative front-end team, belongs to the product R&D department of Zheng Caiyun, and the Base is located in picturesque Hangzhou. The team now has more than 40 front-end partners with an average age of 27 years old. Nearly 30% are full-stack engineers, a proper youth storm troupe. The membership consists of not only “veteran” soldiers from Ali and Netease, as well as newcomers from Zhejiang University, University of Science and Technology of China, Hangzhou Electric Power and other schools. In addition to the daily business docking, the team also conducts technical exploration and actual combat in the material system, engineering platform, building platform, performance experience, cloud application, data analysis and visualization, and promotes and implements a series of internal technical products. Explore the new boundaries of the front-end technology system.

If you want to change that you have been tossed by things, hope to start to toss things; if you want to change and have been warned, you need more ideas, but you can’t break the game; if you want to change you have the ability to make that result, but you don’t need you; if If you want to change what you want to accomplish, you need a team to support it, but there is no position for you to lead people; if you want to change the established rhythm, it will be "5 years of work time and 3 years of work experience"; if you want to change the original The comprehension is good, but there is always the ambiguity of the window paper... If you believe in the power of belief, believe that ordinary people can achieve extraordinary things, believe that you can meet a better self. If you want to participate in the process of business take-off, and personally promote the growth of a front-end team with in-depth business understanding, complete technical system, technology to create value, and influence spillover, I think we should talk. Anytime, waiting for you to write something, send it to [e-mail]

So with your own build and release platform, you can operate all the functions you want by yourself, and you can do all kinds of functions you want in the front end. Isn't it beautiful? I guess many students may be interested in the VsCode plug-in we made. In addition to building the project, there are of course other functions, such as the management of company test accounts, the rapid construction of small programs and other auxiliary development functions, do you want to To learn more about the functions of this plug-in, please look forward to our future sharing.

Reference documents

node child_process document

depth understanding of Node.js processes and threads

Analysis of Node Process and Thread