Gitlab CI/CD tutorial and npm package build and release actual combat

!!!In the process of practice, please pay attention to the version number of the document and the version number of Gitlab you actually use

👉🏻 Chinese document-14.8-pre

👉🏻 version - 12.9

Pre-knowledge

1. yaml syntax, tutorial 👉🏻 YAML introductory tutorial
2. Docker related knowledge, tutorial 👉🏻 Docker tutorial
3. linux command, tutorial 👉🏻 Linux Command Daquan

custom configuration directory

The default configuration file directory is the root directory of mono repo , and the file name is .gitlab-ci.yml .

If you need to customize the path of the CI script file, as follows:

Pipeline configuration

The pipeline configuration in the .gitlab-ci.yml file can be roughly divided into two links:

1. Global configuration, run before or after a single stage .
2. Single stage configuration

The structure can be roughly as follows:

# 指定脚本执行的镜像环境，如下为node环境为14.17.1
image: node:14.17.1

# 单个job执行之前执行
before_script:
  - echo '====== 准备构建中 ========='

# 配置单个stage的执行顺序，串行
stages:
  - install
  - build

# 单个stage配置
# 安装依赖
npm_install:
  only:
    - master
  stage: install
  script:
    - yarn
    - ls -al

# 单个stage配置
# 构建
webpack_build:
  only:
    - master
  stage: build
  script:
    - yarn build

# 单个job全部执行完之后执行
after_script:
  - echo "====== 构建结束 ========="

important concepts

Pipeline

Pipeline, a pipeline is equivalent to a build task, which can contain multiple stages, such as install -> eslint -> build -> deploy and other processes;

stages

Represents a build stage, each of which is executed synchronously in series. Once a job in a stage fails, the tasks of the next stage will not be executed. If the current stage defines multiple tasks, then one task fails, and the other task will continue to be executed. But the build task (Pipeline) is only successful if all stages have completed successfully.

jobs

A job represents the work performed in a stage, and multiple jobs can be defined in a stage.

jobs have the following characteristics:

• Jobs in the same stage are executed in parallel
• The stage will be successful only when the jobs in the same stage are executed successfully
• If any job fails, then the stage fails, that is, the build task (Pipeline) fails

gitlab runner

A service that performs build tasks, which contains a continuous integration environment, usually created by docker. It can be deployed on different hosts, or multiple gitlab-runners can be set up on the same host, and different environments can be set up according to different environments. For example, we need to distinguish the R&D environment, the test environment, and the formal environment.

keywords

image

The docker image of the CI/CD script running environment, the image is a form of file storage, which can be understood as a collection of environments, containing a variety of files. For example, specify the node environment image:

# 最新版本node环境
image: node:@latest

tags

Specifies which runner gitlab uses when executing scripts.

before_script

The content of the script executed before the execution of a single stage , the content is configured in the form of an array, as in the above example:

before_script:
  - echo '====== 准备构建中 ========='

stages

CI allows us to customize the pipeline stage configuration, which can split a pipeline into multiple stages ( stage ), and the stages will be executed serially.

stages:
  - install
  - build

script

Execute the script, and the content of the script is configured in the form of an array. In the above example, the script executed in the npm_install stage is:

script:
    - yarn
    - ls -al

First execute the yarn command to install dependencies, and then view the specific information of the files and directories in the current directory, which is a serial execution process.

cache

Cache files and directories shared between multiple pipeline tasks. The related concepts of caching are described in detail below.

only & except

Set the execution timing of pipeline tasks: use only to define when job runs, and use except to define when job and run.

• Specify the branch trigger execution timing

  job:
  
    only:
      - branches@gitlab-org/gitlab
    except:
      - main@gitlab-org/gitlab
      - /^release/.*$/@gitlab-org/gitlab

  This example runs gitlab-org/gitlab for all branches on job , except main and branches starting with release/ .

• Triggered on merge request

  job1:
    script:
      - echo "This job runs in merge request pipelines"
    only:
      - merge_requests

• Triggered on push

  job1:
    script:
      - echo "branch push"
    only:
      - pushes

• Manual trigger

  Triggered when the run pipeline is clicked in Gitlab Runner/pipeline

  job1:
    only:
      - web

• Trigger execution timing based on git commit messages or judging branches

  build:
    script: 
        - yarn build
    except:
      variables:
        - $CI_COMMIT_MESSAGE =~ /test/ || $CI_COMMIT_BRANCH == "main"

  The push with the git commit message "test" and the push with the commit branch "main" do not trigger this job.

• Judging execution timing based on file modification

  build:
    script: yarn build
    except:
      changes:
        - "*.md"

  This job is not executed as long as the md file is modified.

retry

The number of job retries, the default is 0, and the maximum number of retries is 2, of which when can be set to execute in the case of a specific failure reason.

rules:if

This field can be configured under the individual pipeline job or workflow field.

rules keyword can be configured under the if statement, and if the if is satisfied, some custom configurations can be performed.

rules:
  - if: $CI_COMMIT_REF_NAME =~ /feature/

Note that : only & except and rules:if are both used to determine the execution timing of a single job. Only one can exist during configuration, otherwise an error will be reported.

workflow

It is used in conjunction with rules to control the execution of the pipeline, and is configured in the outermost layer. workflow: rules accepts these keywords:

• if : Check this rule to determine when to run the pipeline.

• when : Specifies what to do when if is true.

  • To run the pipeline, set to always .
  • To prevent the pipeline from running, set to never .
• variables : If not defined, the variable defined elsewhere by is used. Applicable version 13.11 ~14.0

When no rule is true, the pipeline will not run.

In the following example, the rules of the first two days are matched to the timing not to be executed. When else, the pipeline is executed.

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: never
    - if: '$CI_PIPELINE_SOURCE == "push"'
      when: never
    - when: always

when

Controls the behavior of the current stage when the previous stage succeeds or fails.

• on_success (default): The current stage task will be executed only if the previous stage succeeds, or the previous failed task is configured with allow_failure: true .
• on_failure : The current task will only be executed if the previous stage task failed.
• always : Trigger tasks for the current stage regardless of the jobs status of the previous stage.
• never : Do not run current task.
• manual : Manual click trigger in gitlab webpage.

modular

Use the keyword include to import configurations from other yml files.

include:
  - '/yml/job1_install.yml'
  - '/yml/job2_lint.yml'
  - '/yml/job3_build.yml'
  - '/yml/job4_deploy.yml'

cache

important concepts

In GitLab CI/CD, the runner we use is in the form of docker to run different tasks. In the common cache mechanism (ie, no URL is specified, No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. ), the cache is stored locally, so if the actual running location of the two jobs is not on the host, the caches between them cannot be shared.

Distributed cache

Distributed cache requires runner configuration support. After enabling, you need to configure s3ServerAddress, s3BucketName and other information in the cache for cache sharing across runners.

cache path

When configuring the cache, the files/directories of paths and files are relative to the root directory of the project. When storing the cache, the cache path is also distinguished by the project name. When applying the cache, it will be matched with the key value under the project. Application caches, even distributed caches, follow this strategy.

The cache file information will include the last update time (important information), file permissions, cache set in key , etc. The update time is related to the cache policy. If the server time is maliciously tampered with, there may be inconsistent dependencies and resulting packages. Not as expected.

cache binding file

The cache is bound to the current version of the file. When one of these files changes, a new cache key is calculated and a new cache is created, as follows:

cache-job:
  script:
    - echo "This job uses a cache."
  cache:
    key:
      files:
        - Gemfile.lock
        - package.json
    paths:
      - vendor/ruby
      - node_modules

key SHA from the most recent commit that changed each of the listed files. key is the default value of default if no files were changed in any commit.

Multiple file cache

The cache can be configured with multiple key , which is suitable for versions 13.10 ~ 13.12. Other versions can be configured with multiple paths/files under files and key under paths .

disable cache

Use cache: {} to disable caching.

inherited cache

When the cache configuration is reusable, the inheritance writing method can be used to override (rewrite) a certain policy or set the priority under the current job.

cache: &global_cache
  key: $CI_COMMIT_REF_SLUG
  paths:
    - node_modules/
    - public/
    - vendor/
  policy: pull-push

job:
  cache:
    # 继承全局缓存
    <<: *global_cache
    # 重写缓存策略
    policy: pull

fallback cache key

Version 13.4 and above can apply the fallback cache key, which functions like a cache backup. You can use the CACHE_FALLBACK_KEY variable to specify a backup cache key, which will look for the application backup cache when the cache for the key you specify does not exist.

variables:
  CACHE_FALLBACK_KEY: fallback-key

job1:
  script:
    - echo
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - binaries/

Clear cache manually

You can clear the cache in the GitLab UI:

1. On the top bar, select Menu > Project and find your project.
2. On the left sidebar, select CI/CD > Pipeline page.
3. In the upper right corner, select to clear the Runner cache .

On the next commit, your CI/CD job uses the new cache.

Actual combat: build and publish component library to npm warehouse

Write the basic process of .gitlab-ci.yml as follows:

image: node:14.17.1

before_script:
  - echo '====== 准备构建中 ========='

stages:
  - install
  - lint
  - build
  - deploy

### 配置缓存
cache:
  key:
    files:
      - package.json
      - packages/ghost-weapp-ui/package.json
  paths:
    - node_modules/
    - packages/ghost-weapp-ui/node_modules/
    ### 直接缓存.npm，.npm中缓存了所有依赖，因为gitlab缓存是分项目的，所以两种方法个人觉得没有什么区别
    # - .npm/

# eslint检测
job_lint:
  only:
    - master
  stage: lint
  before_script:
    - echo 'eslint检测'
    - ls -a
  script:
    - cd packages/ghost-weapp-ui
    - ls -a
    - yarn lint
    - echo 'eslint检测完成'
  retry: 0
  when: 'on_success'

# 安装依赖
job_install:
  only:
    - master
  stage: install
  before_script:
    - echo '安装依赖'
  script:
    - yarn config set registry https://registry.npm.taobao.org/
    - yarn install
    - cd packages/ghost-weapp-ui
    - ls -a
    - yarn install
    - ls -a
    - echo '依赖安装完成'
  retry: 0

# 打包编译
job_build:
  only:
    - master
  stage: build
  before_script:
    - echo '开始打包'
  script:
    - cd packages/ghost-weapp-ui
    - ls -a
    - yarn
    - ls -a
    - yarn build
    - echo '构建完成'
  when: 'on_success'
  retry: 0

# 发布
job_deploy:
  only:
    - master
  stage: deploy
  before_script:
    - echo '更新补丁版本，准备发布'
  script:
    - cd packages/ghost-weapp-ui
    - node deploy.js ${CI_COMMIT_REF_NAME}
  when: 'on_success'
  retry: 0

after_script:
  - echo "====== 发布完成 ========="

Write a deployment script, which mainly simulates the process of npm login and replaces it with a token to log in and execute npm publish , the process of publishing to npm.

const fs = require('fs')
const path = require('path')
const os = require('os')
const { exec } = require('child_process')

// 替换为自己npm账号的authToken
// token获取方法：vim ~/.npmrc 
const npmrcText = `registry=https://registry.npmjs.org/
home=https://www.npmjs.org
//registry.npmjs.org/:_authToken=${authToken}
`

// 获取命令中第三个参数，此例子中为'master'
// node deploy.js ${CI_COMMIT_REF_NAME}, 分支名为master
const env = process.argv[2]

// 拼接命令，执行npm publish，打tag
function deploy() {
    fs.writeFileSync(path.resolve(os.homedir(), '.npmrc'), npmrcText)
    const argsArray = ['publish'].concat(['--tag', env === 'master' ? 'latest' : 'beta'])
    execa('npm', argsArray);
}

async function execa(a, arry = []) {
    return new Promise((resolve, reject) => {
        exec(`${a} ${arry.join(' ')}`, (err, stdout, stderr) => {
            if (err) {
                console.error(err);
                reject(err);
            }
            resolve(stdout)
        })
    })
}

deploy();

Results of the:

Received npm release success feedback email:

The above are the relevant knowledge points of gitlab CI/CD and the example of actual release npm package, thank you for reading!