[Performance optimization] Speed up your builds with esbuild 🚀

background

Recently, it has been found that the construction of the project (based on Vue2) is relatively slow, and it takes 15 minutes for an online release, which is inefficient.

In today's era, time is money, and efficiency is life.

So I took time to do a build optimization for the project in the past two days. The online (multi-country) build time was optimized from 10 minutes to 4 minutes, and the local single build time was optimized from 300 seconds to 90 seconds, and the effect is still good good.

In the whole process, the transformation cost is not large, but the benefits are considerable.

Today, the detailed transformation process of and related technical principles of are sorted out and shared with you, hoping to help you.

text

First look at the problem in front of you:

It can be clearly seen that the overall construction of takes too long and is inefficient, which affects the release and rollback of services.

Online build process:

Among them, there is room for optimization in stages Build base and Build Region .

Build base stage of optimization, communicated with the operation and maintenance team, and will add cache processing in the future.

This time, we mainly focus on the Build Region stage.

After preliminary optimization, the results are as follows:

Basically meet expectations.

Process comparison before and after optimization:

The details of this optimization are described below.

Project optimization practice

Faced with the problem of time-consuming, the first thing to do is time-consuming data analysis.

SpeedMeasurePlugin is introduced here, the sample code is as follows:

# vue.config.js

const SpeedMeasurePlugin = require("speed-measure-webpack-plugin");

configureWebpack: (config) => {
  config.plugins.push(new SpeedMeasurePlugin());
}

The result is as follows:

得到： 

SMP  ⏱  Loaders

cache-loader, and 

vue-loader, and 

eslint-loader took 3 mins, 39.75 secs

  module count = 1894

cache-loader, and 

thread-loader, and 

babel-loader, and 

ts-loader, and 

eslint-loader took 3 mins, 35.23 secs

  module count = 482

cache-loader, and 

thread-loader, and 

babel-loader, and 

ts-loader, and 

cache-loader, and 

vue-loader took 3 mins, 16.98 secs

  module count = 941

cache-loader, and 

vue-loader, and 

cache-loader, and 

vue-loader took 3 mins, 9.005 secs

  module count = 947

mini-css-extract-plugin, and 

css-loader, and 

vue-loader, and 

postcss-loader, and 

sass-loader, and 

cache-loader, and 

vue-loader took 3 mins, 5.29 secs

  module count = 834

modules with no loaders took 1 min, 52.53 secs

  module count = 3258

mini-css-extract-plugin, and 

css-loader, and 

vue-loader, and 

postcss-loader, and 

cache-loader, and 

vue-loader took 27.29 secs

  module count = 25

css-loader, and 

vue-loader, and 

postcss-loader, and 

cache-loader, and 

vue-loader took 27.13 secs

  module count = 25

file-loader took 12.049 secs

  module count = 30

cache-loader, and 

thread-loader, and 

babel-loader took 11.62 secs

  module count = 30

url-loader took 11.51 secs

  module count = 70

mini-css-extract-plugin, and 

css-loader, and 

postcss-loader took 9.66 secs

  module count = 8

cache-loader, and 

thread-loader, and 

babel-loader, and 

ts-loader took 7.56 secs

  module count = 3

css-loader, and 

// ...


Build complete.

fetch translations

en has been saved!

id has been saved!

sp-MX has been saved!

vi has been saved!

zh-TW has been saved!

zh-CN has been saved!

th has been saved!

$ node ./script/copy-static-asset.js

✨  Done in 289.96s.

Count the loaders that take a lot of time:

Vue-loader 
eslint-loader
babel-loader
Ts-loader,
Thread-loader,
cache-loader

In general, code compilation time is positively related to code size.

According to the past optimization experience, the static check of the code may take up a lot of time, and the eyes are locked on eslint-loader .

In the production build stage, the eslint prompt information is of little value, consider removing it in the build stage, and prepend the step.

For example, check at commit , or add a pipeline at merge for static checking.

Give some sample code:

image: harbor.shopeemobile.com/shopee/nodejs-base:16

stages:
  - ci

ci_job:
  stage: ci
  allow_failure: false
  only:
    - merge_requests
  script:
    - npm i -g pnpm
    - pnpm pre-build && pnpm lint && pnpm test
  cache:
    paths:
      - node_modules
    key: project

Here, two optimization directions are initially determined:

1. Optimize the build process to remove unnecessary checks during the production build phase.
2. integrates esbuild to speed up the underlying build.

1. Optimize the build process

Checking the configuration of the project found:

# vue.config.js

lintOnSave: true,

change into:

# vue.config.js

lintOnSave: process.env.NODE_ENV !== 'production',

That is: Production builds without lint checking.

Vue's official website also has a description of this: https://cli.vuejs.org/zh/config/#lintonsave

Build again and get the following data:

 SMP  ⏱  Loaders
cache-loader, and 
vue-loader took 1 min, 34.33 secs
  module count = 2841
cache-loader, and 
thread-loader, and 
babel-loader, and 
ts-loader took 1 min, 33.56 secs
  module count = 485
vue-loader, and 
cache-loader, and 
thread-loader, and 
babel-loader, and 
ts-loader, and 
cache-loader, and 
vue-loader took 1 min, 31.41 secs
  module count = 1882
vue-loader, and 
mini-css-extract-plugin, and 
css-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 1 min, 29.55 secs
  module count = 1668
css-loader, and 
vue-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 1 min, 27.75 secs
  module count = 834
modules with no loaders took 59.89 secs
  module count = 3258
...

Build complete.
fetch translations
vi has been saved!
zh-TW has been saved!
en has been saved!
th has been saved!
sp-MX has been saved!
zh-CN has been saved!
id has been saved!
$ node ./script/copy-static-asset.js

✨  Done in 160.67s.

There is a certain improvement, and there is no obvious abnormality in the time-consuming data of other loaders.

Let's start integrating esbuid.

Integrate esbuild

This part of the work is mainly: Integrate the esbuild plugin into the scaffolding.

The modification of the specific code depends on the specific situation, which can be roughly divided into two categories:

1. I implemented the packaging logic with webpack myself.
2. It uses the packaging configuration that comes with cli, such as vue-cli.

I will introduce these two methods. Although the form of is different, the principle of is the same.

The core idea is as follows:

rules: [
    {
        test: /\.(js|jsx|ts|tsx)$/,
        loader: 'esbuild-loader',
        options: {
            charset: 'utf8',
            loader: 'tsx',
            target: 'es2015',
            tsconfigRaw: require('../../tsconfig.json'),
        },
        exclude: /node_modules/,
    },
    ...
]

const { ESBuildMinifyPlugin } = require('esbuild-loader');

optimization: {
    minimizer: [
        new ESBuildMinifyPlugin({
            target: 'es2015',
            css: true,
        }),
    ],
    ...
}

In terms of specific implementation, it is simply divided into two categories. The detailed configuration is as follows:

1. webpack.config.js

npm i -D esbuild-loader

1. Javascript & JSX transpilation (eg. Babel)

In webpack.config.js:

  module.exports = {
    module: {
      rules: [
-       {
-         test: /\.js$/,
-         use: 'babel-loader',
-       },
+       {
+         test: /\.js$/,
+         loader: 'esbuild-loader',
+         options: {
+           loader: 'jsx',  // Remove this if you're not using JSX
+           target: 'es2015'  // Syntax to compile to (see options below for possible values)
+         }
+       },

        ...
      ],
    },
  }

2. TypeScript & TSX

In webpack.config.js:

  module.exports = {
    module: {
      rules: [
-       {
-         test: /\.tsx?$/,
-         use: 'ts-loader'
-       },
+       {
+         test: /\.tsx?$/,
+         loader: 'esbuild-loader',
+         options: {
+           loader: 'tsx',  // Or 'ts' if you don't need tsx
+           target: 'es2015',
+            tsconfigRaw: require('./tsconfig.json'), // If you have a tsconfig.json file, esbuild-loader will automatically detect it.
+         }
+       },

        ...
      ]
    },
  }

3. JS Minification (eg. Terser)

esbuild also has a good performance in code compression:

For detailed comparison data, see: https://github.com/privatenumber/minification-benchmarks

In webpack.config.js:

+ const { ESBuildMinifyPlugin } = require('esbuild-loader')

  module.exports = {
    ...,

+   optimization: {
+     minimizer: [
+       new ESBuildMinifyPlugin({
+         target: 'es2015'  // Syntax to compile to (see options below for possible values)
+         css: true  // Apply minification to CSS assets
+       })
+     ]
+   },
  }

4. CSS in JS

If your css styles are not exported as css files, but loaded by eg 'style-loader', it can also be optimized by esbuild.

In webpack.config.js:

  module.exports = {
    module: {
      rules: [
        {
          test: /\.css$/i,
          use: [
            'style-loader',
            'css-loader',
+           {
+             loader: 'esbuild-loader',
+             options: {
+               loader: 'css',
+               minify: true
+             }
+           }
          ]
        }
      ]
    }
  }

For more esbuild examples, please refer to: https://github.com/privatenumber/esbuild-loader-examples

2. vue.config.js

The configuration is relatively simple, just paste the code directly:

// vue.config.js

const { ESBuildMinifyPlugin } = require('esbuild-loader');

module.exports = {
  // ...

  chainWebpack: (config) => {
    // 使用 esbuild 编译 js 文件
    const rule = config.module.rule('js');

    // 清理自带的 babel-loader
    rule.uses.clear();

    // 添加 esbuild-loader
    rule
      .use('esbuild-loader')
      .loader('esbuild-loader')
      .options({
        loader: 'ts', // 如果使用了 ts, 或者 vue 的 class 装饰器，则需要加上这个 option 配置， 否则会报错： ERROR: Unexpected "@"
        target: 'es2015',
        tsconfigRaw: require('./tsconfig.json')
      })

    // 删除底层 terser, 换用 esbuild-minimize-plugin
    config.optimization.minimizers.delete('terser');

    // 使用 esbuild 优化 css 压缩
    config.optimization
      .minimizer('esbuild')
      .use(ESBuildMinifyPlugin, [{ minify: true, css: true }]);
  }
}

After this combination of punches, a single local build:

The effect is still quite obvious.

For one online build, the overall time was reduced from 10 minutes to 4 minutes.

However, I was happy for less than two minutes, and found that the project next door could be done in 2 minutes...

I'm not convinced. It's also esbuild, why is yours so showy?

I did some research and found the reason.

1. Their project is React + TSX, and the project I optimized this time is Vue, which requires more than one layer of vue-loader in file processing.
2. Their project uses a micro-frontend, and the project is split. The main project only needs to load the code related to the base, and the sub-applications are built separately. The main reason is that the amount of main application code that needs to be built is greatly reduced.

This micro-frontend splitting method was mentioned in my previous article , if you are interested, you can take a look.

What you need to know about esbuild

The first part mainly introduces some practical details, which are basically configuration, without much in-depth content. This part will introduce more esbuild principle content as a supplement.

Last year, I also wrote two related content, if you are interested, you can go and see it.

1. "If you don't understand, just ask" Where is esbuild?
2. "If you don't understand, ask" Where is the performance bottleneck of webpack packaging?

This part will introduce you from 4 aspects.

1. What bottlenecks did the front end encounter & what problems can esbuild solve
2. Performance-first design philosophy & win-win cooperation with other tools
3. esbuild official positioning
4. Imagine the future of esbuild

1. What bottlenecks did the front end encounter & what problems can esbuild solve

Front-end engineering bottleneck

Build tools other than JS

Problems solved by esbuild

community plugin set

2. Performance-first design philosophy & win-win cooperation with other tools

Why is esbuild so fast?

1. Written in Golang, the operating efficiency is orders of magnitude different from JS
2. Almost all designs prioritize performance

Performance-first design philosophy

esbuild overall architecture

See: https://github.com/evanw/esbuild/blob/master/docs/architecture.md

If GOMAXPROCS is not configured, Golang will occupy all CPU cores when a large number of goroutines are running.

The above figure shows that, except for the operations related to the dependency graph and IO, all operations are parallel and do not require expensive serialization and copying costs.

It can be simply understood as: due to parallelism, an eight-core CPU can increase the compilation and compression speed by nearly eight times (without considering other process overhead).

In general, calling esbuild directly from the command line is the fastest, but as a front end, we cannot avoid using Node.js to write packaged configurations for the time being.

When the esbuild binary is invoked through Node.js, a child process is spawned and then the standard input and output of Node.js are piped to the child process. Writing data to the child process stdin means sending data, and listening to stdout means receiving the output data of the child process.

On the Golang side, if the --service startup parameter is found, runService will be executed, which will generate a channel called outgoingPackets, and the data written here will eventually be written to stdout (meaning sending data), from stdin in the main loop Reading data means receiving data.

In fact, the project structure of esbuild is not complicated. It is like this after removing documents and other things that are not related to the code. Following the standard project structure of Golang, the approximate call link is cmd -> pkg -> internal.

Because esbuild has more functions, the packages in the internal directory are more complicated than Babel. In addition, most of Babel's conversions are based on presets and plugins, but esbuild comes with the program itself, so the scalability is poor.

The bottom pkg packages are some packages that can be called by other Golang projects. Developers can easily call the esbuild API to build in Golang projects (just like writing a Webpack to call Babel).

An overview of the internal implementation of golang:

https://dreampuf.github.io/GraphvizOnline/

godepgraph -s -novendor ./cmd/esbuild

Win-win cooperation with other tools

Example of calling esbuild with Golang and Node.js (esbuild as part of other tool flow):

3. The official positioning of esbuild

Although esbuild is already excellent and has relatively complete functions, the author's meaning is to "explore another possibility for front-end construction", not to replace tools such as Webpack.

At present, for most projects, the best practice is probably to use esbuild-loader, and use esbuild only as a converter and code compression tool as part of the process.

The changelogs of esbuild in the last half year are very marginal problem fixes, and with the endorsement of Vite, it can be considered basically stable.

esbuild access method

1. Access via esbuild-loader

2. Call the esbuild binary directly

3. Umi comes with esbuild enabled

Two conclusions:

1. You need to decide which method to use to access according to your own project.
2. Optimizations vary from project to project, since build speed is not entirely dependent on esbuild.

4. Imagine the future of esbuild

Epilogue

esbuild is a powerful tool, I hope everyone can make full use of it and bring greater value to the business.

Well, that's all for today's content, I hope it can inspire everyone.

If there are any mistakes in the article, please leave a message to point out.

References

1. https://cli.vuejs.org/zh/config/#lintonsave
2. https://esbuild.github.io/getting-started/#your-first-bundle
3. https://morioh.com/p/cfd2609d744e
4. https://battlehawk233.cn/post/453.html
5. https://esbuild.github.io/api/#build-api
6. https://webpack.docschina.org/configuration/optimization/#optimizationminimizer
7. https://github.com/privatenumber/esbuild-loader