Compilation optimization brought by multi-page application upgrade webpack5

1. Background

The current website project uses multiple pages, and development and reconstruction are separated. When developing new pages, because reconstruction development may often add style files or adjust the order of styles, the team adopted the following development model

With more and more pages, more style files depend on, and more gulp . The slow speed of packaging and compiling is gradually exposed. The screenshot of the packaging time in this mode is as follows:

You can see that the gulp task requires about 30s , and the webpack task requires 30s . In addition, the first build takes more than 1 minute, and the second build takes about 8 seconds. Imagine that this scene jumps out of your head every time you build. This is tolerable ? It can be seen that it has reached the point where (ren) must (wu) must (ke) be changed (ren), okay?

So decided to modify the current construction method, divided into the following steps

1. Remove gulp
The modified mode is as shown in the figure below. The green is the new process. Compared with the original process, the gulp package css process is handed over to webpack processing.

After switching to this mode, the first packaging time is about 20s, and incremental compilation is about 8s

2. webpack build speed optimization

2.1 exclude include optimization

Specify the files that need to be processed and excluded from processing in the matching rules, narrow the scope of search and processing files, ensure that as few files as possible are compiled, and add the corresponding include and exclude configurations for the project. The specific configuration is as follows

rules: [
    {
        test: /\.[jt]s$/,
        include: [path.resolve(process.cwd(), 'src')],
        exclude: [/node_modules/, path.resolve(process.cwd(), 'src/vendor')],
    },
    {
        test: /\.css$/,
        include: RESOURCES.CSS.cwd,
    }
]

After adding the above configuration, the first packaging time is about 16s, and the incremental compilation is about 8s

2.2 thread-loader and esbuild-loader optimization

thread-loader is a multi-process packaging method. The thread-loader behind loader will run in a worker
esbuild by Go developed for packaging compressed ts , js tool is characterized by swift package soon, the official github introduction and webpack , rollup , Parcel compared to tens or even hundreds of times faster, but esbuild not currently support css and no plug-in mechanism, so the time being can not be replaced webpack, but for webpack in loader That esbuild-loader , introducing it to try effect, Vite and Snowpack bottom have adopted esbuild , reference thread-loader official explanation and esbuild-loader official explanation The key configuration is as follows

const cpuNum = require('os').cpus().length;

const tsWorkerPool = {
    workers: 6,
    poolTimeout: Infinity
};

const cssWorkerPool = {
    workers: cpuNum - tsWorkerPool.workers,
    poolTimeout: Infinity
};

threadLoader.warmup(tsWorkerPool, ['esbuild-loader']);
threadLoader.warmup(cssWorkerPool, ['css-loader']);

module: {
    rules: [
        {
            test: /\.[jt]s$/,
            use: [
                {
                    loader: 'thread-loader',
                    options: tsWorkerPool
                },
                {
                    loader: 'esbuild-loader',
                    options: {
                        loader: 'ts',
                        target: 'es2015',
                        tsconfigRaw: require('../tsconfig.json')
                    },
                },
                {
                    loader: path.resolve(__dirname, 'loaders/importcss-loader.js'),
                    options: {
                        include: path.resolve(process.cwd(), 'src/app'),
                    }
                },
            ],
            include: [path.resolve(process.cwd(), 'src')],
            exclude: [/node_modules/, path.resolve(process.cwd(), 'src/vendor')],
        },
        {
            test: /\.css$/,
            use: [
                {
                    loader: MiniCssExtractPlugin.loader,
                },
                {
                    loader: 'thread-loader',
                    options: cssWorkerPool
                },
                {
                    loader: 'css-loader',
                    options: {
                        esModule: false,
                    }
                },
                {
                    loader: 'esbuild-loader',
                    options: {
                        loader: 'css',
                        minify: true
                    }
                }
            ],
            include: RESOURCES.CSS.cwd,
        }
    ],
},

After testing, plus thread-loader the first time about packing 12s , incremental compilation probably 3s , plus esbuild-loader after the first packaged shortened to 8s around, incremental compilation probably 2s
Note: The https://esbuild.github.io/api/#target document mentions that most of the syntax esbuild-loader only supports conversion to es6 , so it is only suitable for use in a development environment, and

2.3 Image compression and removal of redundant css style files

The image compression configuration is as follows

loader: 'image-webpack-loader',
options: {
    // 生产环境启用压缩
    disable: process.env.NODE_ENV === 'production' ? false : true,
    // 压缩 jpg/jpeg 图片
    mozjpeg: {
        progressive: true,
        quality: 80 // 压缩率
    },
    // 压缩 png 图片
    pngquant: {
        quality: [0.65, 0.90],
        speed: 4
    }
}

Remove redundant styles and use the purgecss plug-in, the configuration is as follows, you need to add configuration for special can not be removed

new PurgecssPlugin({
    paths: glob.sync([
        path.resolve(__dirname, '../../server/views/**/*.html'),
        path.resolve(`${PATH.SRC}/**/*.vue`),
    ]),
    safelist: [
        /data-v-.*/, // vue scope样式保留
        /market-message/, // 处理营销通知栏背景颜色由配置决定导致需要剔除样式不确定的特殊处理
        /vip/, // 动态渲染vip等级样式保留
    ]
})

3. Upgrade to webpack5

webpack released last year webpack5 brought a lot of optimization, the main points such as:
1. Persistent cache is turned on by default and cached in memory, while webpack4 needs to use cache-loader and hard-source-webpack-plugin for caching
2, NodeJS the polyfill scripts are removed, while in webpack4 and previous versions, for most Node module automatically adds polyfill script, resulting in the packing volume is large.
3. Better TreeShaking
After the above optimization, I want to further test webpack5 version can bring, upgrade the webpack package and related dependencies to the latest version, remove the incompatible speed-measure-webpack-plugin and the no longer needed hard-source-webpack-plugin plug-ins, and process about 65 files. The packaging time is reduced to 6.5s , and the incremental compilation time is 1.5s . It can be seen that the time has been further shortened.

4. Hot reload and hot update

Add the following configuration in webpack.dev.js

devServer: {
    contentBase: path.join(PATH.DIST, 'js'),
    inline: true,
    compress: true,
    port: 5000,
    writeToDisk: true,
    host: '0.0.0.0',
    hot: true,
    disableHostCheck: true,
    headers: {
        'Access-Control-Allow-Origin': 'https://www.midasbuy.com',
        'Access-Control-Allow-Headers': '*',
        'Access-Control-Allow-Methods': '*',
        'Access-Control-Allow-Credentials': true,
    },
},

Because we used the whistle agent to cooperate with domain name access during websocket to notify the file when the file was updated. Therefore, we need whistle . The first is to establish a websocket connection, and the second is to use request updated content when in hot update, this is mainly achieved vue heat of the updated components and js hot refresh, css not being realized with the new hot.

https://www.midasbuy.com:5000 http://127.0.0.1:5000/
^https://www.midasbuy.com/oversea_web/static/***hot-update***  http://127.0.0.1:5000/oversea_web/static/$1hot-update$2

Pit: When testing the hot refresh, the file is modified, and the following error is displayed after the page is refreshed

Then manually refresh it again. I didn’t find the reason for a long time. I suspected that it was the whistle agent. Later, I suspected that the notification update command was sent too early. Then I wrote a hack method to delay the sending of the update command. I found that it was OK, but The specific reason has not been found, until I switched to the server terminal, I saw the following print when updating the file

I understand that our process is client side, the file will be copied to the server folder to provide it to ejs include use. server uses nodemon . When the file is changed, it will copy the file operation and trigger the restart nodemon Causes the browser reload to fail the request, so the solution is to nodemon configuration file, because the real content of the copy is <style link='*****'></style> and <script src='*****'></script> , so it can be completely ignored. When the file changes after ignoring The server will not restart, and the problem is solved.

5. vue devtools locates to the component directory

For vue devtools provided in the browser to select components, you can directly open when you click the compiler and open source components where the code files, reference documentation in webpack.dev.js add the following configuration

const openInEditor = require('launch-editor-middleware');

devServer: {
    ...
    before(app) {
        app.use('/__open-in-editor', openInEditor('code', path.resolve(process.cwd(), 'src')));
    },
},

Add a line of rules in whistle

https://www.midasbuy.com/__open-in-editor http://127.0.0.1:5000/__open-in-editor

As shown in the figure below, when you click on the circled position, the vscode editor will open and locate the corresponding source file code of the component

6. Summary

In the MacBook Pro Intel Core i7 16G memory 250G hard disk size test environment, the comparison of the compilation time under different schemes and gzip compression results are as follows, in the 65 files, you can see that the compilation time is greatly shortened, and the js is also different. Decrease, but because some pictures are coded base64 css size of the style file has increased.

Program	First compile time	Second compilation time	buypage.js	propsOrder.js	buypage.css	propsOrder.css
gulp+webpack4	60s	8s	178kb	157kb	17.58kb	24.74kb
webpack5	6.5s	1.5s	123kb	112kb	28kb	35kb