Taro compilation and packaging optimization practice

1. Background

As the project gets larger and larger, the time-consuming compilation silently increases. Whether in the development phase or production integration, the time-consuming compilation has become a pain point that cannot be underestimated.

After nearly 5 years of continuous development and iteration, our project also migrated from the original WeChat native applet development method to Taro not long ago. Taro is a set of multi-terminal development solutions using React syntax. With Taro, we can write only one set of code, and then use Taro’s compilation tool to compile the source code separately to be available on different ends (WeChat/Baidu/Alipay/Byte Code running on beating/QQ/Jingdong applet, Kuaiapp, H5, React-Native, etc.). Therefore, many of Ctrip's small programs are also developed using Taro.

However, due to the relatively large number of businesses, the compiled code of the project is close to 12M. It takes nearly 1 minute to execute the build command in the daily development phase, just to compile and package the new development-related part of the file. It takes up to 10 minutes to execute the build command in the production environment and compile all the files in the packaged project. In addition, with the increasing number of infrastructure parts and single complex page functions, the amount of code is also increasing, which will cause the size of the main package or some sub-packages to exceed 2M, which will enable the QR code preview function of the WeChat developer tool Unavailable, the development experience is very bad.

In response to the above problems, we try to optimize the Taro compilation and packaging work. In order to optimize Taro's compilation and packaging, we need to understand the configuration of Taro's built-in Webpack, and then use the method provided by webpack-chain to modify the configuration in a chain. Next, we also need to solve the problem that the subcontract is too large to be able to preview the QR code.

Two, Taro's built-in Webpack configuration

We know that Taro's compilation and packaging work is done by webpack. Since we want to optimize the packaging speed, we must first know how Taro calls webpack for packaging, and we must also understand its built-in webpack configuration.

After reading the Taro source code, you can know that Taro is in the @tarojs/mini-runner/dist/index.js file and called webpack for packaging. You can check the relevant code by yourself.

We focus on the build function in this file, the code is as follows.

function build(appPath, config) {
    return __awaiter(this, void 0, void 0, function* () {
        const mode = config.mode;
        /** process config.sass options */
        const newConfig = yield chain_1.makeConfig(config);
        /** initialized chain */
        const webpackChain = build_conf_1.default(appPath, mode, newConfig);
        /** customized chain */
        yield customizeChain(webpackChain, newConfig.modifyWebpackChain, newConfig.webpackChain);
        if (typeof newConfig.onWebpackChainReady === 'function') {
            newConfig.onWebpackChainReady(webpackChain);
        }
        /** webpack config */
        const webpackConfig = webpackChain.toConfig();
        return new Promise((resolve, reject) => {
            const compiler = webpack(webpackConfig);
            const onBuildFinish = newConfig.onBuildFinish;
            let prerender;
            const onFinish = function (error, stats) {
                if (typeof onBuildFinish !== 'function')
                    return;
                onBuildFinish({
                    error,
                    stats,
                    isWatch: newConfig.isWatch
                });
            };
            const callback = (err, stats) => __awaiter(this, void 0, void 0, function* () {
                if (err || stats.hasErrors()) {
                    const error = err !== null && err !== void 0 ? err : stats.toJson().errors;
                    logHelper_1.printBuildError(error);
                    onFinish(error, null);
                    return reject(error);
                }
                if (!lodash_1.isEmpty(newConfig.prerender)) {
                    prerender = prerender !== null && prerender !== void 0 ? prerender : new prerender_1.Prerender(newConfig, webpackConfig, stats, config.template.Adapter);
                    yield prerender.render();
                }
                onFinish(null, stats);
                resolve(stats);
            });
            if (newConfig.isWatch) {
                logHelper_1.bindDevLogger(compiler);
                compiler.watch({
                    aggregateTimeout: 300,
                    poll: undefined
                }, callback);
            }
            else {
                logHelper_1.bindProdLogger(compiler);
                compiler.run(callback);
            }
        });
    });
}

As you can see, this function accepts two parameters, appPath and config, appPath is the directory of the current project, and the parameter config is the Taro configuration we wrote. Before calling webpack, Taro will process webpackConfig, including configuring Taro's built-in webpack and configuring the user's webpackChain in the Taro configuration file.

Locate the webpack location, then let's take a look at the relevant code of the webpack configuration finally generated by Taro.

const webpack = (options, callback) => {
    const webpackOptionsValidationErrors = validateSchema(
        webpackOptionsSchema,
        options
    );
    if (webpackOptionsValidationErrors.length) {
        throw new WebpackOptionsValidationError(webpackOptionsValidationErrors);
    }
    let compiler;
    if (Array.isArray(options)) {
        compiler = new MultiCompiler(
            Array.from(options).map(options => webpack(options))
        );
    } else if (typeof options === "object") {
        options = new WebpackOptionsDefaulter().process(options);

        compiler = new Compiler(options.context);
        compiler.options = options;
        new NodeEnvironmentPlugin({
            infrastructureLogging: options.infrastructureLogging
        }).apply(compiler);
        if (options.plugins && Array.isArray(options.plugins)) {
            for (const plugin of options.plugins) {
                if (typeof plugin === "function") {
                    plugin.call(compiler, compiler);
                } else {
                    plugin.apply(compiler);
                }
            }
        }
        compiler.hooks.environment.call();
        compiler.hooks.afterEnvironment.call();
        compiler.options = new WebpackOptionsApply().process(options, compiler);
    } else {
        throw new Error("Invalid argument: options");
    }
    if (callback) {
        if (typeof callback !== "function") {
            throw new Error("Invalid argument: callback");
        }
        if (
            options.watch === true ||
            (Array.isArray(options) && options.some(o => o.watch))
        ) {
            const watchOptions = Array.isArray(options)
                ? options.map(o => o.watchOptions || {})
                : options.watchOptions || {};
            return compiler.watch(watchOptions, callback);
        }
        compiler.run(callback);
    }
    return compiler;
};

It should be noted that there is a difference between the built-in webpack configuration in the development and production environments. For example, in the production environment, terser-webpack-plugin will be called for file compression processing. We use the vscode code editor. Before calling the webpack location, the debugger interrupts the point and uses the console command to output the variable webpackConfig, which is the final webpack configuration. In the command line tool DEBUG CONSOLE that comes with vscode, you can easily click to expand the object properties to view the webpack configuration generated by Taro. Shown here, in the development environment, Taro's built-in webpack configuration, as shown below.

These are common webpack configurations. We mainly focus on two parts. One is the rules configured in the module, and various loaders are configured to process matching corresponding files, such as common processing scss files and jsx files. The second is the TaroMiniPlugin plug-in configured in the plugins. The plug-in is built in Taro and is mainly responsible for compiling and packaging the code into a small program code.

Now that we understand the webpack configuration in Taro and one of their working processes, the next thing to consider is how to modify and optimize the configuration to help us optimize the speed of compilation and packaging. It should be noted that Taro uses the webpack-chain mechanism for packaging. The webpack configuration is essentially an object, which is troublesome to create and modify. Webpack-chain provides a chained API to create and modify webpack configurations. The Key part of the API can be referenced by a user-specified name, which helps standardize the configuration of cross-project modification.

webpack-chain itself provides many examples, you can refer to: https://github.com/Yatoo2018/webpack-chain/tree/zh-cmn-Hans

Three, optimize the Webpack packaging configuration

After the previous introduction, we have already understood the webpack configuration generated by Taro, and also mastered the methods to modify these configurations. The next step is to consider how to modify the webpack configuration to optimize the compilation and packaging speed. To this end, we have introduced speed-measure-webpack-plugin, which can count the time consumption of plugin and loader during the compilation and packaging process, which can help us clarify the direction of optimization.

After configuring the speed-measure-webpack-plugin, execute the build command again, and the output result is shown in the figure below.

It can be seen that the TaroMiniPlugin accounts for more than 2 minutes of the total compilation time of 3 minutes, which is still very time-consuming. TaroMiniPlugin is Taro's built-in webpack plugin. Most of Taro's compilation and packaging work is configured here, such as obtaining configuration content, processing sub-packages and tabbars, reading the page of the applet configuration and adding them to the dependencies array for subsequent processing , Generate small program related files, etc. The second most time-consuming one is TerserPlugin, which is mainly used for compressing files.

In the time-consuming statistics of loaders, babel-loader took two and a half minutes, and sass-loader took two minutes. These two are the most time-consuming. These two are also the main reasons why TaroMiniPlugin is so time-consuming. Because of this plug-in, files such as applet pages, components, etc. will be added to the entry file through webpack's compilation.addEntry, and a complete compilation phase in webpack will be executed later, during which the configured loader will be called for processing. Of course, babel-loader and scss-loader are also called to process js files or scss files, which severely slows down the speed of TaroMiniPlugin, leading to serious time-consuming statistics for the plug-in.

Therefore, optimizing the packaging of Webpack is mainly based on these two loaders, which is equivalent to optimizing the TaroMiniPlugin. In the optimization scheme, we have selected two common optimization strategies: multi-core and cache.

3.1 Multi-core

For multi-core, we use the thread-loader officially recommended by webpack, which can dump loaders that consume a lot of resources to the worker pool. According to the above-mentioned time-consuming statistics, it can be known that babel-loader is the most time-consuming loader, so the thread-loader is placed before babel-loader, so that babel-loader will run in a separate worker pool, thereby improving compilation efficiency.

Knowing the optimization method, the next thing to consider is how to configure it in webpack. Here we use the modifyWebpackChain hook provided by the Taro plug-in mechanism, and use the method provided by webpack-chain to modify the webpack configuration in a chain.

The specific method is to first find a way to delete the built-in babel-loader in Taro. We can look back at Taro's built-in webpack configuration and find that the named rule for processing babel-loader is'script', as shown in the figure below, and then use webpack-chain syntax The rule deletes the named rule.

Finally, through the merge method provided by webpack-chain, reconfigure the babel-loader that processes js files, and at the same time introduce thread-loader before babel-loader, as shown below.

ctx.modifyWebpackChain(args => {
  const chain = args.chain
  chain.module.rules.delete('script') // 删除Taro中配置的babel-loader
  chain.merge({ // 重新配置babel-loader
    module: {
      rule: {
        script: {
          test: /\.[tj]sx?$/i,
          use: {
            threadLoader: {
              loader: 'thread-loader', // 多核构建
            },
            babelLoader: {
              loader: 'babel-loader',
              options: {
                cacheDirectory: true, // 开启babel-loader缓存
              },
            },
          },
        },
      },
    }
  })
})

Of course, here we introduce thread-loader only to deal with babel-loader, you can also use it to deal with other time-consuming loaders such as css-loader.

3.2 Cache

In addition to enabling multithreading, in order to optimize the packaging speed, the cache needs to be optimized. The cache optimization strategy is also carried out for these two parts. One is to use cache-loader to cache loaders used to process scss files, and the other is to use babel-loader. Set the parameter cacheDirectory to true to enable babel-loader caching.

When using the cache-loader cache, an extra note is that the cache-loader needs to be placed before the css-loader and after the mini-css-extract-plugin. In practice, it is found that the generated files cannot be effectively cached when placed before the mini-css-extract-plugin/loader.

Similar to the previous approach, first we need to check the cache strategy of Taro's built-in webpack configuration, then use the webpack-chain syntax to locate the corresponding position, and finally call the before method to insert it before the css-loader.

Through the webpack-chain method, place the cache-loader before the css-loader and after the mini-css-extract-plugin, the code is as follows:

chain.module.rule('scss').oneOf('0').use('cacheLoader').loader('cache-loader').before('1')
chain.module.rule('scss').oneOf('1').use('cacheLoader').loader('cache-loader').before('1')

Note: The cache is stored in node_moduls/.cache by default, as shown in the figure below. Therefore, when using the execute compilation and packaging command, you need to pay attention to whether the current packaging environment can retain the cache, otherwise the cache configuration will not bring about the speed optimization effect.

It is worth mentioning that, looking at the picture above, we can find that terser-webpack-plugin also has caching enabled. Let's look back again. The following figure shows the parameters configured in Taro. We can find that both cache and parallel are true, indicating that they are also enabled for caching and parallel compilation respectively.

3.3 taro-plugin-compiler-optimization plugin

With the above optimization solution, we then wrote the optimization plug-in by hand. In general, this plugin uses the modifyWebpackChain hook exposed by the Taro plugin mechanism, and uses the webpack-chain method to modify the webpack configuration in a chain. Configure multi-core and cache optimization strategies into Taro's webpack to improve compilation and packaging speed.

The installation address of the plug-in is as follows:
GitHub：https://github.com/CANntyield/taro-plugin-compiler-optimization
Npm：https://www.npmjs.com/package/taro-plugin-compiler-optimization

First, install the plugin in the project:

npm install --save-dev thread-loader cache-loader taro-plugin-compiler-optimization

Then, add the following script to taro's config.js:

// 将其配置到taro config.js中的plugins中
// 根目录/config/index.js
plugins: ['taro-plugin-compiler-optimization']

Finally, we performed the packaging task again and found that the total time has been shortened to 56.9s. The time consumption of TaroMiniPlugin, babel-loader and css-loader has been significantly shortened, and the TerserPlugin with cache has also been shortened from 22.8s to 13.9s. , The optimization effect is still very significant.

Fourth, compress resource files

In the WeChat developer tool, if you want to debug a small program on a real device, you usually need to preview the QR code. Due to WeChat restrictions, the packaged files, the main package and sub-package files cannot exceed 2M, otherwise the QR code preview will not succeed. However, as the project becomes larger and larger, it is impossible for the main package file to exceed 2M, especially the file processed by babel-loader, which will contain a lot of comments, too long variable names, etc., resulting in files is too big. The most fundamental solution in the industry is subcontracting, because the WeChat applet has adjusted the total package size to 10M. However, this article does not discuss how to subcontract, here we mainly discuss how to adjust the size of the package.

When we execute the build command, we enable the terser-webpack-plugin compressed file to reduce the main package file to less than 2M. However, the problem is also very obvious, that is, it takes a lot of time to build and package each time, and the efficiency is too low. Moreover, in this case, it will not monitor file changes and perform module hot replacement work, which is very inefficient.

Therefore, our strategy is to configure webpack in the development environment and call terser-webpack-plugin for compression. At the same time, configure the plug-in parameters to compress the specified files instead of compressing them all. Open the WeChat developer tool and click on code dependency analysis, as shown in the figure below.

As can be seen from the figure, the main package file has exceeded 2M. Among them, the four files common.js, taro.js, vendors.js, and app.js are obviously larger, and these four files are inevitably generated after each Taro project is compiled and packaged. The pages folder is also as high as 1.41M, which is the tabBar page we configured, so the size of the folder is directly affected by the complexity of the tabBar page. In addition, other files are relatively small and can be ignored for the time being.

First, execute the following command to install terser-webpack-plugin.

npm install -D terser-webpack-plugin@3.0.5

It should be noted that the latest version of terser-webpack-plugin is already v5. This version is optimized according to webpack5, but does not support webpack4, so you need to specify an additional version before you can use it. Here I chose 3.0.5, which is the same version as the terser-webpack-plugin used in Taro. Among them, the incoming parameter configuration is the same as Taro. What we need to do is to add the file path that needs to be compressed to the test array. Common.js, taro.js, vendors.js, and vendors are already configured by default. app.js, pages/homoe/index.js files.

Similarly, we need to introduce the Taro plugin in the Taro configuration file plugins. It is recommended to introduce it in the config/dev.js configuration file. It will only be used in the development environment.

// config/dev.js
plugins: [
    path.resolve(__dirname, 'plugins/minifyMainPackage.js'),
]

Finally, let's take a look at the size of the main package after compression, and we can find that it has been reduced to 1.42M, which is about 50% compressed compared to the previous 3.45M, which can solve most scenarios where QR code preview packaging cannot be performed.

However, at present, the WeChat applet already supports subcontracting Lee, but the main package still cannot exceed 2M. The above method is for the solution where the main package is too large. This article mainly solves two problems: one is to optimize the speed of Taro compilation and packaging, and the other is to provide a solution to solve the problem that the WeChat developer tool cannot be used for QR code preview due to excessive subcontracting.