Webpack principle series 8: product translation packaging logic

The full text is 6000 words, let's talk about packaging closed loop, welcome to like, follow and forward.

Looking back, in the previous article " bit difficult webpack knowledge point: Dependency Graph deep analysis " has been talked, after build (make) stage , Webpack parsed out:

• module content
• module module between 060deb97f14196 and 060deb97f14198

After entering the generation ( seal ) phase , Webpack first calculates the number of Chunk Graphs and the content of the final product according to the module dependencies, module characteristics, entry configuration, etc. bit difficult knowledge point: Detailed explanation of Webpack Chunk subcontracting rules " also has a more detailed description.

This article continues to talk about the behind of Chunk Graph, the module begins to be translated into the process of module merging and packaging, the general process is as follows:

To facilitate understanding, I divided the packaging process horizontally into three stages:

• entry : Refers to all pre-operations from Webpack startup to compilation.codeGeneration
• module translation : traverse the modules array, complete the translation operation of all modules, and store the result in the compilation.codeGenerationResults object
• module merge and package : under a specific context framework, combine business modules and runtime modules, merge and package them into a bundle, and call compilation.emitAsset output the product

The business module here refers to the project code written by the developer; runtime module refers to the runtime code dynamically injected to support various features after Webpack analyzes the business module, in the previous article Webpack Principle series 6: Thorough understanding of Webpack runtime has been explained in detail, so I won’t go into details here.

As you can see, Webpack first modules into module products one by one- module translates , and then splices the module products into bundles- modules merge and pack . We will discuss the principles of these two processes separately according to this logic.

1. The principle of module translation

1.1 Introduction

First review the Webpack products:

The above example consists of index.js / name.js . The corresponding Webpack configuration is shown in the lower left corner of the above figure; the Webpack build product is main.js file on the right. It contains three pieces of content, from top to bottom:

• The translation product corresponding to the name.js
• Runtime code injected by Webpack on demand
• index.js module, IIFE (immediate execution function) form

Among them, the function and generation logic of the runtime code are described in the previous article Webpack Principle Series 6: Thorough understanding of Webpack runtime has been introduced in detail; the other two pieces are name.js and index.js respectively. The product after construction, you can see the product and source code. The semantics and functions are the same, but the form of expression has undergone major changes. For example, the content before and after compilation of index.js

The right side of the above figure is the corresponding code in the Webpack compiled product, which has the following changes relative to the source code on the left:

• The entire module is wrapped into IIFE (immediate execution function)
• Add __webpack_require__.r(__webpack_exports__); statement to adapt to ESM specification
• import statement in the source code is translated into a __webpack_require__ function call
• name variable used in the source code console statement is translated into _name__WEBPACK_IMPORTED_MODULE_0__.default
• add notes

So how are these transformations performed in Webpack?

1.2 Core process

module translates the operation from the module.codeGeneration , which corresponds to the above flowchart:

Summarize the key steps:

• Call JavascriptGenerator object generate method, internal method:

  • Traverse the dependencies and presentationalDependencies arrays of the module
  • Execute the corresponding template.apply method of each array item dependeny object, modify the module code apply initFragments array
• After the traversal is completed, call the InitFragment.addToSource static method to source object and the initFragments array generated by the previous operation into the module product

Simply put, it is to traverse the dependencies, modify the module code in the dependent objects, and finally merge all the changes into the final product. The key points here:

• How to update the module code in Template.apply
• In the InitFragment.addToSource static method, how to Template.apply the side effects generated by 060deb97f14983 into the final product

The logic of these two parts is more complicated and will be explained separately below.

1.3 Template.apply function

In the above process, the JavascriptGenerator class is an undoubted C-bit role, but it does not directly modify module , but after a few layers, it is delegated to Template implemented by the 060deb97f149d8 type.

In the Webpack 5 source code, the JavascriptGenerator.generate function will traverse the dependencies Template subclass apply method corresponding to the dependent object to update the module content. It is a bit convoluted, and the original code is more generous, so I extracted the important steps into the following pseudo code:

class JavascriptGenerator {
    generate(module, generateContext) {
        // 先取出 module 的原始代码内容
        const source = new ReplaceSource(module.originalSource());
        const { dependencies, presentationalDependencies } = module;
        const initFragments = [];
        for (const dependency of [...dependencies, ...presentationalDependencies]) {
            // 找到 dependency 对应的 template
            const template = generateContext.dependencyTemplates.get(dependency.constructor);
            // 调用 template.apply，传入 source、initFragments
            // 在 apply 函数可以直接修改 source 内容，或者更改 initFragments 数组，影响后续转译逻辑
            template.apply(dependency, source, {initFragments})
        }
        // 遍历完毕后，调用 InitFragment.addToSource 合并 source 与 initFragments
        return InitFragment.addToSource(source, initFragments, generateContext);
    }
}

// Dependency 子类
class xxxDependency extends Dependency {}

// Dependency 子类对应的 Template 定义
const xxxDependency.Template = class xxxDependencyTemplate extends Template {
    apply(dep, source, {initFragments}) {
        // 1. 直接操作 source，更改模块代码
        source.replace(dep.range[0], dep.range[1] - 1, 'some thing')
        // 2. 通过添加 InitFragment 实例，补充代码
        initFragments.push(new xxxInitFragment())
    }
}

It can be seen from the above pseudo code that JavascriptGenerator.generate function is relatively solid:

1. Initialize a series of variables
2. Traverse the dependency array of the module object, find the template dependency , and call the template.apply function to modify the content of the module
3. Call the InitFragment.addToSource method, merge the source and initFragments arrays to generate the final result

The point here is that the JavascriptGenerator.generate function does not operate on the module source code. It only provides an execution framework. The logic that actually handles the translation of the module content is implemented in the apply xxxDependencyTemplate object, as shown in lines 24-28 in the pseudo code above.

Each Dependency subclass will be mapped to a unique Template subclass, and usually these two classes will be written in the same file, such as ConstDependency and ConstDependencyTemplate ; NullDependency and NullDependencyTemplate . In the make phase of Dependency , the dependencies between modules under different conditions module code will be modified Template

In summary, the Module , JavascriptGenerator , Dependency Template form the following interaction relationship:

Template object can update the code of module

• Directly manipulate the source object and directly modify the module code. The original content of the object is equal to the source code of the module. After multiple Template.apply functions, it is gradually replaced with a new code form
• Operate the initFragments array and insert supplementary code snippets outside the module source code

The side effects generated by these two operations will eventually be passed into the InitFragment.addToSource function to synthesize the final result. Here are some details.

1.3.1 Use Source to change the code

Source is a tool system for editing strings in Webpack. It provides a series of string manipulation methods, including:

• String merge, replace, insert, etc.
• Module code cache, sourcemap mapping, hash calculation, etc.

Many plugins and loaders inside Webpack and the community will use the Source library to edit the code content, including the Template.apply system introduced above. Logically, when starting the module code generation process, Webpack will first initialize the Source object with the original content of the module, that is :

const source = new ReplaceSource(module.originalSource());

After that, different Dependency source sequentially and as needed, for example, the core code in ConstDependencyTemplate

ConstDependency.Template = class ConstDependencyTemplate extends (
  NullDependency.Template
) {
  apply(dependency, source, templateContext) {
    // ...
    if (typeof dep.range === "number") {
      source.insert(dep.range, dep.expression);
      return;
    }

    source.replace(dep.range[0], dep.range[1] - 1, dep.expression);
  }
};

In the above ConstDependencyTemplate , the apply function calls source.insert insert a piece of code source.replace replace a piece of code.

1.3.2 Update code using InitFragment

In addition to directly operating source , Template.apply can also be used to modify the module product initFragments initFragments array items are usually InitFragment subclass, and they usually have two functions: getContent and getEndContent , which are used to obtain the head and tail parts of the code fragment respectively.

E.g. HarmonyImportDependencyTemplate of apply function:

HarmonyImportDependency.Template = class HarmonyImportDependencyTemplate extends (
  ModuleDependency.Template
) {
  apply(dependency, source, templateContext) {
    // ...
    templateContext.initFragments.push(
        new ConditionalInitFragment(
          importStatement[0] + importStatement[1],
          InitFragment.STAGE_HARMONY_IMPORTS,
          dep.sourceOrder,
          key,
          runtimeCondition
        )
      );
    //...
  }
 }

1.4 Code merger

After the above Template.apply processed, the translated source object and the code fragment initFragments InitFragment.addToSource function needs to be called to merge the two into a module product.

The core code of addToSource

class InitFragment {
  static addToSource(source, initFragments, generateContext) {
    // 先排好顺序
    const sortedFragments = initFragments
      .map(extractFragmentIndex)
      .sort(sortFragmentWithIndex);
    // ...

    const concatSource = new ConcatSource();
    const endContents = [];
    for (const fragment of sortedFragments) {
        // 合并 fragment.getContent 取出的片段内容
      concatSource.add(fragment.getContent(generateContext));
      const endContent = fragment.getEndContent(generateContext);
      if (endContent) {
        endContents.push(endContent);
      }
    }

    // 合并 source
    concatSource.add(source);
    // 合并 fragment.getEndContent 取出的片段内容
    for (const content of endContents.reverse()) {
      concatSource.add(content);
    }
    return concatSource;
  }
}

As you can see, the logic of the addToSource

• Traverse the initFragments array and merge the products of fragment.getContent()
• Merge source objects
• Traverse the initFragments array and merge the products of fragment.getEndContent()

Therefore, the module code merging operation is mainly to wrap the module code source initFragments array, and both are Template.apply level.

1.5 Example: Custom banner plugin

After the Template.apply translation and InitFragment.addToSource merge, the module has completed the transformation from the user code form to the product form. In order to deepen the understanding of the above-mentioned module translation process, we will try to develop a Banner plug-in to realize automatic before each module Insert a string.

In terms of implementation, the plug-in mainly involves Dependency , Template , hooks objects, the code:

const { Dependency, Template } = require("webpack");

class DemoDependency extends Dependency {
  constructor() {
    super();
  }
}

DemoDependency.Template = class DemoDependencyTemplate extends Template {
  apply(dependency, source) {
    const today = new Date().toLocaleDateString();
    source.insert(0, `/* Author: Tecvan */
/* Date: ${today} */
`);
  }
};

module.exports = class DemoPlugin {
  apply(compiler) {
    compiler.hooks.thisCompilation.tap("DemoPlugin", (compilation) => {
      // 调用 dependencyTemplates ，注册 Dependency 到 Template 的映射
      compilation.dependencyTemplates.set(
        DemoDependency,
        new DemoDependency.Template()
      );
      compilation.hooks.succeedModule.tap("DemoPlugin", (module) => {
        // 模块构建完毕后，插入 DemoDependency 对象
        module.addDependency(new DemoDependency());
      });
    });
  }
};

The key steps of the sample plug-in:

• Write the DemoDependency and DemoDependencyTemplate classes, of which DemoDependency is only for example and has no actual function; DemoDependencyTemplate calls source.insert in its apply to insert a string, such as lines 10-14 of the sample code
• Use compilation.dependencyTemplates register the mapping relationship between DemoDependency and DemoDependencyTemplate
• Use the thisCompilation hook to get the compilation object
• Use the succeedModule hook to subscribe to the module build completion event, and call the module.addDependency method to add the DemoDependency dependency

After completing the above operations, the product of the module DemoDependencyTemplate.apply function during the generation process and insert the string we have defined. The effect is as follows:

Interested readers can also directly read the following files in the Webpack 5 warehouse to learn more use cases:

• lib/dependencies/ConstDependency.js, a simple example, you can learn more operation methods of source
• lib/dependencies/HarmonyExportSpecifierDependencyTemplate.js, a simple example to learn more usage of initFragments
• lib/dependencies/HarmonyImportDependencyTemplate.js, a more complex but highly used example, can comprehensively learn the usage of source and initFragments

Second, the principle of module consolidation and packaging

2.1 Introduction

After talking about the translation process of a single module, let's return to this flowchart first:

In the flowchart, the compilation.codeGeneration function is executed-that is, after the module translation phase is completed, the translation results of the modules will be saved to the compilation.codeGenerationResults object one by one, and then a new execution process will be started-the module is merged and packaged .

module combined packaging process will chunk corresponding module according to the rules and runtimeModule into template frame , the final combined output files into a complete bundle, for example, the above example:

In the bundle file on the right side of the example, the part in the red frame is the product generated by the user code file and the runtime module, and the remaining part supports a runtime framework in the form of template framework 160deb97f1559b, which is:

(() => { // webpackBootstrap
    "use strict";
    var __webpack_modules__ = ({
        "module-a": ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
            // ! module 代码，
        }),
        "module-b": ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
            // ! module 代码，
        })
    });
    // The module cache
    var __webpack_module_cache__ = {};
    // The require function
    function __webpack_require__(moduleId) {
        // ! webpack CMD 实现
    }
    /************************************************************************/
    // ! 各种 runtime
    /************************************************************************/
    var __webpack_exports__ = {};
    // This entry need to be wrapped in an IIFE because it need to be isolated against other modules in the chunk.
    (() => {
        // ! entry 模块
    })();
})();

Looking at the logic here, the running framework contains the following key parts:

• The outermost layer is wrapped by an IIFE
• A __webpack_modules__ object that records other module codes entry . The key of the object is the module identifier; the value is the translated code of the module
• An extremely simplified CMD implementation: __webpack_require__ function
• Finally, a parcel of entry IIFE function code

module translation is module translated into a code form that can run on a host environment such as a browser; and module combined operating the series of these modules , so that the entire line with the expected development, the entire application logic can run normally. Next, we reveal the generation principle of this part of the code.

2.2 Core process

After the compilation.codeGeneration completed, that is, after all user code modules and runtime modules have performed the translation operation, the seal function calls the compilation.createChunkAssets function to trigger the renderManifest hook, and the JavascriptModulesPlugin plug-in listens to the hook message and starts to assemble the bundle. The pseudo code:

// Webpack 5
// lib/Compilation.js
class Compilation {
  seal() {
    // 先把所有模块的代码都转译，准备好
    this.codeGenerationResults = this.codeGeneration(this.modules);
    // 1. 调用 createChunkAssets
    this.createChunkAssets();
  }

  createChunkAssets() {
    // 遍历 chunks ，为每个 chunk 执行 render 操作
    for (const chunk of this.chunks) {
      // 2. 触发 renderManifest 钩子
      const res = this.hooks.renderManifest.call([], {
        chunk,
        codeGenerationResults: this.codeGenerationResults,
        ...others,
      });
      // 提交组装结果
      this.emitAsset(res.render(), ...others);
    }
  }
}

// lib/javascript/JavascriptModulesPlugin.js
class JavascriptModulesPlugin {
  apply() {
    compiler.hooks.compilation.tap("JavascriptModulesPlugin", (compilation) => {
      compilation.hooks.renderManifest.tap("JavascriptModulesPlugin", (result, options) => {
          // JavascriptModulesPlugin 插件中通过 renderManifest 钩子返回组装函数 render
          const render = () =>
            // render 内部根据 chunk 内容，选择使用模板 `renderMain` 或 `renderChunk`
            // 3. 监听钩子，返回打包函数
            this.renderMain(options);

          result.push({ render /* arguments */ });
          return result;
        }
      );
    });
  }

  renderMain() {/*  */}

  renderChunk() {/*  */}
}

The core logic here is that compilation releases the bundle packaging requirements in the form of a renderManifest JavascriptModulesPlugin monitors this hook and calls different packaging functions according to the content characteristics of the chunk.

The above is only for Webpack 5. In Webpack 4, the packaging logic is concentrated in MainTemplate .

JavascriptModulesPlugin built-in packaging functions of 060deb97f15754 are:

• renderMain : used when packaging the main chunk
• renderChunk : package sub-chunks, such as asynchronous module chunks

The logic of the implementation of the two packing functions is renderMain , and the modules are spliced in order. The implementation of 060deb97f1584d is briefly introduced below.

2.3 renderMain function

renderMain function involves multiple scene judgments. The original code is very long and convoluted. I have taken a few key steps:

class JavascriptModulesPlugin {
  renderMain(renderContext, hooks, compilation) {
    const { chunk, chunkGraph, runtimeTemplate } = renderContext;

    const source = new ConcatSource();
    // ...
    // 1. 先计算出 bundle CMD 核心代码，包含：
    //      - "var __webpack_module_cache__ = {};" 语句
    //      - "__webpack_require__" 函数
    const bootstrap = this.renderBootstrap(renderContext, hooks);

    // 2. 计算出当前 chunk 下，除 entry 外其它模块的代码
    const chunkModules = Template.renderChunkModules(
      renderContext,
      inlinedModules
        ? allModules.filter((m) => !inlinedModules.has(m))
        : allModules,
      (module) =>
        this.renderModule(
          module,
          renderContext,
          hooks,
          allStrict ? "strict" : true
        ),
      prefix
    );

    // 3. 计算出运行时模块代码
    const runtimeModules =
      renderContext.chunkGraph.getChunkRuntimeModulesInOrder(chunk);

    // 4. 重点来了，开始拼接 bundle
    // 4.1 首先，合并核心 CMD 实现，即上述 bootstrap 代码
    const beforeStartup = Template.asString(bootstrap.beforeStartup) + "\n";
    source.add(
      new PrefixSource(
        prefix,
        useSourceMap
          ? new OriginalSource(beforeStartup, "webpack/before-startup")
          : new RawSource(beforeStartup)
      )
    );

    // 4.2 合并 runtime 模块代码
    if (runtimeModules.length > 0) {
      for (const module of runtimeModules) {
        compilation.codeGeneratedModules.add(module);
      }
    }
    // 4.3 合并除 entry 外其它模块代码
    for (const m of chunkModules) {
      const renderedModule = this.renderModule(m, renderContext, hooks, false);
      source.add(renderedModule)
    }

    // 4.4 合并 entry 模块代码
    if (
      hasEntryModules &&
      runtimeRequirements.has(RuntimeGlobals.returnExportsFromRuntime)
    ) {
      source.add(`${prefix}return __webpack_exports__;\n`);
    }

    return source;
  }
}

The core logic is:

• First calculate the bundle CMD code, that is, __webpack_require__ function
• Calculate the code of other modules except entry chunkModules
• Calculate the runtime module code

• Start the merge operation, the sub-steps are:

  • Combine CMD codes
  • Merged runtime module code
  • Traverse chunkModules variables and merge other module codes except entry
  • Merging entry module code
• Return result

Summary: First calculate the product form of the different components, then splice and package in order, and output the merged version.

At this point, Webpack completes the translation and packaging process of the bundle, and then calls compilation.emitAsset to output the product to fs according to the context, and the single compilation and packaging process of Webpack is over.

Three, summary

This article goes deep into the Webpack source code and discusses in detail the second half of the packaging process-the implementation logic from chunk graph generation to the final output product, focusing on:

• First traverse all modules in the chunk, perform translation operations for each module, and produce module-level products
• According to the type of chunk, choose different structural frameworks, assemble the module products one by one in order, and package them into the final bundle

To recap, we:

• In the "160deb97f15b01 [Summary of Ten Thousand Words] "A thorough understanding of the core principles of ", a high-level discussion of the Webpack workflow from front to back, to help readers have a more abstract understanding of the implementation principles of Webpack;
• In " [Source code interpretation] Webpack plug-in architecture in-depth explanation of ", the implementation principle of the Webpack plug-in mechanism is introduced in detail to help readers understand the Webpack architecture and hook design;
• In " bit difficult webpack knowledge point: Dependency Graph in-depth analysis " introduced in detail the module dependency graph concept, to help readers understand the dependency discovery and dependency construction process in Webpack
• In " bit difficult knowledge points: Webpack Chunk subcontracting rules detailed " detailed introduction of the basic logic and implementation methods of chunk subcontracting, to help readers understand the principle of product fragmentation
• In " Webpack Principle Series Six: Thorough Understanding of Webpack Runtime ", it introduces in detail the origin and function of other runtime codes in the bundle, except for business modules, to help readers understand the operational logic of the product
• Finally, go to the module translation and merge packaging logic introduced in this article

At this point, the main process of Webpack compilation and packaging has been well connected. I believe that readers will follow the context of this article, carefully study the source code and study patiently, and they must have a deep understanding of front-end packaging and engineering, and encourage each other.