Read the Babel compilation process in one article, and no longer be afraid of the interviewer’s difficulties

Preface

Babel is a powerful js compiler. With Babel, we can use the new features of js without regard to browser compatibility issues. Not only that, based on the babel system, we can modify some grammars, optimize some grammars, and even create new grammars through plug-in methods.

So, how is such a powerful and flexible feature realized? Let's start from the beginning and understand the compilation process of Babel.

Process

Babel generation configuration

package.json

Project configuration file

"devDependencies": {
    "@babel/cli": "7.10.5",
    "@babel/core": "7.11.1",
    "@babel/plugin-proposal-class-properties": "7.10.4",
    "@babel/plugin-proposal-decorators": "7.10.5",
    "@babel/plugin-proposal-do-expressions": "7.10.4",
    "@babel/plugin-proposal-object-rest-spread": "7.11.0",
    "@babel/plugin-syntax-dynamic-import": "7.8.3",
    "@babel/plugin-transform-react-jsx": "7.12.17",
    "@babel/plugin-transform-runtime": "7.11.0",
    "@babel/preset-env": "7.11.0",
    "@babel/preset-react": "7.12.13",
    "@babel/preset-typescript": "7.12.17",
      .......
}

We often come into contact with have babel , babel-Loader , @ babel / Core , @ babel / PRESET-env , @ babel / polyfill , and @ babel / plugin-the Transform-Runtime , What are these all for?

1、babel：

Babel's official website has a very clear definition:

Babel is a toolchain that is mainly used to convert ECMAScript 2015+ code into a backward compatible version of JavaScript code in an old browser or environment:
Conversion grammar
Polyfill realizes the missing features in the target environment (via @babel/polyfill )
Source code conversion (codemods)
More!

We can see that babel is a tool chain that contains many functions such as syntax conversion. Through the use of this tool chain, lower-version browsers can be compatible with the latest javascript syntax.

It should be noted that babel is also an installable package, and it is used as a shorthand for loader in the webpack 1.x configuration. Such as:

{
  test: /\.js$/,
  loader: 'babel',
}

But this method is no longer supported after webpack 2.x and I get an error message:

The node API forbabelhas been moved tobabel-core

At this time, delete the babel package, install babel-loader, and specify loader:'babel-loader'

2、@babel/core：

@babel/core is the core of the entire babel. It is responsible for scheduling the various components of babel for code compilation, and is the organizer and scheduler of the entire behavior.

The transform method will call transformFileRunner to compile the file. First, the loadConfig method will generate a complete configuration. Then read the code in the file and compile it according to this configuration.

const transformFileRunner = gensync<[string, ?InputOptions], FileResult | null>(
  function* (filename, opts) {
    const options = { ...opts, filename };

    const config: ResolvedConfig | null = yield* loadConfig(options);
    if (config === null) return null;

    const code = yield* fs.readFile(filename, "utf8");
    return yield* run(config, code);
  },
);

3、@babel/preset-env：

This is a preset plug-in collection, including a set of related plug-ins, Bable uses various plug-ins to guide how to perform code conversion. This plugin contains all the translation rules for es6 to es5

The official website of babel explains this as follows:

Transformations come in the form of plugins, which are small JavaScript programs that instruct Babel on how to carry out transformations to the code. You can even write your own plugins to apply any transformations you want to your code. To transform ES2015+ syntax into ES5 we can rely on official plugins like@babel/plugin-transform-arrow-functions

Roughly, the grammar conversion from es6 to es5 is implemented in the form of plug-ins, which can be your own plug-ins or officially provided plug-ins such as arrow function conversion plug-ins @babel/plugin-transform-arrow-functions.

From this we can see that we can list the relevant plug-ins one by one for which new syntax we need to convert, but this is actually very complicated, because we often need to determine which plug-ins need to be introduced according to the different versions of compatible browsers In order to solve this problem, babel provides us with a preset plugin group, namely @babel/preset-env, which can be flexibly decided which plugins to provide according to the option parameters

{
    "presets":["es2015","react","stage-1"],
    "plugins": [["transform-runtime"],["import", {
        "libraryName": "cheui-react",
        "libraryDirectory": "lib/components",
        "camel2DashComponentName": true // default: true
    }]]
  }

Three key parameters:

1、targets:

Describes the environments you support/target for your project.

Simply put, this parameter determines the environment that our project needs to adapt to. For example, you can declare the browser version that it adapts, so that Babel will automatically introduce the required polyfill based on the browser's support.

2、useBuiltIns:

"usage" | "entry" | false, defaults to false

This option configures how @babel/preset-env handles polyfills.

This parameter determines how preset-env handles polyfills.

false`: 这种方式下，不会引入 polyfills，你需要人为在入口文件处`import '@babel/polyfill';

But the above method @babel@7.4 . Instead, import the following code at the entry file by itself.

import 'core-js/stable';
import 'regenerator-runtime/runtime';
// your code

not recommended to use false , this will put all polyfills in, resulting in a huge package

usage:

We do not need to import the corresponding polyfills related libraries at the entry file of the project. Babel will inject relevant polyfills according to the usage of user code and targets.

entry:

We import the corresponding polyfills related libraries at the entry file of the project, for example

import 'core-js/stable';
import 'regenerator-runtime/runtime';
// your code

At this time, babel will introduce all the polyfills needed into your entry file according to the current target description (note that it is all, regardless of whether you use advanced APIs)

3、corejs:

String or { version: string, proposals: boolean }, defaults to "2.0".

corejs

Note that corejs is not a special concept, but the browser polyfill is managed by it.

for example

javascript const one = Symbol('one');

==Babel==>

"use strict";

require("core-js/modules/es.symbol.js");

require("core-js/modules/es.symbol.description.js");

require("core-js/modules/es.object.to-string.js");

var one = Symbol('one');

Some people may not be very clear here, what is the difference between 2 and 3, you can look at the official document core-js@3, babel and a look into the future

Simply speaking, corejs-2 will not be maintained. The polyfills of all new browser features will be maintained on corejs-3.

summarizes: use corejs-3, turn on proposals: true , and if the proposals are true, we can use the API of the proposal phase.

4、@babel/polyfill：

@babel/preset-env only provides rules for grammar conversion, but it can’t make up for some new features that browsers lack, such as some built-in methods and objects, such as Promise, Array.from, etc. At this time, polyfill is needed. Make js shims to make up for these new features that are missing in lower version browsers.

What we need to pay attention to is that the volume of the polyfill is very large. If we don't make special instructions, it will shim all the new features of es6 that are missing in your target browser. However, the conversion of the part of the function that we did not use is actually meaningless, resulting in a needless increase in the packaged volume. So usually, we will configure "useBuiltIns": "usage" in the presets option. On the one hand, we only shim the new functions used. On the other hand, we don’t need to separately import import ' @babel/polyfill ', it will be injected automatically where it is used.

5、babel-loader:

The above @babel/core, @babel/preset-env and @babel/polyfill are actually doing es6 syntax conversion and making up for the missing functions, but when we are using webpack to package js, webpack does not know how to call it These rules to compile js. At this time, babel-loader is needed, which acts as an intermediate bridge to tell webpack how to handle js by calling the api in babel/core.

6、@babel/plugin-transform-runtime：

The polyfill shim is to mount the missing function of the target browser on the global variable. Therefore, when developing class libraries, third-party modules or component libraries, you can no longer use babel-polyfill, otherwise it may cause global pollution. Transform-runtime should be used. The transform of transform-runtime is non-invasive, that is, it will not pollute your original method. When it encounters a method that needs to be converted, it will have a different name, otherwise it will directly affect the business code that uses the library.

.babelrc

If we don't configure anything, there will be no changes in the packaged file. You need to configure babel in the babelrc file as follows. Then pack. We will analyze the mechanism of this configuration later.

{
    "presets": ["@babel/preset-env"]
}

@babel/cli parses the command line, but if only the parameters in the command line are used, babel cannot compile, and it lacks some key parameters, that is, the plug-in information configured in the .babelrc file.

@babel/core Before performing the transformFile operation, the first step is to read the configuration in the .babelrc file.

The process is like this. Babel will first determine whether there is a configuration file (-config-file) specified in the command line, and then parse it. If not, Babel will look for the default configuration file in the current root directory. The default file name is defined as follows. The priority is from top to bottom.

babel-main\packages\babel-core\src\config\files\configuration.js

const RELATIVE_CONFIG_FILENAMES = [
  ".babelrc",
  ".babelrc.js",
  ".babelrc.cjs",
  ".babelrc.mjs",
  ".babelrc.json",
];

In the .babelrc file, we often configure plugins and presets. The plugin is what really does in babel. The conversion of the code depends on it. However, with the increase of plugins, how to manage these plugins is also a challenge. Therefore, Babel put some plugins together and called it preset.

For plugins and presets in babelrc, babel converts each item into a ConfigItem. Presets is an array of ConfigItem, and plugins is also an array of ConfigItem.

Assuming the following .babelrc file, such json configuration will be generated.

{
    "presets": ["@babel/preset-env"],
    "plugins": ["@babel/plugin-proposal-class-properties"]
}
plugins: [
     ConfigItem {
      value: [Function],
      options: undefined,
      dirname: 'babel\\babel-demo',
      name: undefined,
      file: {
        request: '@babel/plugin-proposal-class-properties',
        resolved: 'babel\\babel-demo\\node_modules\\@babel\\plugin-proposal-class-properties\\lib\\index.js'
      }
    }
  ],
  presets: [
    ConfigItem {
      value: [Function],
      options: undefined,
      dirname: 'babel\\babel-demo',
      name: undefined,
      file: {
        request: '@babel/preset-env',
        resolved: 'babel\\babel-demo\\node_modules\\@babel\\preset-env\\lib\\index.js'
      }
    }
  ]

For plugins, babel will load the contents in sequence and parse out the pre, visitor and other objects defined in the plugin. Since the presets will contain a pair of plugins, and even a new preset, babel needs to parse the content of the preset and parse out the plugins contained in it. Taking @babel/preset-env as an example, babel will parse 40 of the plugins, and then re-parse the plugins in presets.

There is a very interesting point here, that is, for the parsed plug-in list, the processing method is to use unshift to insert it into the head of a list.

if (plugins.length > 0) {
  pass.unshift(...plugins);
}

This is actually because the loading order of presets is different from the general understanding. For example, presets are written as ["es2015", "stage-0"]. Since stage-x is some proposal of Javascript grammar, this part may rely on ES6 grammar and analysis. At the time, you need to parse the new grammar into ES6 first, and then parse ES6 into ES5. This is why unshift is used. The plugins in the new preset will be executed first.

Of course, no matter what the order of presets is, the plugins in the plugins we define are always the highest priority. The reason is that the plug-in in plugins uses unshift to insert the head of the column after the presets are processed.

The final generated configuration contains two pieces of options and passes. In most cases, the presets in options is an empty array, plugins store the collection of plugins, and the content in passes is the same as options.plugins.

{
  options: {
    babelrc: false,
    caller: {name: "@babel/cli"},
    cloneInputAst: true,
    configFile: false,
    envName: "development",
    filename: "babel-demo\src\index.js",
    plugins: Array(41),
    presets: []
  }
  passes: [Array(41)]
}

Babel executes compilation

Process

Let's take a look at the main code of run

export function* run(
  config: ResolvedConfig,
  code: string,
  ast: ?(BabelNodeFile | BabelNodeProgram),
): Handler<FileResult> {

  const file = yield* normalizeFile(
    config.passes,
    normalizeOptions(config),
    code,
    ast,
  );

  const opts = file.opts;
  try {
    yield* transformFile(file, config.passes);
  } catch (e) {
    ...
  }

  let outputCode, outputMap;
  try {
    if (opts.code !== false) {
      ({ outputCode, outputMap } = generateCode(config.passes, file));
    }
  } catch (e) {
    ...
  }

  return {
    metadata: file.metadata,
    options: opts,
    ast: opts.ast === true ? file.ast : null,
    code: outputCode === undefined ? null : outputCode,
    map: outputMap === undefined ? null : outputMap,
    sourceType: file.ast.program.sourceType,
  };
}

1. The first is to execute the normalizeFile method, the function of this method is to convert the code into an abstract syntax tree (AST);
2. Then execute the transformFile method, which includes our plug-in list. What this step does is to modify the content of the AST according to the plug-in;
3. Finally, execute the generateCode method to convert the modified AST into code.

The entire compilation process is quite clear. Simply put, it is parse, transform, and generate. We look at each process in detail.

Parse

Before understanding the parsing process, we must first understand the abstract syntax tree (AST), which expresses the grammatical structure of the programming language in a tree-like form. Each node on the tree represents a structure in the source code. Different languages have different rules for generating AST. In JS, AST is a JSON string used to describe the code.

For a simple example, for a simple constant declaration, the generated AST code looks like this.

const a = 1
{
  "type": "Program",
  "start": 0,
  "end": 11,
  "body": [
    {
      "type": "VariableDeclaration",
      "start": 0,
      "end": 11,
      "declarations": [
        {
          "type": "VariableDeclarator",
          "start": 6,
          "end": 11,
          "id": {
            "type": "Identifier",
            "start": 6,
            "end": 7,
            "name": "a"
          },
          "init": {
            "type": "Literal",
            "start": 10,
            "end": 11,
            "value": 1,
            "raw": "1"
          }
        }
      ],
      "kind": "const"
    }
  ],
  "sourceType": "module"
}

Back to the normalizeFile method, the parser method is called in this method.

export default function* normalizeFile(
  pluginPasses: PluginPasses,
  options: Object,
  code: string,
  ast: ?(BabelNodeFile | BabelNodeProgram),
): Handler<File> {
  ...
  ast = yield* parser(pluginPasses, options, code);
  ...
}

The parser will traverse all plug-ins to see which plug-in defines the parserOverride method. In order to facilitate understanding, let's skip this part first, and look at the parse method first. The parse method is a method provided by @babel/parser to convert JS code into an AST.

Under normal circumstances, the rules in @babel/parser can complete the AST conversion very well, but if we need to customize the grammar, or modify/extend these rules, @babel/parser is not enough. Babel thought of a way, that is, you can write a parser yourself, and then specify this parser as the compiler of babel through a plug-in.

import { parse } from "@babel/parser";

export default function* parser(
  pluginPasses: PluginPasses,
  { parserOpts, highlightCode = true, filename = "unknown" }: Object,
  code: string,
): Handler<ParseResult> {
  try {
    const results = [];
    for (const plugins of pluginPasses) {
      for (const plugin of plugins) {
        const { parserOverride } = plugin;
        if (parserOverride) {
          const ast = parserOverride(code, parserOpts, parse);

          if (ast !== undefined) results.push(ast);
        }
      }
    }

    if (results.length === 0) {

      return parse(code, parserOpts);

    } else if (results.length === 1) {
      yield* []; // If we want to allow async parsers

      ...

      return results[0];
    }
    throw new Error("More than one plugin attempted to override parsing.");
  } catch (err) {
    ...
  }
}

Now looking back at the previous loop, it is easy to understand, traversing the plug-in, if the parserOverride method is defined in the plug-in, it is considered that the user has specified a custom compiler. It is known from the code that there can only be one compiler defined by the plug-in, otherwise babel will not know which compiler to execute.

The following is an example of a custom compiler plugin.

const parse = require("custom-fork-of-babel-parser-on-npm-here");

module.exports = {
  plugins: [{
    parserOverride(code, opts) {
      return parse(code, opts);
    },
  }]
}

The process of converting JS to AST relies on @babel/parser. Users can already write a parser by plug-in to override the default. The process of @babel/parser is quite complicated. We will analyze it separately later, as long as we know that it converts JS code into AST.

Transform

AST needs to make some changes according to the content of the plug-in. Let's first look at what the next plug-in looks like. As shown below, the Babel plug-in returns a function, the input parameter is a babel object, and an Object is returned. Among them, pre and post are triggered when entering/leaving the AST respectively, so they are generally used to initialize/delete objects respectively. The visitor (visitor) defines a method for obtaining specific nodes in a tree structure.

module.exports = (babel) => {
  return {
    pre(path) {
      this.runtimeData = {}
    },
    visitor: {},
    post(path) {
      delete this.runtimeData
    }
  }
}

After understanding the structure of the plug-in, it is easier to look at the transformFile method. First, babel adds a loadBlockHoistPlugin plug-in to the plug-in collection, which is used for sorting, so there is no need to go into it. Then it is to execute the pre method of the plug-in. After all the pre methods of the plug-in are executed, execute the method in the visitor (not a simple execution method, but executed when the corresponding node or attribute is encountered according to the visitor pattern. For specific rules, see Babel Plugin Manual ). For optimization, babel combines multiple visitors into one, uses traverse to traverse AST nodes, and executes the plugin during the traversal process. Finally, the post method of the plugin is executed.

import traverse from "@babel/traverse";

function* transformFile(file: File, pluginPasses: PluginPasses): Handler<void> {
  for (const pluginPairs of pluginPasses) {
    const passPairs = [];
    const passes = [];
    const visitors = [];

    for (const plugin of pluginPairs.concat([loadBlockHoistPlugin()])) {
      const pass = new PluginPass(file, plugin.key, plugin.options);

      passPairs.push([plugin, pass]);
      passes.push(pass);
      visitors.push(plugin.visitor);
    }

    for (const [plugin, pass] of passPairs) {
      const fn = plugin.pre;
      if (fn) {
        const result = fn.call(pass, file);

        yield* [];
        ...
      }
    }

    // merge all plugin visitors into a single visitor
    const visitor = traverse.visitors.merge(
      visitors,
      passes,
      file.opts.wrapPluginVisitorMethod,
    );

    traverse(file.ast, visitor, file.scope);

    for (const [plugin, pass] of passPairs) {
      const fn = plugin.post;
      if (fn) {
        const result = fn.call(pass, file);

        yield* [];
        ...
      }
    }
  }
}

The core of this phase is the plug-in. The plug-in uses the visitor pattern to define how to operate after encountering a specific node. Babel puts methods such as traversing the AST tree and adding, deleting and modifying nodes in the @babel/traverse package.

Generate

After the AST has been converted, the code needs to be regenerated from the AST.

@babel/generator provides a default generate method. If you need to customize it, you can customize one through the generatorOverride method of the plugin. This method corresponds to the first stage of parserOverride. After the target code is generated, sourceMap-related code will also be generated at the same time.

import generate from "@babel/generator";

export default function generateCode(
  pluginPasses: PluginPasses,
  file: File,
): {
  outputCode: string,
  outputMap: SourceMap | null,
} {
  const { opts, ast, code, inputMap } = file;

  const results = [];
  for (const plugins of pluginPasses) {
    for (const plugin of plugins) {
      const { generatorOverride } = plugin;
      if (generatorOverride) {
        const result = generatorOverride(
          ast,
          opts.generatorOpts,
          code,
          generate,
        );

        if (result !== undefined) results.push(result);
      }
    }
  }

  let result;
  if (results.length === 0) {
    result = generate(ast, opts.generatorOpts, code);
  } else if (results.length === 1) {
    result = results[0];
    ...
  } else {
    throw new Error("More than one plugin attempted to override codegen.");
  }

  let { code: outputCode, map: outputMap } = result;

  if (outputMap && inputMap) {
    outputMap = mergeSourceMap(inputMap.toObject(), outputMap);
  }

  if (opts.sourceMaps === "inline" || opts.sourceMaps === "both") {
    outputCode += "\n" + convertSourceMap.fromObject(outputMap).toComment();
  }

  if (opts.sourceMaps === "inline") {
    outputMap = null;
  }

  return { outputCode, outputMap };
}