Take you hand in hand into the compilation world of Babel

Author: BoBoooooo

foreword

When it comes to Babel, AST is bound to be inseparable. The knowledge about AST is actually very important, but because it involves the code compilation stage, most of the cases are built-in related processing by each framework, so as a developer (user), this process is often ignored. I hope that through this article, I will bring students into AST and use AST to exert more imagination.

AST overview

Presumably everyone has always heard the concept of AST, so what is AST?

The full name of AST is Abstract Syntax Tree , which is an abstract syntax tree in Chinese, which converts the code we write into a tree structure that can be recognized by machines. It itself is composed of a bunch of nodes (Node), each node represents a structure in the source code. Different structures are distinguished by type (Type). Common types are: Identifier (identifier), Expression (expression), VariableDeclaration (variable definition), FunctionDeclaration (function definition) and so on.

AST structure

With the development of JavaScript, in order to unify the syntax expression of the ECMAScript standard. The community derived ESTree Spec , which is a grammatical expression standard currently followed by the community.

ESTree provides common node types such as Identifier、Literal .

Node type

Types of	illustrate
File	file (top-level node contains Program)
Program	The entire program node (including the body attribute representing the program body)
Directive	Directives (eg "use strict")
Comment	code comments
Statement	Statement (a statement that can be executed independently)
Literal	Literals (primitive data types, complex data types, and other value types)
Identifier	Identifiers (variable names, property names, function names, parameter names, etc.)
Declaration	Declarations (variable declarations, function declarations, Import, Export declarations, etc.)
Specifier	Keywords (ImportSpecifier, ImportDefaultSpecifier, ImportNamespaceSpecifier, ExportSpecifier)
Expression	expression

public property

Types of	illustrate
type	Type of AST node
start	Record the starting subscript of the node code string
end	Record the end subscript of the node code string
loc	Contains line and column attributes, which record the row and column numbers of the start and end respectively
leadingComments	start note
innerComments	middle note
trailingComments	end note
extra	extra information

AST example

Some students may ask, do we need to remember so many types? Actually not, we can use the following two tools to query the AST structure.

Combined with an example, I will take you to a quick understanding of the AST structure.

function test(args) {
  const a = 1;
  console.log(args);
}

The above code declares a function named test with a formal parameter args .

In the function body:

A const type variable a is declared with a value of 1
A console.log statement was executed

Paste the above code into AST Explorer , the result is as shown:

Next, we continue to analyze the internal structure, taking const a = 1 as an example:

The variable declaration corresponds to the node of type VariableDeclaration in the AST. This node contains two required attributes, kind and declarations , which represent the declared variable type and variable content respectively.

Careful students may find that declarations is an array. Why is this? Because the variable declaration itself supports the writing method of const a=1,b=2 , it needs to support multiple VariableDeclarator , so it is an array here.

The node whose type is VariableDeclarator represents the declaration statement a=1 , which contains id and init attributes.

id is Identifier , and the value of name corresponds to the variable name.

init the initial value, including type , value attributes. Indicates the initial value type and initial value, respectively. Here type is NumberLiteral , indicating that the initial value type is number type .

`Babel overview`

Babel is a JavaScript compiler. In the actual development process, Babel is usually used to complete related AST operations.

`Babel Workflow`

`Babel AST`

The AST generated by Babel after parsing the code is based on ESTree with slight modifications.

The official text is as follows:

The Babel parser generates AST according to Babel AST format. It is based on ESTree spec with the following deviations:
Literal token is replaced with StringLiteral, NumericLiteral, BigIntLiteral, BooleanLiteral, NullLiteral, RegExpLiteral
Property token is replaced with ObjectProperty and ObjectMethod
MethodDefinition is replaced with ClassMethod
Program and BlockStatement contain additional directives field with Directive and DirectiveLiteral
ClassMethod, ObjectProperty, and ObjectMethod value property's properties in FunctionExpression is coerced/brought into the main method node.
ChainExpression is replaced with OptionalMemberExpression and OptionalCallExpression
ImportExpression is replaced with a CallExpression whose callee is an Import node.

`Babel core package`

Toolkit	illustrate
`@babel/core`	The core package of Babel transcoding, including the entire babel workflow (integrated @babel/types)
`@babel/parser`	Parser, which parses code into AST
`@babel/traverse`	Tools for traversing/modifying the AST
`@babel/generator`	A generator that restores the AST to code
`@babel/types`	Contains methods for manually building the AST and checking the type of AST nodes
`@babel/template`	Convert string snippets to AST nodes

npm i @babel/parser @babel/traverse @babel/types @babel/generator @babel/template -D

`Babel plugin`

Babel plugins are roughly divided into two types: syntax plugins and transformation plugins. The syntax plugin acts on @babel/parser and is responsible for parsing the code into an abstract syntax tree (AST) (the official syntax plugin starts with babel-plugin-syntax); the conversion plugin acts on @babel/core and is responsible for transforming the form of the AST. Most of the time we are writing transform plugins.

Babel works with plugins. The plugin is equivalent to the instruction to tell Babel what to do. Without the plugin, Babel will output the code as-is.

The Babel plugin is essentially writing various visitor to access nodes on the AST, and perform traverse . When encountering a node of the corresponding type, visitor will make corresponding processing, thereby transforming the the final code.

export default function (babel) {
  // 即@babel/types，用于生成AST节点
  const { types: t } = babel;

  return {
    name: "ast-transform", // not required
    visitor: {
      Identifier(path) {
        path.node.name = path.node.name.split("").reverse().join("");
      },
    },
  };
}

This is a piece of transform template code on AST Explorer . The effect of the above code is that reverses the node names of all Identifier types of the input code.

In fact, writing a Babel plugin is very simple. All we have to do is pass back a visitor object defining a function with the name Node Type . This function receives two parameters: path , state .

Where path (path) provides a way for to access/manipulate AST nodes. path itself represents object connected between two nodes. For example, path.node can access the current node, path.parent can access the parent node, etc. path.remove() can remove the current node. The specific API is shown in the figure below. other visible handlebook。

`Babel Types`

The Babel Types module is a Lodash-style library of tools for AST nodes that contains methods for constructing, validating, and transforming AST nodes.

`Type judgment`

Babel Types provides a method for judging node types, and each type of node has a corresponding judging method. For more see babel-types API .

import * as types from "@babel/types";

// 是否为标识符类型节点
if (types.isIdentifier(node)) {
  // ...
}

// 是否为数字字面量节点
if (types.isNumberLiteral(node)) {
  // ...
}

// 是否为表达式语句节点
if (types.isExpressionStatement(node)) {
  // ...
}

`create node`

Babel Types also provides methods for creating various types of nodes, see the following examples for details.

Note: The AST node generated by Babel Types needs to be converted by @babel/generator to get the corresponding code.

import * as types from "@babel/types";
import generator from "@babel/generator";

const log = (node: types.Node) => {
  console.log(generator(node).code);
};

log(types.stringLiteral("Hello World")); // output: Hello World

`basic data type`

types.stringLiteral("Hello World"); // string
types.numericLiteral(100); // number
types.booleanLiteral(true); // boolean
types.nullLiteral(); // null
types.identifier(); // undefined
types.regExpLiteral("\\.js?$", "g"); // 正则

"Hello World"
100
true
null
undefined
/\.js?$/g

`complex data types`

array

types.arrayExpression([
  types.stringLiteral("Hello World"),
  types.numericLiteral(100),
  types.booleanLiteral(true),
  types.regExpLiteral("\\.js?$", "g"),
]);

["Hello World", 100, true, /\.js?$/g];

object

types.objectExpression([
  types.objectProperty(
    types.identifier("key"),
    types.stringLiteral("HelloWorld")
  ),
  types.objectProperty(
    // 字符串类型 key
    types.stringLiteral("str"),
    types.arrayExpression([])
  ),
  types.objectProperty(
    types.memberExpression(
      types.identifier("obj"),
      types.identifier("propName")
    ),
    types.booleanLiteral(false),
    // 计算值 key
    true
  ),
]);

{
  key: "HelloWorld",
  "str": [],
  [obj.propName]: false
}

`JSX node`

Creating a JSX AST node is slightly different from creating a data type node, and a diagram is compiled here.

JSXElement

types.jsxElement(
  types.jsxOpeningElement(types.jsxIdentifier("Button"), []),
  types.jsxClosingElement(types.jsxIdentifier("Button")),
  [types.jsxExpressionContainer(types.identifier("props.name"))]
);

<Button>{props.name}</Button>

JSXFragment

types.jsxFragment(types.jsxOpeningFragment(), types.jsxClosingFragment(), [
  types.jsxElement(
    types.jsxOpeningElement(types.jsxIdentifier("Button"), []),
    types.jsxClosingElement(types.jsxIdentifier("Button")),
    [types.jsxExpressionContainer(types.identifier("props.name"))]
  ),
  types.jsxElement(
    types.jsxOpeningElement(types.jsxIdentifier("Button"), []),
    types.jsxClosingElement(types.jsxIdentifier("Button")),
    [types.jsxExpressionContainer(types.identifier("props.age"))]
  ),
]);

<>
  <Button>{props.name}</Button>
  <Button>{props.age}</Button>
</>

`statement`

Variable Declaration (variableDeclaration)

types.variableDeclaration("const", [
  types.variableDeclarator(types.identifier("a"), types.numericLiteral(1)),
]);

const a = 1;

Function Declaration (functionDeclaration)

types.functionDeclaration(
  types.identifier("test"),
  [types.identifier("params")],
  types.blockStatement([
    types.variableDeclaration("const", [
      types.variableDeclarator(
        types.identifier("a"),
        types.numericLiteral(1)
      ),
    ]),
    types.expressionStatement(
      types.callExpression(types.identifier("console.log"), [
        types.identifier("params"),
      ])
    ),
  ])
);

function test(params) {
  const a = 1;
  console.log(params);
}

`React functional components`

Based on the above content, let's have a little practice~

We need to generate button.js code via Babel Types. At first glance, don't know where to start?

// button.js
import React from "react";
import { Button } from "antd";

export default (props) => {
  const handleClick = (ev) => {
    console.log(ev);
  };
  return <Button onClick={handleClick}>{props.name}</Button>;
};

Tips: First use the AST Explorer website to observe the AST tree structure. Then write code layer by layer through Babel Types. Do more with less!

types.program([
  types.importDeclaration(
    [types.importDefaultSpecifier(types.identifier("React"))],
    types.stringLiteral("react")
  ),
  types.importDeclaration(
    [
      types.importSpecifier(
        types.identifier("Button"),
        types.identifier("Button")
      ),
    ],
    types.stringLiteral("antd")
  ),
  types.exportDefaultDeclaration(
    types.arrowFunctionExpression(
      [types.identifier("props")],
      types.blockStatement([
        types.variableDeclaration("const", [
          types.variableDeclarator(
            types.identifier("handleClick"),
            types.arrowFunctionExpression(
              [types.identifier("ev")],
              types.blockStatement([
                types.expressionStatement(
                  types.callExpression(types.identifier("console.log"), [
                    types.identifier("ev"),
                  ])
                ),
              ])
            )
          ),
        ]),
        types.returnStatement(
          types.jsxElement(
            types.jsxOpeningElement(types.jsxIdentifier("Button"), [
              types.jsxAttribute(
                types.jsxIdentifier("onClick"),
                types.jSXExpressionContainer(types.identifier("handleClick"))
              ),
            ]),
            types.jsxClosingElement(types.jsxIdentifier("Button")),
            [types.jsxExpressionContainer(types.identifier("props.name"))],
            false
          )
        ),
      ])
    )
  ),
]);

`Application scenarios`

AST itself is widely used, such as: Babel plugin (ES6 to ES5), compression code at build time, css preprocessor compilation, webpack plugin, etc. It can be said that it is everywhere.

As shown in the figure, it is not difficult to find that once it comes to compilation, or the processing of the code itself, it is closely related to AST. Some common applications are listed below, let's see how they are handled.

`code conversion`

// ES6 => ES5 let 转 var
export default function (babel) {
  const { types: t } = babel;

  return {
    name: "let-to-var",
    visitor: {
      VariableDeclaration(path) {
        if (path.node.kind === "let") {
          path.node.kind = "var";
        }
      },
    },
  };
}

`babel-plugin-import`

Under the CommonJS specification, this plugin is usually used when we need to introduce antd on demand.

The plugin does the following:

// 通过es规范，具名引入Button组件
import { Button } from "antd";
ReactDOM.render(<Button>xxxx</Button>);

// babel编译阶段转化为require实现按需引入
var _button = require("antd/lib/button");
ReactDOM.render(<_button>xxxx</_button>);

Simple analysis, core processing: replace the import statement with the corresponding require statement.

export default function (babel) {
  const { types: t } = babel;

  return {
    name: "import-to-require",
    visitor: {
      ImportDeclaration(path) {
        if (path.node.source.value === "antd") {
          // var _button = require("antd/lib/button");
          const _botton = t.variableDeclaration("var", [
            t.variableDeclarator(
              t.identifier("_button"),
              t.callExpression(t.identifier("require"), [
                t.stringLiteral("antd/lib/button"),
              ])
            ),
          ]);
          // 替换当前import语句
          path.replaceWith(_botton);
        }
      },
    },
  };
}

TIPS: The esm specification file is currently included in the antd package, which can be imported on demand by relying on webpack's native TreeShaking.

`LowCode Visual Coding`

At present, LowCode is still a popular front-end area. The current mainstream practices are roughly as follows.

Schema driven
The current mainstream practice is to describe the configuration of a form or table as a Schema. The visual designer is driven by Schema and combined with the drag-and-drop capability to quickly build.
AST drive
Compile and encode online through browsers such as CloudIDE and CodeSandbox . With the addition of a visual designer, visual coding is finally realized.

The general process is shown in the figure above. Since code modification is involved, the operation of AST is inseparable, so the ability of babel can be used again.

Suppose the initial code of the designer is as follows:

import React from "react";

export default () => {
  return <Container></Container>;
};

At this point, we dragged a Button to the designer. According to the process in the above figure, the core AST modification process is as follows:

Added import statement import { Button } from "antd";
Insert <Button></Button> into <Container></Container>

Not much to say, go directly to the code:

import traverse from "@babel/traverse";
import generator from "@babel/generator";
import * as parser from "@babel/parser";
import * as t from "@babel/types";

// 源代码
const code = `
  import React from "react";

  export default () => {
    return <Container></Container>;
  };
`;

const ast = parser.parse(code, {
  sourceType: "module",
  plugins: ["jsx"],
});

traverse(ast, {
  // 1. 程序顶层 新增import语句
  Program(path) {
    path.node.body.unshift(
      t.importDeclaration(
        // importSpecifier表示具名导入，相应的匿名导入为ImportDefaultSpecifier
        // 具名导入对应代码为 import { Button as Button } from 'antd'
        // 如果相同会自动合并为 import { Button } from 'antd'
        [t.importSpecifier(t.identifier("Button"), t.identifier("Button"))],
        t.stringLiteral("antd")
      )
    );
  },
  // 访问JSX节点，插入Button
  JSXElement(path) {
    if (path.node.openingElement.name.name === "Container") {
      path.node.children.push(
        t.jsxElement(
          t.jsxOpeningElement(t.jsxIdentifier("Button"), []),
          t.jsxClosingElement(t.jsxIdentifier("Button")),
          [t.jsxText("按钮")],
          false
        )
      );
    }
  },
});

const newCode = generator(ast).code;
console.log(newCode);

The result is as follows:

import { Button } from "antd";
import React from "react";
export default () => {
  return (
    <Container>
      <Button>按钮</Button>
    </Container>
  );
};

`ESLint`

Custom eslint-rule, in essence, also accesses AST nodes, is it similar to the way the Babel plugin is written?

module.exports.rules = {
  "var-length": (context) => ({
    VariableDeclarator: (node) => {
      if (node.id.name.length <= 2) {
        context.report(node, "变量名长度需要大于2");
      }
    },
  }),
};

`Code2Code`

Take Vue To React as an example, the general process is similar to ES6 => ES5 , vue-template-compiler Vue AST => Convert to React AST => Output React code by compiling .

Interested students can refer to vue-to-react

Other multi-terminal frameworks: one code => multi-terminal, the general idea is the same.

`Summarize`

In actual development, the situations encountered are often more complicated. It is recommended that you document more, observe more, and feel attentively~

`Reference article`

babel-handlebook
@babel/types
[First visit to AST by making Babel-plugin ](https://blog.techbridge.cc/2018/09/22/visit-ast-with-babel-plugin/)
[@babel/types deep application ](https://juejin.cn/post/6984945589859385358#heading-7)

This article is published from the NetEase Cloud Music technical team, and any form of reprinting of the article is prohibited without authorization. We recruit various technical positions all year round. If you are ready to change jobs and happen to like cloud music, then join us at grp.music-fe(at)corp.netease.com!