The principle of webpack packaging, handwriting a own bundler

Packaging tools such as webpack can help us package the code organized by esModule into a js file and run it in the browser. Realize the modularization of front-end projects, while optimizing the number of requests, file size, etc.

Not much to say, let's implement a similar bundler ourselves, package the modular front-end code, and output the js file that can be run in the browser.

Ready to work

Let’s first take a look at how the project we want to deal with is organized. Let’s put a src folder with index.js, hello.js, and word.js. The contents of each file are as follows

//index.js

import hello from "./hello.js"
console.log(hello)

//hello.js

import word from './word.js'
export default `hello ${word}`

//word.js

 const word = "word";
 export default word;

What I want to do is also very simple. It is to use esModule to finally assemble a console.log('hello word') in index.js, execute this js in the browser, and print a'hello' on the console word'.

Then we create a bundler.js at the same level of the src folder to help us package the code and enter the executable js.

Parse the entry file

We know that webpack uses an entry to enter the entry of the file to be packaged. Similarly, we also hope to tell our bundler which file to be packaged as the entry by entering the file access address.
Let's look at the code first:

const fs = require('fs')
const path = require('path')
const paser = require('@babel/parser')
const traverse = require('@babel/traverse').default
const { transformFromAst } = require('@babel/core')
const moduleAnalyser = (filename) => {
    const content = fs.readFileSync(filename, 'utf-8');    //{1}
    const ast = paser.parse(content,{            //{2}
        sourceType: 'module'
    })
    const dependencies = {};
    traverse(ast, {                    //{3}
        ImportDeclaration({node}){
            const dirname = path.dirname(filename);
            const newFile = './' + path.join(dirname, node.source.value)
            dependencies[node.source.value] = newFile
        }
    })
    const { code } = transformFromAst(ast, null, {    //{4}
        presets: ["@babel/preset-env"]
    })
    return {
        filename,
        dependencies,
        code
    }
}

1. File reading

We define a moduleAnalyser method to analyze the module. Since we want to analyze the file, we need to use the node's fs module to read the file in. So at {1}, we read the file in.

2. Generate abstract syntax tree

After getting the content of the file, we need to parse it. It happens that @babel/parser provided by Babel can help me parse the file and generate an abstract syntax tree, so we parse the file obtained by fs at {2} , Generated an AST. as follows:

{
  type: 'File',
  start: 0,
  end: 50,
  loc: SourceLocation {
    start: Position { line: 1, column: 0 },
    end: Position { line: 3, column: 18 },
    filename: undefined,
    identifierName: undefined
  },
  errors: [],
  program: Node {
    type: 'Program',
    start: 0,
    end: 50,
    loc: SourceLocation {
      start: [Position],
      end: [Position],
      filename: undefined,
      identifierName: undefined
    },
    sourceType: 'module',
    interpreter: null,
    body: [ [Node], [Node] ],
    directives: []
  },
  comments: []
}

We focus on program.body. There are two objects inside, which are actually two statements in index.js. If you print it, you can see the following:

[
  Node {
    type: 'ImportDeclaration',
    start: 0,
    end: 30,
    loc: SourceLocation {
      start: [Position],
      end: [Position],
      filename: undefined,
      identifierName: undefined
    },
    specifiers: [ [Node] ],
    source: Node {
      type: 'StringLiteral',
      start: 18,
      end: 30,
      loc: [SourceLocation],
      extra: [Object],
      value: './hello.js'
    }
  },
  Node {
    type: 'ExpressionStatement',
    start: 32,
    end: 50,
    loc: SourceLocation {
      start: [Position],
      end: [Position],
      filename: undefined,
      identifierName: undefined
    },
    expression: Node {
      type: 'CallExpression',
      start: 32,
      end: 50,
      loc: [SourceLocation],
      callee: [Node],
      arguments: [Array]
    }
  }
]

3. Obtain dependencies

Looking at the type, you can know that the first one is actually a quotation statement. It should be very sensitive to see here. We need to package the files. Of course, this quotation relationship is very important. We need to continue the analysis next, and we must find the referenced file through such a reference relationship, so this import statement must be saved. Fortunately, Babel provides the @babel/traverse (traverse) method to maintain the overall state of the AST. We use it in {3} to help us find the dependent modules.

It is worth mentioning that the traverse parsed out is a relative path, but in order to facilitate our next processing, we must convert this relative path to an absolute path. The specific method is as shown in the code.

4. AST to executable code

In addition to taking dependencies, we also need to convert the AST into browser executable code. The @babel/core and @babel/preset-env provided by Babel can do just that, so in {4}, we did this One step conversion.

At this point, we have completed the analysis of a module, let's take a look at what results we will get:

{
  filename: './src/index.js',
  dependencies: { './hello.js': './src\\hello.js' },
  code: '"use strict";\n' +
    '\n' +
    'var _hello = _interopRequireDefault(require("./hello.js"));\n' +
    '\n' +
    'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }\n' +
    '\n' +
    'console.log(_hello["default"]);'
}

As you can see, we know who the parsed file is, what dependencies it has, and what the executable js code is.

Obtain dependency map

Up to now, we have obtained the analysis of a module. To fully implement a function, we also need to process all the modules it depends on. So we need a way to help us get the entire dependency graph, so we define the makeDenpendenciesGraph method to help us do this.
Look at the code directly:

const makeDenpendenciesGraph = (entry) => {         //分析所有依赖模块，获得依赖图谱
    const entryModule = moduleAnalyser(entry);
    const graph = {};
    const graphArray = [ entryModule ];
    while(graphArray.length > 0){
        [...graphArray].forEach(item => {
            graphArray.shift();
            const { dependencies } = item;
            graph[item.filename] = {
                dependencies: item.dependencies,
                code: item.code
            }
            if(dependencies) {
                for(let j in dependencies){
                    graphArray.push(moduleAnalyser(dependencies[j]))
                }  
            }
        });
    }
    return graph;
}

This part is actually relatively simple. We use a breadth-first traversal to see if there are any dependencies from the parsed results of the moduleAnalyser. If so, we will continue to parse them and put all the parsed results together. Take a look at the generated dependency graph:

{
  './src/index.js': {
    dependencies: { './hello.js': './src\\hello.js' },
    code: '"use strict";\n' +
      '\n' +
      'var _hello = _interopRequireDefault(require("./hello.js"));\n' +
      '\n' +
      'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }\n' +
      '\n' +
      'console.log(_hello["default"]);'
  },
  './src\\hello.js': {
    dependencies: { './word.js': './src\\word.js' },
    code: '"use strict";\n' +
      '\n' +
      'Object.defineProperty(exports, "__esModule", {\n' +
      '  value: true\n' +
      '});\n' +
      'exports["default"] = void 0;\n' +
      '\n' +
      'var _word = _interopRequireDefault(require("./word.js"));\n' +
      '\n' +
      'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }\n' +
      '\n' +
      'var _default = "hello ".concat(_word["default"]);\n' +
      '\n' +
      'exports["default"] = _default;'
  },
  './src\\word.js': {
    dependencies: {},
    code: '"use strict";\n' +
      '\n' +
      'Object.defineProperty(exports, "__esModule", {\n' +
      '  value: true\n' +
      '});\n' +
      'exports["default"] = void 0;\n' +
      'var word = "word";\n' +
      'var _default = word;\n' +
      'exports["default"] = _default;'
  }
}

Generate executable js

We got the dependency graph, in fact, there is only the last step left, to integrate the parsed content together, and generate a js file that can be executed. On the code:

const generateCode = (entry) => {
    const graph = makeDenpendenciesGraph(entry);
    return `(function(graph){
        function require(module){
            function localRequire(relativePath){
                return require(graph[module].dependencies[relativePath]);
            }
            var exports = {};
            (function(require, exports, code){
                eval(code)
            })(localRequire, exports, graph[module].code);
            return exports;
        };
        require('${entry}')
    })(${JSON.stringify(graph)})`;
}

In fact, we just want to put the code in the dependency graph together and return an executable js, which is actually a js string.

We noticed that there is a require method and an exports object in the code. If we do not define these two things, an error will be reported when js is executed.

In the closure, we take require as the entrance, and take a closure to divide the modules to prevent internal variable pollution. At the same time, we noticed that the relative path is used in the code, so a localRequire is defined to do an absolute path conversion to find the dependent modules.

At this point, a package of the code organized by esModule has been completed, let's take a look at the result:

(function(graph){
    function require(module){
        function localRequire(relativePath){
            return require(graph[module].dependencies[relativePath]);
        }
        var exports = {};
        (function(require, exports, code){
            eval(code)
        })(localRequire, exports, graph[module].code);
        return exports;
    };
    require('./src/index.js')
 })({"./src/index.js":{"dependencies":{"./hello.js":"./src\\hello.js"},"code":"\"use strict\";\n\nvar _hello = _interopRequireDefault(require(\"./hello.js\"));\n\nfunction _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { \"default\": obj }; }\n\nconsole.log(_hello[\"default\"]);"},"./src\\hello.js":{"dependencies":{"./word.js":"./src\\word.js"},"code":"\"use strict\";\n\nObject.defineProperty(exports, \"__esModule\", {\n  value: true\n});\nexports[\"default\"] = void 0;\n\nvar _word = _interopRequireDefault(require(\"./word.js\"));\n\nfunction _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { \"default\": obj }; }\n\nvar _default = \"hello \".concat(_word[\"default\"]);\n\nexports[\"default\"] = _default;"},"./src\\word.js":{"dependencies":{},"code":"\"use strict\";\n\nObject.defineProperty(exports, \"__esModule\", {\n  value: true\n});\nexports[\"default\"] =void 0;\nvar word = \"word\";\nvar _default = word;\nexports[\"default\"] = _default;"}})

Put this code in the browser and print out the expected "hello word"

The complete code is as follows:

const fs = require('fs')
const path = require('path')
const paser = require('@babel/parser')
const traverse = require('@babel/traverse').default
const { transformFromAst } = require('@babel/core')
const moduleAnalyser = (filename) => {          //解析一个模块，生成抽象语法树，并转换成好处理的对象
    const content = fs.readFileSync(filename, 'utf-8');
    const ast = paser.parse(content,{
        sourceType: 'module'
    })
    const dependencies = {};
    traverse(ast, {
        ImportDeclaration({node}){
            const dirname = path.dirname(filename);
            const newFile = './' + path.join(dirname, node.source.value)
            dependencies[node.source.value] = newFile
        }
    })
    const { code } = transformFromAst(ast, null, {
        presets: ["@babel/preset-env"]
    })
    return {
        filename,
        dependencies,
        code
    }
}
const makeDenpendenciesGraph = (entry) => {
    const entryModule = moduleAnalyser(entry);
    const graph = {};
    const graphArray = [ entryModule ];
    while(graphArray.length > 0){
        [...graphArray].forEach(item => {
            graphArray.shift();
            const { dependencies } = item;
            graph[item.filename] = {
                dependencies: item.dependencies,
                code: item.code
            }
            if(dependencies) {
                for(let j in dependencies){
                    graphArray.push(moduleAnalyser(dependencies[j]))
                }  
            }
        });
    }
    return graph;
}
const generateCode = (entry) => {
    const graph = makeDenpendenciesGraph(entry);
    return `(function(graph){
        function require(module){
            function localRequire(relativePath){
                return require(graph[module].dependencies[relativePath]);
            }
            var exports = {};
            (function(require, exports, code){
                eval(code)
            })(localRequire, exports, graph[module].code);
            return exports;
        };
        require('${entry}')
    })(${JSON.stringify(graph)})`;
}

const code = generateCode('./src/index.js')
console.log(code)