Metro unpacking working principle and actual combat

1. Background

Students who have touched RN know that hot update, as one of the biggest features of RN, allows developers to launch new iterations and fix online bugs at any time. In the previous article, we talked about the construction of the hot update platform . Today, let's talk about the unpacking link in the hot update.

Hot update and unpacking are topics that everyone talks about a lot. Usually, a technical topic that is talked a lot will have a set of mature technical solutions. For example, the hot update platform has mature solutions such as CodePush, but there is no unpacking. A set of programs that everyone recognizes as mature. However, there are react-native-multibundler , Ctrip's moles-packer , and 58.com's metro-code-split that support unpacking on the market. Since the first two have stopped updating, no special introduction will be made.

As we all know, Facebook's open source Metro packaging tool does not have an unpacking function itself. Its main function is to package JavaScript code into a Bundle file, and Metro does not support third-party plug-ins, so the community does not have third-party unpacking plug-ins.

However, when we read the Metro source code, we found a configurable function customSerializer , and thus found a way to write third-party plug-ins for Metro without invading the Metro source code. With Metro's customSerializer method, we can now write plug-ins for Metro, and provide individual unpacking capabilities through plug-ins.

2. Basic use of metro-code-split

metro-code-split is a plug-in developed by the technical team of 58.com to support RN unpacking. It currently supports the latest version 0.66.2. For related article introductions, please refer to: 58RN page second opening plan and practice

Next, let's take a look at how to integrate metro-code-split into an existing project. First, we install the metro-code-split plugin in the project.

 npm i metro-code-split -D
//或者
yarn add metro-code-split -D

Then, add the following script to the package.json configuration file:

 "scripts": {
    "start": "mcs-scripts start -p 8081",
    "build:dllJson": "mcs-scripts build -t dllJson -od public/dll",
    "build:dll": "mcs-scripts build -t dll -od public/dll",
    "build": "mcs-scripts build -t busine -e index.js"
  }

The specific meaning of the script is as follows:

• start: start the local debugging service;
• build:dllJson: The module file for building the public package;
• build:dll: build public package;
• build: Build business packages and load packages on demand.

If it is a development environment, the above configuration script requires the NODE_ENV=xxx parameter, which is modified as shown below.

 "scripts": {
    "start": "NODE_ENV=production react-native start --port 8081",
    "build:dllJson": "NODE_ENV=production react-native bundle --platform ios --entry-file node_modules/.cache/metro-code-split/dll-entry.js --bundle-output public/dll/_dll.ios.json --dev false & NODE_ENV=production react-native bundle --platform android --entry-file node_modules/.cache/metro-code-split/dll-entry.js --bundle-output public/dll/_dll.android.json --dev false",
    "build:dll": "NODE_ENV=production react-native bundle --platform ios --entry-file node_modules/.cache/metro-code-split/dll-entry.js --bundle-output public/dll/_dll.ios.bundle --dev false & NODE_ENV=production react-native bundle --platform android --entry-file node_modules/.cache/metro-code-split/dll-entry.js --bundle-output public/dll/_dll.android.bundle --dev false",
    "build": "NODE_ENV=production react-native bundle --platform ios --entry-file index.js --bundle-output dist/buz.ios.bundle --dev false & NODE_ENV=production react-native bundle --platform android --entry-file index.js --bundle-output dist/buz.android.bundle --dev false"
  }

Next, modify the metro.config.js file configuration as follows:

 const Mcs = require('metro-code-split')

// 拆包的配置
const mcs = new Mcs({
  output: {
    // 配置你的 CDN 的 BaseURL 
    publicPath: 'https://static001.geekbang.org/resource/rn',
  },
  dll: {
    entry: ['react-native', 'react'], // 要内置的 npm 库
    referenceDir: './public/dll', // 拆包的路径
  },
  dynamicImports: {}, // dynamic import 是默认开启的
})

// 业务的 metro 配置
const busineConfig = {
  transformer: {
    getTransformOptions: async () => ({
      transform: {
        experimentalImportSupport: false,
        inlineRequires: true,
      },
    }),
  },
}

// Dynamic Import 在本地和线上环境的实现是不同的
module.exports = process.env.NODE_ENV === 'production' ? mcs.mergeTo(busineConfig) : busineConfig

There are two unpacking parameters to pay attention to: one is publicPath, which is used to configure the root path of the package on demand in the online environment. Another parameter to be aware of is the dll, which is used to configure the built-in npm library.

Usually in a mixed React Native application, the two packages "react" and "react-native" are basically unchanged, so you can split the two npm libraries into a common package, which only Can follow the App release version update. For other business codes or third-party libraries, such as "reanimated", these codes change relatively frequently, so they can be integrated with the business package to facilitate dynamic updates.

After configuring metro-code-split, how to use metro-code-split for unpacking? metro-code-split supports the splitting of three types of packages, including public packages, business packages, and on-demand packages.

public package

When you fill in "react" and "react-native" in the dll configuration item, "react" and "react-native" will be treated as public packages every time you package.

 dll: {
    entry: ['react-native', 'react'], // 要内置的 npm 库
    referenceDir: './public/dll', // 拆包的路径
  },

Next, run the yarn build:dll command directly to remove the public package. After the operation is completed, you will check the public/dll directory, and you will find that there are two more files under the directory, namely _dll.android.bundle and _dll.ios.bundle, these two files are integrated with "react" and "react-native" a common package for all code.

If you want to see the modules included in the public package, you can use the following command:

 yarn build:dllJson

After running the above command, you can find two files, _dll.android.json and _dll.ios.json, which contain all the modules that "react" and "react-native" depend on, as follows.

 [
  "__prelude__", // 框架预制模块
  "require-node_modules/react-native/Libraries/Core/InitializeCore.js", // react-native 初始化模块
  "node_modules/@babel/runtime/helpers/createClass.js", // babel 的类模块
  "node_modules/react-native/index.js", // react-native 入口模块
  "node_modules/metro-runtime/src/polyfills/require.js", // require 运行时模块 
  "node_modules/react/index.js" // react 模块
]

_dll.json records all common modules, _dll.bundle contains all common module code, such as framework prefab module __prelude__ that manages React Native global variables, InitializeCore module that manages initialization, modules that manage babel, require, and react and react- The entry module for the native framework.

Business Packages and On-Demand Packages

When you get the built-in package, except for the built-in code of "react" and "react-native", all other codes belong to the business package, with the exception of one type of file, which is to load modules on demand. However, due to the strong coupling between the business package and the on-demand loading package, the on-demand loading package cannot be packaged independently from the business package, so I will introduce the business package and the on-demand loading package together.

Usually, you introduce a common business module and use import * from "xxx" , then the code of this module will be directly typed into the business package. However, when the on-demand loading business module is introduced, import("xxx") is used, then the module code will be directly typed into the on-demand loading package. For example, there is the following code:

 import React, {lazy, Suspense} from 'react';
import {
  Text,
} from 'react-native';
import {NavigationContainer} from '@react-navigation/native';
import {
  createNativeStackNavigator,
} from '@react-navigation/native-stack';
import {Views, RootStackParamList} from './types';
import Main from './component/Main';

const Stack = createNativeStackNavigator<RootStackParamList>();

const Foo = lazy(() => import('./component/Foo'));
const Bar = lazy(() => import('./component/Bar'));

export default function App() {
  return (
    <Suspense fallback={<Text>Loading...</Text>}>
      <NavigationContainer>
        <Stack.Navigator initialRouteName={Views.Main}>
          <Stack.Screen name={Views.Main} component={Main} />
          <Stack.Screen name={Views.Foo} component={Foo} />
          <Stack.Screen name={Views.Bar} component={Bar} />
        </Stack.Navigator>
      </NavigationContainer>
    </Suspense>
  );
}

It can be seen that the Main component is introduced through import * from "xxx" , which belongs to the ordinary business module; while the Foo component and the Bar component are introduced through import("xxx") , they belong to the on-demand loading business module. After we have finished writing the code, we can use the following commands to generate business packages and load packages on demand.

 yarn build

After the construction is completed, the service package and the on-demand loading package will be placed in the dist directory, where buz.android.bundle and buz.ios.bundle are the service packages, and the packages starting with the MD5 value in the chunks directory are on-demand Load the package.

 dist
├── buz.android.bundle
├── buz.ios.bundle
└── chunks
    ├── 22b3a0e5af84f7184abd.bundle
    └── 479c3b2dc4e8fef12a34.bundle

It can be seen that through yarn build:dll and yarn build , we have completed the construction of public packages, business packages, and on-demand packages.

Attachment: Mcs default configuration parameters

Third, the principle of unpacking

3.1 Metro packaging process

metro is a RN packaging tool, and now we can also use it to unpack. The metro packaging process is divided into the following steps:

1. Resolution : Metro needs to build a graph of all the modules it needs from the entry point, to find the file it needs from another file, the Metro resolver needs to be used. In actual development, this stage is parallel to the Transformation stage.
2. Transformation : All modules go through the Transformation stage. Transformation is responsible for converting modules into a syntax format that the target platform can understand (such as React Naitve). The conversion of modules is based on the number of cores you have.
3. Serialization : All modules will be serialized as soon as they are converted. Serialization will combine these modules to generate one or more packages. A package is a package that combines modules into a JavaScript file. Serialization provides a series of methods for developers to Customize some content, such as module id, module filtering, etc.

Open the createModuleIdFactory code of the Metro library, the path is node_modules/metro/src/lib/createModuleIdFactory.js , you can see the following code.

 function createModuleIdFactory() {
  const fileToIdMap = new Map();
  let nextId = 0;
  return path => {
    let id = fileToIdMap.get(path);

    if (typeof id !== "number") {
      id = nextId++;
      fileToIdMap.set(path, id);
    }

    return id;
  };
}

module.exports = createModuleIdFactory;

The logic of the above code is: if it is found that this module is not recorded in the map, the id will be incremented automatically, and then the module will be recorded in the map, so it can be seen from this that the rule for generating moduleId from the official code is to increment itself, so it needs to be replaced here. If we want to unpack the package, we need to ensure that the id cannot be repeated, but this id is only generated during packaging. If we package the business package and the basic package separately, the continuity of the id will be lost, so for For the processing of id, we can still refer to the above open source projects. Each package has a 100,000-bit interval space. The basic package starts to increase from 0, and the business A starts to increase from 1,000,000, or through each module’s own path or uuid, etc. are allocated to avoid collisions, but strings will increase the size of the package, which is not recommended here.

3.2 Module-based unpacking scheme

Let's take a look at the metro-code-split unpacking tool. Compared with the text-based unpacking method, the module-based unpacking and loading speed is faster. Why does module-based unpacking load faster than text-based unpacking? This is because module-based unpacking works independently.

So why can the module-based unpacking method run independently, but the text-based unpacking method cannot?

Let's first look at the text-based unpacking method. Suppose we are using a multi-Bundle text-based unpacking method. The common code part between multiple bundles is the "react" and "react-native" libraries, here I use console.log("react"), console.log("react-native") instead. The different code parts among multiple bundles are business codes, here console.log("Foo") is used to replace a specific business code.

For text-based unpacking, we use Google's open source diff-match-patch algorithm, which also provides an online computing website. The schematic diagram of calculating hot update packages is as follows:

As you can see, in the hot update diagram above, we will build the Old Version string file, and this part of the code will not be easily changed except to upgrade the React Native version. The string of New Version is the target code of this hot update, that is, the complete Bundle file, but the developer does not need to download the complete Bundle file, because the Old Version has been built into the App, we only need to issue the Patch Hot update package is enough. After the client receives the Patch hot update package, it will merge with the built-in package represented by Old Version, and finally load the merged complete Bundle package.

It can be seen that, based on the principle of text-based unpacking and co-packing, the Patch hot update package is a piece of text that records the modified location and modified content, rather than an independently executable code. The direct result is that it can only be generated after the download is completed. The complete Bundle file can be executed as a whole. This is why text-based unpacking is not independently executable.

However, based on the unpacking method of the module, the built-in package and the hot update package can be executed independently. Similarly, taking the hot update of the Foo business in the multi-Bundle mode as an example, the following seems to be based on the schematic diagram of module unpacking.

It can be seen that the hot update package based on the module unpacking scheme can be run independently. Therefore, after using the module unpacking scheme, you can run the built-in package on the client side first, and download the hot update package in parallel, wait for the hot update package to be downloaded, and then run the hot update package. Of course, you can also download it after the application starts, so as to Reduced the loading time of hot update packages.

3.2 Hot update and unpacking

After the previous operations, we have generated public packages, business packages, and on-demand loading packages. The next step is how to implement hot updates and run them. The following is a schematic diagram of a hot update of the unpacking scheme.

Because we use the module unpacking scheme, although in theory each package can run independently, in fact there is a dependency between modules and modules. On the whole, the on-demand loading package will depend on the business package. modules, the business package will depend on the modules in the public package. Therefore, it is necessary to execute the public package first, then the business package, and finally the on-demand loading package.

Of course, there will also be dependencies between each independent on-demand loading package, but these loading dependencies, metro-code-split has already taken into account for you, you can use it directly. For a multi-bundle hybrid application where the home page is a Native page and other pages are React Native pages, the overall loading process is as follows:

First, after starting the App, find a spare time, pre-create the React Native environment, and then pre-load the "disassembled public package".

Then, when the user clicks to enter the React Native page, pass in the unique identifier or CDN address of the React Native page in the relevant jump protocol, download the service package and load the page:

 https://static001.geekbang.org/resource/rn/id999.buz.android.bundle

However, for some complex services, there will be more page content, and putting some code below the fold in the business package will slow down the loading speed of the fold. Therefore, a better solution is to put these codes on demand. Load the package. When the user clicks a button or pulls down, the related on-demand loading logic is triggered.

At this point, metro-code-split will find the corresponding CDN address according to the parameter path in import(‘xxx’) . For example, the Foo.js module corresponds to the following CDN address:

 https://static001.geekbang.org/resource/rn/03ad61906ed0e1ec92c2.bundle

Then, load the package on demand according to the CDN address request, and execute the downloaded code through new Function(code), load the Foo component into the current JavaScript context, and perform final rendering. The above solutions are suitable for hybrid applications where the home page is a Native page. What if the home page is also a React Native page?

1. The home page is a React Native page, and it adopts a multi-Bundle strategy

Then, the public package still needs to be built-in, and the homepage business package also needs to be built-in. At this time, the home page service package adopts the silent update strategy, that is, the strategy of current download and effective next time. In this way, each time the home page is started, the service package of the home page is loaded locally, and the startup speed of the home page will be faster without going through the network request. The service packages or on-demand loading packages of other pages continue to be updated in the form of dynamic delivery that takes effect at that time.

The current method takes about 300ms~500ms to download the bundle, but the advantage is that the business can be updated at any time, and the bugs can be fixed at any time, without waiting for the user to enter the page next time to take effect.

2. The home page is a React Native page, but it adopts a single Bundle strategy

Then, the public package and the business package need to be built in separately. The public package goes through the release version update process, and the business package goes through the CodePush silent update process. Compared with the pure CodePush solution, the download volume of the CodePush update can be saved by unpacking. If you also use the load-on-demand package at the same time, you can also save the execution time of the code below the fold.

If you encounter an emergency bug, CodePush also supports the immediate effect. However, due to the principle of the underlying mechanism of CodePush, it not only needs to download the hot update Bundle, but also needs to reload the entire JavaScript environment, which takes a long time. Therefore, it is not recommended that you use it as the default update method.

4. Summary

Now, using the open source unpacking tool metro-code-split can easily help you split the entire Bundle package into public packages, business packages and on-demand packages. You only need to download, configure and execute the command to complete the unpacking operation.

Local unpacking is only one part of the hot update process, so you need to use it together with your hot update process. According to different businesses, applications can be roughly divided into three forms, including pure React Native applications with a single bundle, pure React Native applications with multiple bundles, and mixed applications with multiple bundles. Package strategies are different, and you need to analyze them in combination with specific scenarios.

Although it is easy to unpack using metro-code-split, it is not easy to implement metro-code-split. There is a lot of work to be done at compile time and runtime, and you have to make forward dependencies and reverses of all modules Only when the dependencies are clearly explained can the unpacking be carried out reasonably.

Reference: metro-code-split example