What should I do if NEJ Build is too slow? Try MOOC NEJ, it only takes two steps to increase the build performance by 70%!

What should I do if NEJ Build is too slow? Try MOOC NEJ, it only takes two steps to increase the build performance by 70%!

Due to historical burdens, the construction time of the master station project of the China University MOOC (abbreviated as China M) is about 21 minutes, and the construction adopts NEJ Build. Since NEJ is currently unmaintained and has many applications in the department, China M passed the fork original Under the premise of retaining the original functions of NEJ, the project will transform and upgrade the core packaging process of NEJ, and propose a set of general solutions, namely MOOC NEJ. MOOC NEJ has been verified online at the beginning of this year. The construction time has been shortened to 6 minutes, and the construction performance has been improved by 70%. After 8 months of stable online operation, no problems caused by migration have been found.

1. Historical background

China University MOOC (hereinafter referred to as China M), like many NetEase front-end er, has used RegularJS and NEJ to build projects in the past few years. At the same time, it uses NEJ's toolkit toolkit2 or NEJ Build ( https://github. com/genify/toolkit2 )

The beginning of the project is always happy. At that time, the people who develop and maintain NEJ are still there (you can even mention some features), and the people who work on the business project are also there. The amount of code is always refreshing, and the time of construction is always "咻咻"It passed in a moment.

However! Three to five years have passed. When I took over the project of the nearly 21 minutes .

I usually wait and wait. At most, when the back-end boss is deployed, it is built in advance and waited. What is terrible is that it is 10 o'clock after the line is completed. At this time, the test boss will report: the back-end is okay, there is a front-end here. Fix the problem.

Then, it takes about 5 minutes to repair, and half an hour to package and verify. . . . . At the same time, bless you not to ask (deng) one (ban) (ge) to ask (xiao) questions (shi).

2. Why is it so slow?

There are two reasons for the slow speed:

1. Chinese M project architecture:
Due to historical reasons, the main site of China M puts the web and mobile terminals together under the front-main project. At the same time, it loads two super-large libs that cannot be seen to be different, resulting in a lot of files that need to be packaged. Naturally, I will live in Bengbu.

2. NEJ merger strategy
Toolkit has a merge strategy parameter to determine how many times a file reference count will be merged into core.js. When this parameter is set larger, the size of core.js will be smaller and the build time will be longer. In order to quickly improve development efficiency, we adjusted this parameter to 6 in the test environment, and the packaging time was about 11 minutes, while the pre-release and online parameters were at 13, and the packaging time was about 21 minutes. (After all, we can't rudely increase the resources of the production environment just because we want to pack quickly 😖)

3. Why does NEJ's merger strategy affect the packaging speed [core principle]

Because NEJ has been unmaintained a long time ago, but fortunately it is open source. Although we can't find the author, we can find the answer by reading the source code of toolkit2. I roughly sorted out the process of NEJ packaging.

1. The whole process of toolkit2 uses synchronous packaging, and the processing results of each step are passed through the relay form of one by one.
2. The essence of toolkit2 packaging is to convert the code of each file into an abstract syntax tree (AST) through parameters such as the merge strategy (the parameters mentioned in 2.2), and then compress and obfuscate the AST through UgilifyJS.

It seems that there is no problem with this process, because AST and UgilifyJS are also very mainstream operations now (for example, webpack also uses UgilifyJS), and the front-end can't be more familiar.

So why does it cause slow packaging? Until I saw the AST generation process:

1. all files through concat, a big AST (source code lib/adapter/script.js _mergeCodeAndToAST
   
2. Uglify compression and obfuscation for and only the AST (source code lib/adapter/script.js parse)

Combined with the historical structure of the main site of ZhongM, when I print out this AST,
single master station is approximately formed by 400 small ASTs. . . .
Then it starts to compress. . . .
Then it got stuck. . . . Living. . . . NS. . . .

Four, solution considerations

Based on our current understanding of our own project and NEJ, we probably have the following ideas to optimize the packaging time:

[For China M main station project]

1. The old-fashioned splitting engineering & refactoring migration: Splitting requires a sufficient understanding of the business in order to split the domain. Refactoring is too expensive. At the same time, there are too many functional regression points. At present, some functions have been replaced with React+Webpack due to iteration.
2. Webpack replaces nej: It needs to be compatible with regular and nej modules, and the plug-ins required by webpack must be replaced inside the old project, which is costly.

[For NEJ tool set]

1. Uglify2.0 upgrade 3.0: 3.0 has a fast packaging mode, but after testing, 3.0 has some incompatible apis, which is useful for nej. (Here is the experience of stepping on the pit)
2. Multi-process packaging: do not modify the packaging product, only change the packaging process, the cost is controllable, and the risk is small after going online. If you encounter an emergency problem, you can quickly roll back to the nej build, and the risk is controllable

end, we chose to put the same multi-process function of webpack on NEJ, and we made for what we needed for packaging.

Five, implement the architecture

We don't want to build one more wheel to make some subversive changes to improve performance, but also uncomfortable for the writers (think about writing too much redundant code) and uncomfortable for the users (high migration costs).
Therefore, we chose to fork toolkit2, keep all its APIs and functions, without affecting the posture of any old functions, and only modify the critical path to reduce the cost of transformation and migration.

The last generated library is: @edu/toolkit3
http://npm.hz.netease.com/package/@edu/toolkit3

Our general idea is this:

1. Don't do concat operation on ast, each file separately uglify .
2. Refer to the multi-process idea of ​​the uglify-webpack-plugin plug-in, uses each file as a task, and concurrently packs .
   (I went to take a look at the source code of uglify-webpack-plugin and made a picture. After all, only by knowing yourself and the enemy can the transformation be carried out smoothly.)
   
3. the synchronous process to asynchronous , call back the result after completing all tasks, and call subsequent operations

Six, code implementation

After the fork directory, a new cluster folder is added for storing multi-process processes.

1. minifiy.js: Perform ugilify compression operation to generate compressed and obfuscated ast and string codes;
2. TaskRunner.js: Use worker-farm to distribute, execute, count, and call back tasks;
3. worker.js: worker-farm must create a new workerfile before it can be used, this workerfile is used to undertake;

Cooperate to modify the original file lib/adapter/script.js

1. _mergeCodeAndToAST: cancel the generation of concat and this.ast;
2. parse: Add callback parameters, create tasks based on filename, file, and minify parameters, run each task, and execute callback;

Cooperate to modify the original file lib/deploy.js

1. _afterResPrepared: Change the last embed (embedding variables into css, js, html) operation to asynchronous, and pass it in as a callback. After all the compression is completed, the embed operation before the final output is executed.
   (Detailed code can go to the warehouse to view)

7. Usage & Migration (Amway)

After talking for so long, I finally start to migrate, remember the title,

Just two steps! Worry-free migration!

Does not affect the use of any old functions!

first step: install mooc-nej (currently the stable version is 0.3.0, 0.4.0 is still in beta version, in internal testing)

npm install @edu/toolkit3 -g

Step 2: Use it without even modifying the command, just replace nej build with mooc-nej

 // 所有的nej build改成mooc-nej build即可
 mooc-nej build deploy/mobile/release.js 
 
 // 或者构建机安装直接引用build文件
<property name="mooc-nej-build.js" value="/home/appops/study/install/node_modules/@edu/toolkit3/bin/build.js"/>

<exec dir="." executable="${mooc-nej-build.js}"failonerror="true">
<arg line ="${ob_baseline.dir}/deploy/web/release.js"/>
</exec>

backup: If you find a problem, you can quickly roll back to nej build.

8. Production verification

China M has used mooc-nej at the beginning of the year. has been running stably for more than half a year. No bug caused by migration has been found yet. Please rest assured to eat .

Nine, summary

Probably from the beginning of the renovation to the end of the renovation, it took the free time of daily life in December last year. The mental journey during this period is roughly as follows:

😢 (packed for a long time 555)

🤔 (what is stuck for so long)

😯 (wow found the reason)

😎 (remake, rush!)

😏😤😏😤😏😤... (After the change, I hang up... After I change it, I hang up again...*n)

🤨 (change the idea)

​💓 (Applying verification)

😁 (Successfully launched)

But looking back, these pits are all worthwhile, and we finally don't have to wait for a long time to build!

-END-