Out of memory when building the project? Learn about node memory limits

background

In a previous post , we encountered an issue with a project building 内存溢出 .

The solution at the time was: 直接调大 node 的内存限制，避免达到内存上限。

Today, I heard a colleague share one 新方法 , I think it is good, I hereby record it, and share it with you by the way, I hope it will be helpful to everyone.

text

Directly report the error diagram:

The hint is already obvious: Javascript Heap out of memory.

When we see the keyword memory overflow, we usually consider that it is caused by insufficient memory in Node.js.

But what would the Node process's 内存限制 be?

I found the following description online:

Currently, by default V8 has a memory limit of 512mb on 32-bit systems, and 1gb on 64-bit systems. The limit can be raised by setting --max-old-space-size to a maximum of ~1gb (32- bit) and ~1.7gb (64-bit), but it is recommended that you split your single process into several workers if you are hitting memory limits.

translate:

Currently, by default, V8 has a memory limit of 512mb on 32-bit systems and 1gb on 64-bit systems.

This limit can be raised by setting --max-old-space-size to a maximum of ~1gb (32-bit) and ~1.7gb (64-bit), but if you hit memory limits, it is recommended that you split 单个进程 is 多个工作进程 .

If you want to know how big the memory limit of your computer is, you can directly burst the memory and see the error.

Run the following code:

 // Small program to test the maximum amount of allocations in multiple blocks.
// This script searches for the largest allocation amount.

// Allocate a certain size to test if it can be done.
function alloc (size) {
  const numbers = size / 8;
  const arr = []
  arr.length = numbers; // Simulate allocation of 'size' bytes.
  for (let i = 0; i < numbers; i++) {
    arr[i] = i;
  }
  return arr;
};

// Keep allocations referenced so they aren't garbage collected.
const allocations = []; 

// Allocate successively larger sizes, doubling each time until we hit the limit.
function allocToMax () {
  console.log("Start");

  const field = 'heapUsed';
  const mu = process.memoryUsage();

  console.log(mu);

  const gbStart = mu[field] / 1024 / 1024 / 1024;

  console.log(`Start ${Math.round(gbStart * 100) / 100} GB`);

  let allocationStep = 100 * 1024;

  // Infinite loop
  while (true) {
    // Allocate memory.
    const allocation = alloc(allocationStep);
    // Allocate and keep a reference so the allocated memory isn't garbage collected.
    allocations.push(allocation);
    // Check how much memory is now allocated.
    const mu = process.memoryUsage();
    const gbNow = mu[field] / 1024 / 1024 / 1024;

    console.log(`Allocated since start ${Math.round((gbNow - gbStart) * 100) / 100} GB`);
  }

  // Infinite loop, never get here.
};

allocToMax();

Not surprisingly, you will like to report the following error:

My computer is a Macbook Pro masOS Catalina 16GB, and the Node version is v12.16.1. This code throws an exception when the memory is about 1.6 GB.

Now we know that Node Process does have a memory limit, so let's increase its memory limit and try again.

Running this code with node --max-old-space-size=6000 gives the following results:

It also overflowed when the memory reached 4.6G.

You might ask, doesn't node have memory reclamation? We will talk about this below.

Using this parameter: node --max-old-space-size=6000 , the size of our increased memory 老生代区域 is more violent.

As mentioned above: If the memory limit is reached, it is recommended that you split 单个进程 into 多个工作进程 .

This project is a ts project, and the compilation of ts files takes up more memory. If this part is separated into a separate process, the situation will be improved.

Because ts-loader internally calls tsc , when using ts-loader, the tsconfig.js configuration file will be used.

When the code in the project becomes more and more and the volume becomes larger and larger, the project 编译时间 also increases.

This is because Typescript's 语义检查器 must be in 每次重建时检查所有文件 .

ts-loader provides a transpileOnly option, it defaults false , we can set it to true , so that when the project does not compile Type checking is done and no declaration file is output.

Compare the project construction speed of ---71066861e8f8cc9476d1c5a50dc60f13---set false and true transpileOnly respectively:

• When transpileOnly is false, the overall build time is 4.88s.
• When transpileOnly is true, the overall build time is 2.40s.

Although the build speed is improved, there is a drawback: 打包编译不会进行类型检查 .

Fortunately, the official recommended such a plug-in, which provides such capabilities: fork-ts-checker-webpack-plugin .

The use of the official example is also very simple:

 const ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin')

module.exports = {
  ...
  plugins: [
    new ForkTsCheckerWebpackPlugin()
  ]
}

In my actual project, vue.config.js is modified as follows:

 configureWebpack: config => {
 // get a reference to the existing ForkTsCheckerWebpackPlugin
 const existingForkTsChecker = config.plugins.filter(
   p => p instanceof ForkTsCheckerWebpackPlugin,
 )[0];

// remove the existing ForkTsCheckerWebpackPlugin
// so that we can replace it with our modified version
 config.plugins = config.plugins.filter(
   p => !(p instanceof ForkTsCheckerWebpackPlugin),
 );

 // copy the options from the original ForkTsCheckerWebpackPlugin
 // instance and add the memoryLimit property
 const forkTsCheckerOptions = existingForkTsChecker.options;
 
 forkTsCheckerOptions.memoryLimit = 4096;
 
 config.plugins.push(new ForkTsCheckerWebpackPlugin(forkTsCheckerOptions));
}

After the modification, the build succeeded.

About garbage collection

In Node.js, V8 automatically helps us with garbage collection, let's take a brief look at how memory is handled in V8.

some definitions

• Resident set size: is the portion of memory occupied by processes saved in RAM, including:

  1. the code itself
  2. stack
  3. heap
• stack: contains primitive types and references to objects
• Heap: Stores reference types such as objects, strings or closures
• Shallow size of the object: the size of the memory held by the object itself
• Retained size of an object: The amount of memory freed after deleting an object and its related objects

How the Garbage Collector Works

Garbage collection is the process of reclaiming memory occupied by objects that are no longer used by an application.

Usually, memory allocation is cheap, and collection is expensive when the memory pool runs out.

If an object cannot be accessed from the root node, the object is a candidate for garbage collection, so the object is not referenced by the root object or any other live objects.

The root object can be a global object, a DOM element or a local variable.

The heap has two main parts, namely New Space and Old Space .

New space is where new allocations are made.

Garbage collection is fast here, about 1-8MB in size.

The object remaining in the new space is called 新生代 .

Objects that survived in the new space were lifted into the old space - they were called 老生代 .

Allocations in old spaces are fast, but collections are expensive, so they are rarely performed.

node garbage collection

Why is garbage collection expensive?

The V8 JavaScript engine employs a stop-the-world garbage collector mechanism.

In practice, it means that the program stops execution while garbage collection is in progress.

Typically, about 20% of the young generation can survive to the old generation, and the collection of the old space will not start until it is exhausted.

For this, the V8 engine uses 两种不同的收集算法 :

1. Scavenge: very fast, works on 新生代 ,
2. Mark-Sweep: Slower and works on 老生代 .

Space is limited. For more information on v8 garbage collection, you can refer to the following articles:

1. http://jayconrod.com/posts/55/a-tour-of-v8-garbage-collection
2. https://juejin.cn/post/6844903878928891911
3. https://juejin.cn/post/6844903859089866760

Summarize

To sum up, there are two methods described above:

1. Directly increase the memory, use: node --max-old-space-size=4096
2. Separate some memory-consuming processes and use a plugin: fork-ts-checker-webpack-plugin

I hope you will leave an impression and remember these two ways.

Okay, that's all for that, thank you.

I am ignorant, if I am wrong, please correct me.

thanks.

Relevant information

1. https://www.cloudbees.com/blog/understanding-garbage-collection-in-node-js/
2. http://jayconrod.com/posts/55/a-tour-of-v8-garbage-collection
3. https://blog.risingstack.com/finding-a-memory-leak-in-node-js/

At last

If you find the content helpful, you can follow my official account to keep up with the latest developments and learn together!