Summary of the troubleshooting process for too many node file operations at one time

• Front-end advanced tour: https://interview2.poetries.top
• Blog: https://blog.poetries.top
• Official account/mini program: "Front-end advanced journey" Share technical dry goods every day, learn the front-end without getting lost
• Author: programmer poetry

Hi, my name is poetry. Recently, when optimizing the company's internal scaffolding, I encountered a problem, Error: EMFILE, too many open files that is, if nodejs opens too many files, it will lead to errors. After checking again and again, I finally found an effective method, summarize and record it

When I try to manipulate a large number of files

 for(var i=0; i<200000; i++) {
    fq.readFile('./somefile.txt', {encoding: 'utf8'}, function(err, somefile) {
        console.log("data from somefile.txt without crashing!", somefile);
    });
}

The above results in Error: EMFILE: too many open files error. I don't have to close the file because apparently fs.readFile on the file and closes it for me. I'm on Mac OS and my ulimit is set to 8192

 $ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1392
-n: file descriptors                256

You can modify the system configuration, but it is not recommended

 $ echo kern.maxfiles=65536 | sudo tee -a /etc/sysctl.conf
$ echo kern.maxfilesperproc=65536 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ ulimit -n 65536

• This error occurs due to the asynchronous nature of node.js . The process attempted to open more files than allowed, so an error was generated.
• This can be fixed by creating open 文件队列 , 以使它们永远不会超过限制 , here are some libraries that can do this for you:

We can use a file queue to limit the number of files opened at a time

Instantiate Filequeue with a maximum number of files to be opened at once (default is 200)

how to use

 var FileQueue = require('filequeue');
var fq = new FileQueue(100);

// additional instances will attempt to use the same instance (and therefore the same maxfiles)

var FileQueue2 = require('filequeue');
var fq2 = new FileQueue2(100);

console.log(fq === fq2); // => true

// you can force a new instance of filequeue with the `newQueue` parameter

var fq3 = new FileQueue(100, true);

console.log(fq === fq3); // => false

filequeue supports the following methods

 readFile
writeFile
readdir
rename
symlink
mkdir
stat
exists
createReadStream
createWriteStream

Use filequeue to run normally

 var FileQueue = require('filequeue');
var fq = new FileQueue(100); // 限制每次打开的文件数量

for(var i=0; i<200000; i++) {
    fq.readFile('./demo.txt', {encoding: 'utf8'}, function(err, somefile) {
        console.log("data from somefile.txt without crashing!", somefile);
    });
}

This article is published by mdnice Multiplatform