Node abnormal data response troubleshooting (pm2 Cluster Mode, asynchronous)

Received an ironclad feedback yesterday

There is a method in a js file written in node. This js file declares a var global variable (global), and then the method mentioned above is to first judge whether the global variable has a value, if it has a value, it will return directly, if there is no value Just get the value through the interface;
Then after a specific operation on the page, the value of the global variable will be cleared to null, and then the value of the interface will be obtained from the above.
The local is normal, the server is wrong

The information they gave was definitely not enough , so I asked for some information.

• Is it an error? Because the error will cause the Node application to restart, and then cause the state to become invalid.

  no error

• Can you provide a reproduction code? This generally depends on the volume of the project, or the current problem location progress

  can not provide. The local is normal, the server is wrong

• What is the execution environment? commonjs? ESM? Ts? Here I want to see if there is any sleazy operation, such as serverless and the like that cannot be saved.

  Node

• Then I also asked what services are started locally and on the server? Here I want to confirm whether it is a Cluster, because the Cluster state is not shared and requires a special solution.

  node

In fact, I know that this person is not a professional Node, and the previous information may also be poisonous.

At this time, a screenshot of the log was suddenly sent to me, which solved the case directly. Cluster mode data sharing problem, the service from the local node does not have this problem, the server should be pm2 start index.js -i 4 and the like.

 0| www xxxxx
1| www xxxxx
3| www xxxxx
0| www xxxxx

The next step is to troubleshoot the problem with the minimal reproduction demo and fix it.

Reproduce Cluster data sharing issues

In fact, when he showed me that it was Cluster, he had already located the problem, a very obvious data sharing problem .

Let's look at our reproduced example, and we can find that the output of a single instance is correct , precisely because the request falls to different machines (instances) resulting in different responses .

 if (!global.a) {
    global.a = 1
}
console.log(global.a, Date.now())
function randomTask() {
    console.log(++global.a, Date.now())
    if (global.a < 5) {
        setTimeout(randomTask, Math.random() * 1000)
    }
}
randomTask();

Statelessify your application
Be sure your application is stateless meaning that no local data is stored in the process, for example sessions/websocket connections, session-memory and related. Use Redis, Mongo or other databases to share states between processes.
Another resource on how to write efficient, production ready stateless application is The Twelve Factor Application manifesto.

repair

Do not start Cluster cluster mode

Because the local is non-Cluster cluster mode, it behaves normally. Then the first solution is to not turn on the cluster mode in the production environment, but in general this solution is not advisable, the request in the production environment is relatively high, and the cluster mode is the optimal solution.

Increase single-instance data service | Downgrade to single-instance mode

Similar to redis, but a new single-instance nodeJs script. Getting data & updating data is requesting this script service.

Because cluster mode is not used, there is no sharing problem. At the same time, it also avoids the problem of the previous solution, because the data service is not open to the outside world, only for the service of the intranet, so the request level will not be too large.

redis

Published & subscribe

The publish-subscribe function is implemented through redis. When updating data, Published All Workers update data. Subscribe updates its own data when it receives an update.

code show as below.
As for why there are multiple redis instances? This is because a redis instance can only be a publisher or a subscriber , so we need two instances, one for publishing updated data and one for monitoring updates sent by other workers.

 // ioredis
const Redis = require("ioredis")
let redisClient3 = new Redis()
let redisClient4 = new Redis()

setInterval(() => {
    const message = { foo: Math.random(), pid: process.pid };
    const channel = `my-channel-${1 + Math.round(Math.random())}`;
    
    redisClient3.publish(channel, JSON.stringify(message));
    console.log("Published %s to %s", message, channel);
}, 5000);

redisClient4.subscribe("my-channel-1", "my-channel-2", (err, count) => {
    if (err) {
        console.error("Failed to subscribe: %s", err.message);
    } else {
        console.log(
            `Subscribed successfully! This client is currently subscribed to ${count} channels.`
        );
    }
});

redisClient4.on("message", (channel, message) => {
    console.log(`Received ${message} from ${channel}`);
});

fs

Because there is no communication between cluster instances, it is necessary to find a common access, so local disk is also a feasible solution. However, there may be conflicts in fs, so it is not recommended to use it.

After trying it, it seems that there will be no error, and there will be no confusion in the content, that is, the content that may be taken out is empty.

 const fs = require('fs');
const str = `process.pid: ${process.pid}`.repeat(999) + '\n';
console.log(`process.pid: ${process.pid}`)
const test = ()=>{
    for(var i = 0; i < 10; i++){
        console.log(`process.pid: ${process.pid} ${i}`)
        fs.writeFile('message.txt', `${i} ${str}`, (err)=>{
            if(err) console.log(err)
        });
    }
    setTimeout(test, Math.random() * 100);
    // setTimeout(test);
}
test();

cluster module

Because pm2 starts all workers, this solution is not suitable for us.

 if (cluster.isMaster) {
  const worker = cluster.fork();
  worker.send('你好');
} else if (cluster.isWorker) {
  process.on('message', (msg) => {
    process.send(msg);
  });
}