RocketMQ -- file inconsistency solution

As mentioned earlier, ConsumeQueue and IndexFile synchronize the commitlog log file information asynchronously through the ReputMessageService thread. If the Broker is down after the Commitlog log file is successfully written, there is no way to synchronize, resulting in inconsistent data in Commitlog, ConsumeQueue, and IndexFile files. So how did RocketMQ solve it?

There is a file in the stroe directory called abort.

This abort file will be created when the Broker starts, and will be destroyed if it exits normally.

So under normal circumstances, the abort file will only exist in the running period, and when it stops, there is no such file.

When the Broker starts, if there is no abort file, it means that it exits normally. If it is found that there is an abort file, it means that it exited abnormally before, and there may be data inconsistencies in the Commitlog, ConsumeQueue, and IndexFile files, and corrections are required at this time.

There is also a file in the stroe directory, called checkpoint (same as the abort diagram), which stores the commit log file refresh time point, the ConsumeQueue file refresh time point, and the IndexFile file refresh time point. These information are used for judgment processing when an abnormal exit occurs. .

When loading IndexFile, if it exits abnormally, the file will be destroyed immediately if the index file refresh time point is less than the maximum message timestamp of the index file.

For ConsumeQueue files, the file recovery mechanism is divided into abnormal stop and normal stop. The abnormal stop is judged by the abort file.

If it is a normal stop, resume from the third from the bottom. If there are less than three, go directly to the first traversal, find the refresh pointer and commit pointer of the current Commitlog log file, and delete all files after this pointer in the ConsumeQueue file.

The rest is to restore the content of the ConsumeQueue file and the IndexFile file by forwarding the message through the offset. This step is the same as the previous one.

Suppose there are 4 Commitlog log files, the third from last is 2, and then start searching from 2. Since it is stopped normally, there is no problem in setting aside 3 Commitlog log files for verification.

If it stops abnormally, directly read the last file, and then traverse forward to the first correctly stored file, and the remaining steps are similar to the normal stop.

It is also assumed that there are 4 Commitlog log files. Since they exit abnormally, I don't know where the problem is, so I can only traverse from the last one to the front until the correct file is found.