RocketMQ -- flashing mechanism

The process of sending messages to memory is described above. In order not to lose messages, they need to be persisted to disk. This is divided into synchronous screen refresh and asynchronous screen refresh.

After processing in memory, the lock is released, and the result of processing is called result. So at the stage of refreshing the screen, there may be one or more results that need to be processed.

synchronous brush

The service of synchronously brushing the disk is also a thread, which is specially used to process these results. He maintains a linked list of requestsWrite and requestsRead. requestsWrite is the result that needs to be processed, and requestsRead is the result that is ready to be processed.

When each result needs to be processed by the synchronous brushing service, it will be put into requestsWrite. Since requestsWrite is not thread-safe, the operations of locking and releasing locks are also required here.

As mentioned above, this service is also a thread, and this thread is used to process requestsRead.

This thread will wait up to 10ms after processing requestsRead until it is reawakened.

So when requestsWrite has data, it will wake up this thread, and this thread will exchange the data in requestsWrite and requestsRead.

Since writing requestsWrite is concurrent, there may be multiple results in requestsRead when swapping.

After the exchange, the data in requestsRead starts to be placed on the disk. This thread does not affect the writing of requestsWrite.

Asynchronous brush

Asynchronous brushing, simply put, is to return directly without waiting for the disk to be placed.

Similarly, asynchronous flashing also has a thread service that starts flashing every once in a while. But there is also a small difference here, that is, whether to enable the off-heap memory mechanism.

If not enabled, the message is appended to the memory mapped with the physical file and then written to disk.

If enabled, an off-heap memory of the same size as a commitlog will be created, so messages will be appended to this off-heap memory, then committed to the memory mapped with the physical file, and finally written to disk.

mmap memory map

To improve performance, RocketMQ uses memory-mapped files to improve IO access performance.

The so-called memory mapping is that the physical address and the memory address are mapped, and the data is not actually copied to the memory.

The memory map is limited in size, between 1.5 and 2G, so the file size of Commitlog is 1G, and the file size of ConsumeQueue is 5.72M.