RocketMQ -- message consumption queue and index file

In the store directory, in addition to the commilog directory, there are consumequeue and index directories.

consumequeue is the message consumption queue storage directory. For example, we have built a TopicTest with four MessageQueues. Under the consumequeue directory, there is a TopicTest directory, and there are four directories 0, 1, 2, and 3 in the TopicTest directory, corresponding to The number of MessageQueues. The file below these numbers is the actual data.

As mentioned earlier, the messages all fall in the commitlog log file. When we consume it, it comes from the topic. If we need to traverse a topic from the commitlog log file one by one, the efficiency is very low. So there is a ConsumeQueue to record the position of each topic in the commitlog.

Another index is the message index file storage directory. The Hash indexing mechanism is used to create an index for messages. RocketMQ writes the message index key and message offset mapping relationship to IndexFile.

Since both ConsumeQueue and IndexFile are associated with commitlog log files, when we write to the commitlog log file, when are ConsumeQueue and IndexFile written?

When the broker starts, it also starts a thread called ReputMessageService, which is responsible for disseminating the update events of the commitlog log. It records the offset from which the commitlog starts to forward messages to ConsumeQueue and IndexFile.

Every 1ms, this thread will go to the commitlog log file through the offset to see if a new message comes in. If there is, it will forward the message. Since it needs to be written to ConsumeQueue and IndexFile, there are two Forwarder.

CommitLogDispatcherBuildConsumeQueue will obtain the corresponding ConsumeQueue file (refer to the above directory tree) according to the topic and the ID of the queue, and write the data into it.

CommitLogDispatcherBuildIndex will build the index key and then write the data.