RocketMQ -- message pull

As mentioned in the previous article, due to the message queue load mechanism, one PullRequest will be put into the pullRequestQueue queue.

These PullRequests will have a dedicated thread, which will be taken out and packaged into a Request on the server side and sent to the broker.

Before sending the server, you need to know where the broker's address is, what offset to start pulling from (the offset is stored in the broker in cluster mode, and the local in broadcast mode), and how much data to pull at one time.

After the broker receives the request, it will take out the required message from the commitlog according to the offset. Due to the master-slave synchronization of the broker, when the message is returned here, it will also inform that the next pull is from the master broker. Pull or slave broker to pull data.

When pulling messages from commit, there are several situations (the offsetCheckInSlave judgment is ignored below):

1. If there is no message in the current message queue, the next time the message is pulled, if the broker is the master node, it is still this offset. If this broker is a slave node, it will start directly from 0 next time.

2. The current message queue has messages, but the offset we pull is smaller than the minimum offset in the queue. For example, we need to pull 100 data, but the minimum offset of the message queue is 500. The next time the message is pulled, if the broker is the master node, it is still this offset. If this broker is a slave node, it will start directly from 500 next time.

3. If the offset we pull is exactly equal to the largest offset in the message queue, so we have no data to consume, then the next time we pull, it will still be the offset of 100.

4. If the offset we pull is greater than the largest offset in the message queue, so we have no data to consume, then the next time we pull, we also need to determine whether the smallest offset in the message queue is equal to 0.

5. Under normal circumstances, the message we need is between the smallest offset and the largest offset in the message queue, then directly take the data out of the commitlog and return it.

After the consumer receives the broker response, it will update it according to the offsets of the above situations, and store the message in the processQueue of the PullRequest.

And then put the PullRequest into the pullRequestQueue queue and wait for the next pull.

If the messages of the processQueue are not consumed in time, they will keep accumulating, so when PullRequest pulls messages, it will first determine whether the number of messages in the processQueue has exceeded 1000. , you can continue to pull after 50ms.

In addition, it will also judge the size of the message in the processQueue, the distance between the maximum offset and the minimum offset of the queue in the processQueue, and if it exceeds the threshold, it will also be placed in the pullRequestQueue queue, and it can continue to be pulled after 50ms.