RocketMQ -- message consumption process

The last article mentioned that after the message is pulled from the broker, the message will be stored in the processQueue of the PullRequest. At this point, other threads will consume these messages.

Other threads include sequential consumption and concurrent consumption.

Concurrent consumption

In concurrent consumption, the thread used to consume messages is called ConsumeRequest. Each ConsumeRequest consumes 32 messages by default. If more than 32 messages need to be consumed, multiple ConsumeRequests will be created. These ConsumeRequest will be put into the thread pool and wait to run.

When the ConsumeRequest thread runs, it will call the listener to get the result of message consumption. There are two types of concurrent consumption results, RECONSUME_LATER and CONSUME_SUCCESS.

RECONSUME_LATER indicates that re-consumption is required, but in broadcast mode, only warning-level logs are printed. In cluster mode, the message is repackaged into a ConsumeRequest, placed in the thread pool, and consumed after 5s.

At the same time, it is also necessary to tell the broker that the message consumption failed. The broker creates a retry topic and sets the number of message retries. When the message pull exceeds the number of retries, it will enter the DLQ queue.

Which messages need to be re-consumed? For example, there are 10 messages this time, the ackIndex is 2, and the messages with subscripts 3 to 9 must be re-consumed. If the result is CONSUME_LATER, ackIndex will be set to -1, that is, all messages must be re-consumed.

NSUME_SUCCESS indicates that the consumption is successful. Calculate the success and failure information according to the ackIndex.

After the message of RECONSUME_LATER or NSUME_SUCCESS is processed, it is necessary to remove the message from the ProcessQueue and update the message consumption progress in the OffsetStore.

The message consumption progress in the OffsetStore will be submitted to the broker every 5s by a scheduled task, and then the broker will update the offset. Therefore, even if the message consumption is successful, but it is down before it is submitted to the broker, the data will be read from the previous offset next time, resulting in repeated consumption of messages.

sequential consumption

In sequential consumption, a ConsumeRequest is also encapsulated. This ConsumeRequest is different from the ConsumeRequest for concurrent consumption. It does not have a message list, so the message is taken directly from the ProcessQueue.

When a thread consumes, it first acquires a lock. The granularity of this lock is the message queue, that is, a message consumption queue can only be consumed by one thread.

After the lock is acquired, the message is taken out from the ProcessQueue.

If the ProcessQueue has a message, you also need to get the ConsumeLock of the ProcessQueue.

Then consume the message, get the status of the consumption result through the listener, and release the lock.

If the result is SUCCESS, update the message consumption information in the ProcessQueue.

If the result is SUSPEND_CURRENT_QUEUE_A_MOMENT, the number of retries for the message is checked.

If the number of retries exceeds the maximum number of retries allowed, the message will be sent to the broker and stored in the dead letter queue.

If it does not exceed, it will continue to re-consume and accumulate the number of consumptions.

Finally, update the message consumption progress in OffsetStore.

The processing of OffsetStore is the same as concurrent consumption.

After the thread gets the messageQueueLock lock, it will not hold the lock all the time. By default, the lock will be released after consuming 60s.