RocketMQ -- written before message pull

When consumers consume, they need to specify which Topic messages to consume, such as TopicTest.

Like the producer, the consumer also needs to go to the NameServer to pull the metadata information corresponding to the TopicTest.

As shown in the figure above, TopicTest corresponds to these two brokers, and each broker has two MessageQueues. Where do consumers pull data from?

For example, we have two consumers. When the consumer starts, it will send a heartbeat packet to each broker, so that each broker has consumer information, and we can also get all consumer information from any broker.

Now that you have metadata information and consumer information, you can use the allocation algorithm to know which MessageQueue the current consumer goes to to pull data.

RocketMQ currently provides 6 allocation algorithms by default, including average allocation (AllocateMessageQueueAveragely), average polling allocation (AllocateMessageQueueAveragelyByCircle), consistent hash (AllocateMessageQueueConsistentHash), according to configuration (AllocateMessageQueueByConfig), according to the Broker deployment room name (AllocateMessageQueueByMachineRoom), the nearest machine room mode ( AllocateMachineRoomNearby).

The default is an even distribution, for example, there are 4 MessageQueues, 2 for each of the two consumers. Consumer 1 allocates MessageQueue0 and MessageQueue1, and consumer 2 allocates MessageQueue2 and MessageQueue3.

If it is an average round-robin allocation, then consumer 1 allocates 1, consumer 2 allocates 1, consumer 1 allocates another, consumer 2 allocates another, and finally consumer 1 allocates MessageQueue0 and MessageQueue2, consumer 2 Assign MessageQueue1 and MessageQueue3.

Let's move on to the even distribution, taking Consumer 1 as an example.

At this point, it is known that consumers consume from MessageQueue0 and MessageQueue1, so where do they start to consume?

The consumption mode is further divided into consumption from the latest offset of the queue (CONSUME_FROM_LAST_OFFSET), consumption from scratch (CONSUME_FROM_FIRST_OFFSET), consumption from the consumption progress corresponding to the timestamp initiated by the consumer (CONSUME_FROM_TIMESTAMP).

Based on the above information, we can integrate a request information PullRequest, including consumer group, offset, MessageQueue, ProcessQueue (this will be explained later), and store it in the pullRequestQueue linked list.