In-depth analysis of RocketMQ source code - load balancing mechanism

I. Introduction

RocketMQ is an excellent distributed message middleware. Its performance is better than the existing message queues in all aspects. RocketMQ adopts the pull mode of long polling by default, and a single machine supports the accumulation of tens of millions of messages, which can be very good. Applied in mass message system.

RocketMQ is mainly composed of components such as Producer, Broker, Consumer, and Namesvr. The Producer is responsible for producing messages, the Consumer is responsible for consuming messages, the Broker is responsible for storing messages, and the Namesvr is responsible for storing metadata. The main functions of each component are as follows:

• Message producer (Producer) : Responsible for producing messages, generally the business system is responsible for producing messages. A message producer will send the message generated in the business application system to the Broker server. RocketMQ provides a variety of sending methods, synchronous sending, asynchronous sending, sequential sending, and one-way sending. Both synchronous and asynchronous modes require the broker to return confirmation information, and one-way sending is not required.
• Message consumer (Consumer) : responsible for consuming messages, generally the background system is responsible for asynchronous consumption. A message consumer pulls messages from the Broker server and serves them to the application. From the perspective of user applications, two consumption forms are provided: pull consumption and push consumption.
• Proxy server (Broker Server) : message relay role, responsible for storing and forwarding messages. The proxy server in the RocketMQ system is responsible for receiving and storing messages sent from producers, and preparing for pull requests from consumers. The proxy server also stores message-related metadata, including consumer groups, consumption progress offsets, and topic and queue messages.
• Name Service (Name Server) : The name service acts as a provider for routing messages. Producers or consumers can look up the Broker IP list corresponding to each topic through the name service. Multiple Namesrv instances form a cluster, but are independent of each other and do not exchange information.
• Producer Group : A collection of producers of the same type, which send messages of the same type with the same sending logic. If a transactional message is sent and the original producer crashes after sending, the Broker server contacts other producer instances of the same producer group to commit or backtrack the consumption.
• Consumer Group : A collection of consumers of the same type, which usually consume the same type of messages and have the same consumption logic. Consumer groups make it very easy to achieve the goals of load balancing and fault tolerance when it comes to message consumption.

The overall message processing logic of RocketMQ is produced and consumed in the Topic dimension, and physically stored in a MessageQueue on a specific Broker. Just because a Topic will have multiple MessageQueues on multiple Broker nodes, messages are naturally generated. Load balancing requirements for production and consumption.

The core of the analysis in this article is to introduce how RocketMQ's message producer (Producer) and message consumer (Consumer) implement load balancing and the implementation details in the entire message production and consumption process.

Second, the overall architecture of RocketMQ

(The picture comes from Apache RocketMQ)

The RocketMQ architecture is mainly divided into four parts, as shown in the figure above:

• Producer : The role of message publishing, which supports distributed cluster deployment. The Producer selects the corresponding Broker cluster queue for message delivery through the MQ load balancing module. The delivery process supports fast failure and low latency.
• Consumer : The role of message consumption, which supports distributed cluster deployment. Supports message consumption in two modes: push and pull. At the same time, it also supports consumption in cluster mode and broadcast mode. It provides a real-time message subscription mechanism, which can meet the needs of most users.
• NameServer : NameServer is a very simple topic routing registration center that supports distributed cluster deployment. Its role is similar to zookeeper in Dubbo, and supports dynamic registration and discovery of Broker.
• BrokerServer : Broker is mainly responsible for message storage, delivery and query, as well as service high availability assurance, and supports distributed cluster deployment.

The physical distribution of RocketMQ topics is shown in the figure above:

Topic is a logical concept of message production and consumption, and the specific message storage is distributed among different Brokers.

The Queue in the Broker is the physical storage unit of the message corresponding to the topic.

In the overall design concept of RocketMQ, the production and consumption of messages are carried out in the Topic dimension, and each Topic will create a corresponding MessageQueue on the Broker node in the RocketMQ cluster.

The process of the producer producing messages is essentially to select all the MessageQueues of the Topic in the Broker and select one of them to send messages according to certain rules. The normal strategy is polling.

The process of consumer consuming messages is essentially that each consumer under the consumerGroup subscribing to the same Topic is responsible for the consumption of the next part of the Topic's MessageQueue according to certain rules.

In the entire message life cycle of RocketMQ, the concept of load balancing is involved in both production and consumption of messages. The message generation process mainly involves the load balancing selected by the Broker, and the message consumption process mainly involves multiple consumers and multiple Brokers. responsible balance.

3. Producer message production process

Producer message production process:

• The producer first accesses the namesvr to obtain routing information, and the namesvr stores all the routing information of the topic dimension (including the queue distribution of each topic in each broker).
• The producer parses the routing information to generate local routing information, parses the topic information in the broker queue and converts it into routing information for local message production.
• The producer sends a message to the Broker according to the local routing information, and selects the specific Broker in the local routing to send the message.

3.1 Route synchronization process

 public class MQClientInstance {
 
    public boolean updateTopicRouteInfoFromNameServer(final String topic) {
        return updateTopicRouteInfoFromNameServer(topic, false, null);
    }
 
 
    public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
        DefaultMQProducer defaultMQProducer) {
        try {
            if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
                try {
                    TopicRouteData topicRouteData;
                    if (isDefault && defaultMQProducer != null) {
                        // 省略对应的代码
                    } else {
                        // 1、负责查询指定的Topic对应的路由信息
                        topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
                    }
 
                    if (topicRouteData != null) {
                        // 2、比较路由数据topicRouteData是否发生变更
                        TopicRouteData old = this.topicRouteTable.get(topic);
                        boolean changed = topicRouteDataIsChange(old, topicRouteData);
                        if (!changed) {
                            changed = this.isNeedUpdateTopicRouteInfo(topic);
                        }
                        // 3、解析路由信息转化为生产者的路由信息和消费者的路由信息
                        if (changed) {
                            TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();
 
                            for (BrokerData bd : topicRouteData.getBrokerDatas()) {
                                this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
                            }
 
                            // 生成生产者对应的Topic信息
                            {
                                TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
                                publishInfo.setHaveTopicRouterInfo(true);
                                Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
                                while (it.hasNext()) {
                                    Entry<String, MQProducerInner> entry = it.next();
                                    MQProducerInner impl = entry.getValue();
                                    if (impl != null) {
                                        impl.updateTopicPublishInfo(topic, publishInfo);
                                    }
                                }
                            }
                            // 保存到本地生产者路由表当中
                            this.topicRouteTable.put(topic, cloneTopicRouteData);
                            return true;
                        }
                    }
                } finally {
                    this.lockNamesrv.unlock();
                }
            } else {
            }
        } catch (InterruptedException e) {
        }
 
        return false;
    }
}

Route synchronization process :

• The route synchronization process is a precondition for the message producer to send the message. Without the synchronization of the route, it is impossible to perceive which Broker node it is sent to.
• Routing synchronization is mainly responsible for querying the routing information corresponding to the specified topic, comparing whether the routing data topicRouteData has changed, and finally converting the routing information into the routing information of the producer and the routing information of the consumer.

 public class TopicRouteData extends RemotingSerializable {
    private String orderTopicConf;
    // 按照broker维度保存的Queue信息
    private List<QueueData> queueDatas;
    // 按照broker维度保存的broker信息
    private List<BrokerData> brokerDatas;
    private HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}
 
 
public class QueueData implements Comparable<QueueData> {
    // broker的名称
    private String brokerName;
    // 读队列大小
    private int readQueueNums;
    // 写队列大小
    private int writeQueueNums;
    // 读写权限
    private int perm;
    private int topicSynFlag;
}
 
 
public class BrokerData implements Comparable<BrokerData> {
    // broker所属集群信息
    private String cluster;
    // broker的名称
    private String brokerName;
    // broker对应的ip地址信息
    private HashMap<Long/* brokerId */, String/* broker address */> brokerAddrs;
    private final Random random = new Random();
}
 
 
--------------------------------------------------------------------------------------------------
 
 
public class TopicPublishInfo {
    private boolean orderTopic = false;
    private boolean haveTopicRouterInfo = false;
    // 最细粒度的队列信息
    private List<MessageQueue> messageQueueList = new ArrayList<MessageQueue>();
    private volatile ThreadLocalIndex sendWhichQueue = new ThreadLocalIndex();
    private TopicRouteData topicRouteData;
}
 
public class MessageQueue implements Comparable<MessageQueue>, Serializable {
    private static final long serialVersionUID = 6191200464116433425L;
    // Topic信息
    private String topic;
    // 所属的brokerName信息
    private String brokerName;
    // Topic下的队列信息Id
    private int queueId;
}

Route resolution process:

• The TopicRouteData core variable QueueData saves the queue information of each Broker, and BrokerData saves the address information of the Broker.
• The TopicPublishInfo core variable MessageQueue holds the most fine-grained queue information.
• The producer is responsible for converting the TopicRouteData obtained from the namesvr into the producer's local TopicPublishInfo.

 public class MQClientInstance {
 
    public static TopicPublishInfo topicRouteData2TopicPublishInfo(final String topic, final TopicRouteData route) {
 
        TopicPublishInfo info = new TopicPublishInfo();
 
        info.setTopicRouteData(route);
        if (route.getOrderTopicConf() != null && route.getOrderTopicConf().length() > 0) {
          // 省略相关代码
        } else {
 
            List<QueueData> qds = route.getQueueDatas();
 
            // 按照brokerName进行排序
            Collections.sort(qds);
 
            // 遍历所有broker生成队列维度信息
            for (QueueData qd : qds) {
                // 具备写能力的QueueData能够用于队列生成
                if (PermName.isWriteable(qd.getPerm())) {
                    // 遍历获得指定brokerData进行异常条件过滤
                    BrokerData brokerData = null;
                    for (BrokerData bd : route.getBrokerDatas()) {
                        if (bd.getBrokerName().equals(qd.getBrokerName())) {
                            brokerData = bd;
                            break;
                        }
                    }
                    if (null == brokerData) {
                        continue;
                    }
                    if (!brokerData.getBrokerAddrs().containsKey(MixAll.MASTER_ID)) {
                        continue;
                    }
 
                    // 遍历QueueData的写队列的数量大小，生成MessageQueue保存指定TopicPublishInfo
                    for (int i = 0; i < qd.getWriteQueueNums(); i++) {
                        MessageQueue mq = new MessageQueue(topic, qd.getBrokerName(), i);
                        info.getMessageQueueList().add(mq);
                    }
                }
            }
 
            info.setOrderTopic(false);
        }
 
        return info;
    }
}

Route generation process:

• The route generation process mainly generates MessageQueue objects according to the BrokerName and writeQueueNums of QueueData.
• MessageQueue is the most fine-grained queue that can send messages selected during the message sending process.

 {
    "TBW102": [{
        "brokerName": "broker-a",
        "perm": 7,
        "readQueueNums": 8,
        "topicSynFlag": 0,
        "writeQueueNums": 8
    }, {
        "brokerName": "broker-b",
        "perm": 7,
        "readQueueNums": 8,
        "topicSynFlag": 0,
        "writeQueueNums": 8
    }]
}

Routing parsing example:

• Topic (TBW102) has queue information on broker-a and broker-b, and the number of read and write queues is 8.
• First, sort the broker information according to the name order of broker-a and broker-b.
• For broker-a, 8 MessageQueue objects with topic TBW102 will be generated, and queueIds are 0-7 respectively.
• For broker-b, 8 MessageQueue objects with topic TBW102 will be generated, and queueIds are 0-7 respectively.
• TopicPublishInfo of topic (named TBW102) contains 16 MessageQueue objects as a whole, including 8 MessageQueues of broker-a and 8 MessageQueues of broker-b.
• The routing in the message sending process is to obtain one of the 16 MessageQueue objects for message sending.

3.2 Load Balancing Process

 public class DefaultMQProducerImpl implements MQProducerInner {
 
    private SendResult sendDefaultImpl(
        Message msg,
        final CommunicationMode communicationMode,
        final SendCallback sendCallback,
        final long timeout
    ) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
        
        // 1、查询消息发送的TopicPublishInfo信息
        TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
 
        if (topicPublishInfo != null && topicPublishInfo.ok()) {
             
            String[] brokersSent = new String[timesTotal];
            // 根据重试次数进行消息发送
            for (; times < timesTotal; times++) {
                // 记录上次发送失败的brokerName
                String lastBrokerName = null == mq ? null : mq.getBrokerName();
                // 2、从TopicPublishInfo获取发送消息的队列
                MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
                if (mqSelected != null) {
                    mq = mqSelected;
                    brokersSent[times] = mq.getBrokerName();
                    try {
                        // 3、执行发送并判断发送结果，如果发送失败根据重试次数选择消息队列进行重新发送
                        sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
                        switch (communicationMode) {
                            case SYNC:
                                if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
                                    if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
                                        continue;
                                    }
                                }
 
                                return sendResult;
                            default:
                                break;
                        }
                    } catch (MQBrokerException e) {
                        // 省略相关代码
                    } catch (InterruptedException e) {
                        // 省略相关代码
                    }
                } else {
                    break;
                }
            }
 
            if (sendResult != null) {
                return sendResult;
            }
        }
    }
}

Message sending process:

• Query the routing information object TopicPublishInfo corresponding to the topic.
• Obtain the queue for sending messages from TopicPublishInfo through selectOneMessageQueue, and the queue representative falls into the specific Broker's queue queue.
• Execute the sending and judge the sending result. If the sending fails, select the message queue for re-sending according to the number of retries. The re-selecting the queue will avoid the queue of the Broker that failed to send the last time.

 public class TopicPublishInfo {
 
    public MessageQueue selectOneMessageQueue(final String lastBrokerName) {
        if (lastBrokerName == null) {
            return selectOneMessageQueue();
        } else {
            // 按照轮询进行选择发送的MessageQueue
            for (int i = 0; i < this.messageQueueList.size(); i++) {
                int index = this.sendWhichQueue.getAndIncrement();
                int pos = Math.abs(index) % this.messageQueueList.size();
                if (pos < 0)
                    pos = 0;
                MessageQueue mq = this.messageQueueList.get(pos);
                // 避开上一次上一次发送失败的MessageQueue
                if (!mq.getBrokerName().equals(lastBrokerName)) {
                    return mq;
                }
            }
            return selectOneMessageQueue();
        }
    }
}

Routing process:

• The selection of MessageQueue is based on polling, and the global maintenance index is used to accumulate and modulo select the sending queue.
• During the selection process of MessageQueue, the MessageQueue corresponding to the Broker that failed to send the last time will be avoided.

Producer message sending diagram :

• The queue distribution of a topic is Broker\_A\_Queue1, Broker\_A\_Queue2, Broker\_B\_Queue1, Broker\_B\_Queue2, Broker\_C\_Queue1, Broker\_C\_Queue2, which are selected in sequence according to the polling strategy.
• In the scenario of sending failure, such as Broker\_A\_Queue1 fails to send, then it will skip Broker\_A and select Broker\_B_Queue1 to send.

Fourth, the consumer message consumption process

Consumer message consumption process :

• The consumer accesses the routing information corresponding to the namesvr synchronization topic.
• The consumer parses the remote routing information locally and saves it locally.
• The consumer performs Reblance load balancing locally to determine the MessageQueue that this node is responsible for consuming.
• The consumer accesses the Broker to consume the messages of the specified MessageQueue.

4.1 Route synchronization process

 public class MQClientInstance {
 
    // 1、启动定时任务从namesvr定时同步路由信息
    private void startScheduledTask() {
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
 
            @Override
            public void run() {
                try {
                    MQClientInstance.this.updateTopicRouteInfoFromNameServer();
                } catch (Exception e) {
                    log.error("ScheduledTask updateTopicRouteInfoFromNameServer exception", e);
                }
            }
        }, 10, this.clientConfig.getPollNameServerInterval(), TimeUnit.MILLISECONDS);
    }
 
    public void updateTopicRouteInfoFromNameServer() {
        Set<String> topicList = new HashSet<String>();
 
        // 遍历所有的consumer订阅的Topic并从namesvr获取路由信息
        {
            Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
            while (it.hasNext()) {
                Entry<String, MQConsumerInner> entry = it.next();
                MQConsumerInner impl = entry.getValue();
                if (impl != null) {
                    Set<SubscriptionData> subList = impl.subscriptions();
                    if (subList != null) {
                        for (SubscriptionData subData : subList) {
                            topicList.add(subData.getTopic());
                        }
                    }
                }
            }
        }
 
        for (String topic : topicList) {
            this.updateTopicRouteInfoFromNameServer(topic);
        }
    }
 
    public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
        DefaultMQProducer defaultMQProducer) {
 
        try {
            if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
                try {
                    TopicRouteData topicRouteData;
                    if (isDefault && defaultMQProducer != null) {
                        // 省略代码
                    } else {
                        topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
                    }
 
                    if (topicRouteData != null) {
                        TopicRouteData old = this.topicRouteTable.get(topic);
                        boolean changed = topicRouteDataIsChange(old, topicRouteData);
                        if (!changed) {
                            changed = this.isNeedUpdateTopicRouteInfo(topic);
                        }
 
                        if (changed) {
                            TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();
 
                            for (BrokerData bd : topicRouteData.getBrokerDatas()) {
                                this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
                            }
 
                            // 构建consumer侧的路由信息
                            {
                                Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
                                Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
                                while (it.hasNext()) {
                                    Entry<String, MQConsumerInner> entry = it.next();
                                    MQConsumerInner impl = entry.getValue();
                                    if (impl != null) {
                                        impl.updateTopicSubscribeInfo(topic, subscribeInfo);
                                    }
                                }
                            }
     
                            this.topicRouteTable.put(topic, cloneTopicRouteData);
                            return true;
                        }
                    }
                } finally {
                    this.lockNamesrv.unlock();
                }
            }
        } catch (InterruptedException e) {
        }
 
        return false;
    }
}

Route synchronization process :

• The routing synchronization process is a precondition for message consumers to consume messages. Without the synchronization of routing, the broker node cannot perceive the specific message to be consumed.
• The consumer node periodically synchronizes the routing information of the topic subscribed by the consumer node from the namesvr through the scheduled task.
• The consumer constructs the synchronized routing information into local routing information through updateTopicSubscribeInfo and uses it for subsequent balancing.

4.2 Load Balancing Process

 public class RebalanceService extends ServiceThread {
 
    private static long waitInterval =
        Long.parseLong(System.getProperty(
            "rocketmq.client.rebalance.waitInterval", "20000"));
 
    private final MQClientInstance mqClientFactory;
 
    public RebalanceService(MQClientInstance mqClientFactory) {
        this.mqClientFactory = mqClientFactory;
    }
 
    @Override
    public void run() {
 
        while (!this.isStopped()) {
            this.waitForRunning(waitInterval);
            this.mqClientFactory.doRebalance();
        }
 
    }
}

Load balancing process :

• The consumer periodically re-balances the load through the RebalanceService.
• The core of RebalanceService is to complete the distribution relationship between MessageQueue and consumer.

 public abstract class RebalanceImpl {
 
    private void rebalanceByTopic(final String topic, final boolean isOrder) {
        switch (messageModel) {
            case BROADCASTING: {
                // 省略相关代码
                break;
            }
            case CLUSTERING: { // 集群模式下的负载均衡
                // 1、获取topic下所有的MessageQueue
                Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
 
                // 2、获取topic下该consumerGroup下所有的consumer对象
                List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
                 
                // 3、开始重新分配进行rebalance
                if (mqSet != null && cidAll != null) {
                    List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
                    mqAll.addAll(mqSet);
 
                    Collections.sort(mqAll);
                    Collections.sort(cidAll);
 
                    AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
 
                    List<MessageQueue> allocateResult = null;
                    try {
                        // 4、通过分配策略重新进行分配
                        allocateResult = strategy.allocate(
                            this.consumerGroup,
                            this.mQClientFactory.getClientId(),
                            mqAll,
                            cidAll);
                    } catch (Throwable e) {
 
                        return;
                    }
 
                    Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
                    if (allocateResult != null) {
                        allocateResultSet.addAll(allocateResult);
                    }
                    // 5、根据分配结果执行真正的rebalance动作
                    boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
                    if (changed) {
                        this.messageQueueChanged(topic, mqSet, allocateResultSet);
                    }
                }
                break;
            }
            default:
                break;
        }
    }

Reassignment Process :

• Get all MessageQueues under topic.
• Get the cid of all consumers under the consumerGroup under the topic (such as 192.168.0.8@15958).
• Sort mqAll and cidAll. The sorting order of mqAll is based on BrokerName and then BrokerId, and cidAll is sorted by string.
• by allocation strategy
• AllocateMessageQueueStrategy reallocate.
• Perform the real rebalance action based on the allocation result.

 public class AllocateMessageQueueAveragely implements AllocateMessageQueueStrategy {
    private final InternalLogger log = ClientLogger.getLog();
 
    @Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
         
        List<MessageQueue> result = new ArrayList<MessageQueue>();
         
        // 核心逻辑计算开始
 
        // 计算当前cid的下标
        int index = cidAll.indexOf(currentCID);
         
        // 计算多余的模值
        int mod = mqAll.size() % cidAll.size();
 
        // 计算平均大小
        int averageSize =
            mqAll.size() <= cidAll.size() ? 1 : (mod > 0 && index < mod ? mqAll.size() / cidAll.size()
                + 1 : mqAll.size() / cidAll.size());
        // 计算起始下标
        int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
        // 计算范围大小
        int range = Math.min(averageSize, mqAll.size() - startIndex);
        // 组装结果
        for (int i = 0; i < range; i++) {
            result.add(mqAll.get((startIndex + i) % mqAll.size()));
        }
        return result;
    }
    // 核心逻辑计算结束
 
    @Override
    public String getName() {
        return "AVG";
    }
}
 
------------------------------------------------------------------------------------
 
rocketMq的集群存在3个broker，分别是broker_a、broker_b、broker_c。
 
rocketMq上存在名为topic_demo的topic，writeQueue写队列数量为3，分布在3个broker。
排序后的mqAll的大小为9，依次为
[broker_a_0  broker_a_1  broker_a_2  broker_b_0  broker_b_1  broker_b_2  broker_c_0  broker_c_1  broker_c_2]
 
rocketMq存在包含4个consumer的consumer_group，排序后cidAll依次为
[192.168.0.6@15956  192.168.0.7@15957  192.168.0.8@15958  192.168.0.9@15959]
 
192.168.0.6@15956 的分配MessageQueue结算过程
index：0
mod：9%4=1
averageSize：9 / 4 + 1 = 3
startIndex：0
range：3
messageQueue：[broker_a_0、broker_a_1、broker_a_2]
 
 
192.168.0.6@15957 的分配MessageQueue结算过程
index：1
mod：9%4=1
averageSize：9 / 4 = 2
startIndex：3
range：2
messageQueue：[broker_b_0、broker_b_1]
 
 
192.168.0.6@15958 的分配MessageQueue结算过程
index：2
mod：9%4=1
averageSize：9 / 4 = 2
startIndex：5
range：2
messageQueue：[broker_b_2、broker_c_0]
 
 
192.168.0.6@15959 的分配MessageQueue结算过程
index：3
mod：9%4=1
averageSize：9 / 4 = 2
startIndex：7
range：2
messageQueue：[broker_c_1、broker_c_2]

Distribution strategy analysis:

• For the overall allocation strategy, you can refer to the specific example in the figure above to better understand the allocation logic.

The allocation of consumers :

• Consumer objects under the same consumerGroup will be assigned to different MessageQueues under the same Topic.
• Each MessageQueue will eventually be assigned to a specific consumer.

5. RocketMQ specifies machine consumption design ideas

In the daily test environment, there will be multiple consumers for consumption, but in actual development, after a new function is added to a consumer, it is hoped that messages will only be consumed by this machine for logical coverage. At this time, the cluster mode of consumerGroup will cause us trouble, because The reason for the consumption load balancing is not sure which consumer the message is consumed by. Of course, we can achieve specified machine consumption by intervening in the consumer's load balancing mechanism.

 public class AllocateMessageQueueAveragely implements AllocateMessageQueueStrategy {
    private final InternalLogger log = ClientLogger.getLog();
 
    @Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
        List<String> cidAll) {
        
 
        List<MessageQueue> result = new ArrayList<MessageQueue>();
        // 通过改写这部分逻辑，增加判断是否是指定IP的机器，如果不是直接返回空列表表示该机器不负责消费
        if (!cidAll.contains(currentCID)) {
            return result;
        }
 
        int index = cidAll.indexOf(currentCID);
        int mod = mqAll.size() % cidAll.size();
        int averageSize =
            mqAll.size() <= cidAll.size() ? 1 : (mod > 0 && index < mod ? mqAll.size() / cidAll.size()
                + 1 : mqAll.size() / cidAll.size());
        int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
        int range = Math.min(averageSize, mqAll.size() - startIndex);
        for (int i = 0; i < range; i++) {
            result.add(mqAll.get((startIndex + i) % mqAll.size()));
        }
        return result;
    }
}

The consumer load balancing strategy is rewritten :

• By rewriting the load balancing strategy AllocateMessageQueueAveragely's allocate mechanism to ensure that only machines with the specified IP can consume.
• Judging by IP is based on RocketMQ's cid format of 192.168.0.6@15956, where the preceding IP address is the IP address of the consumer machine. The entire solution is feasible and can actually be implemented.

6. Summary

This article mainly introduces the load balancing mechanism of RocketMQ in the production and consumption process. Combining the source code and actual cases, it strives to give readers an easy-to-understand technology popularization. I hope it can be of reference and reference value for readers. Due to the length of the article, some aspects are not covered, and many technical details are not elaborated in detail. If you have any questions, please continue to communicate.

Author: vivo internet server team - Wang Zhi