RocketMQ - Theory

RocketMQ is an open source message middleware with pure Java, distributed, and queue model. It is a message middleware with a queue model developed by Alibaba with reference to the characteristics of Kafka, and is later open sourced to the Apache Foundation.

I have written articles on RabbitMQ before. After all, I am preconceived. When introducing the functions of RocketMQ, I may interspersed with RabbitMQ for comparison. The current message middleware selection in the company has also changed from RabbitMQ to RocketMQ. The reason the technical director told me is also very simple, because RocketMQ's distributed cluster has higher availability and simpler operation and maintenance.

It is true that aside from Kafka, which was born for big data, there is a big gap between the two in terms of architecture and usage, but there is not much difference in performance and effect in practical applications. Some people say that the response speed of RabbitMQ is faster, and some people say that the data throughput of RocketMQ is higher, but the gap is not big, and each has its own advantages. There are not many companies in China that have such a large volume and are so serious about performance.

1. Basic Components

1.1. Noun concept

Name Server

The Name Server is the coordinator of the RocketMQ cluster, and each component of the cluster obtains various attributes and address information through the Name Server. The main function consists of two parts:

1. Each Broker regularly reports its own status information to the Name Server to maintain the heartbeat.
2. Each client, including Producer, Consumer, and command-line tools, obtains the latest status information such as Broker through Name Server.

So, before starting Brokers, producers and consumers, they must tell them the address of the Name Server. To improve reliability, it is recommended to start multiple Name Servers to form a cluster and deploy them separately. Therefore, in the production line, the number of nodes in the Name Server cluster can be dynamically increased or decreased.

Name Server can be compared to ZooKeeper in Kafka, so why not use ZooKeeper directly? Because RocketMQ can only use a small part of ZooKeeper's functions, it will be too heavy to use directly. In comparison, we have developed a more lightweight Name Server that meets its own characteristics.

Broker

Broker is mainly responsible for the storage, delivery and query of messages, as well as service high availability assurance. To put it bluntly, it is the server of RocketMQ. Broker is the core of middleware, it must not be hung, and its reliability must be guaranteed. Usually, a master-slave high-availability architecture is built. Therefore, Broker is divided into Master Broker (BrokerId is 0) and Slave Broker (BrokerId is not 0).

Each Broker establishes long-term connections with all nodes in the Name Server cluster, and regularly registers topic information to all Name Servers. After the Broker is started, it needs to complete an operation of registering itself to the Name Server; then it reports topic routing information to the Name Server periodically every 30s.

Name Server cluster nodes do not communicate with each other, so all nodes need to be reported when reporting information. In addition, the Name Server is stateless, that is, the data will not be persistently stored, all stored in memory, and will disappear after restarting.

Producer

Establish a long link (Keep-alive) with one of the nodes (randomly) in the Name Server cluster, periodically read topic routing information from Name Server, establish a long link to the Master Broker that provides Topic services, and regularly send heartbeats to the Master Broker .

Consumer

Establish a long connection with one of the nodes (randomly) in the Name Server cluster, regularly pull topic routing information from Name Server, establish long connections to the Master Broker and Slave Broker that provide Topic services, and regularly send messages to the Master Broker and Slave Broker heartbeat. Consumers can subscribe to messages from both the Master Broker and the Slave Broker. The subscription rules are determined by the Broker configuration.

Topic

The message topic, the first-level message classification, classifies different business messages through the Topic.

Tag

The message label is used to further distinguish the message classification under a topic, and the message is sent from the producer and the attribute is carried.

Queue

For RocketMQ, Topic is a logical concept. A Topic can be distributed on each Broker, and a subset of a Topic distributed on a Broker is defined as a Topic shard. On the basis of sharding, one of the parts is divided into several parts (the number of parts can be specified), which is the Queue queue.

Queue is the basic unit of resource allocation in the load balancing process.

1.2. Consumer Group and Queue

Consumer Group Consumer Group

When actually consuming messages, you need to declare the topic name and consumer group name. In the cluster consumption mode , the messages in the same topic will be distributed to different consumers in the same Consumer Group to achieve the effect of load balancing.

How to understand different Consumers in the same Consumer Group? It is considered that different business systems consume messages in the same topic based on the same Consumer Group; in order to increase the concurrency, it is also considered to directly expand the process of a business system horizontally (such as adding several pods to k8s).

But if you just add a few threads in the code of a certain Consumer, such as @RocketMQMessageListener.consumeThreadMax , it is not considered to increase the Consumer. Just as a Consumer, after pulling a batch of messages, add threads to execute concurrently.

read and write queue

When creating or changing a topic, you will be required to set the number of read queues and write queues.
When sending messages, the routing information will be returned according to the number of topic write queues; when consuming messages, routing information will be returned according to the number of topic read queues.

Read and write queues are not physically opposite queues, such as:

• has 8 write queues and 4 read queues: will create 8 folders (0, 1, 2, 3, 4, 5, 6, 7), and messages will be sent to these 8 queues, but only when consuming Consume to 4 queues (0, 1, 2, 3), and messages in the other 4 queues (4, 5, 6, 7) will not be consumed.
• has 4 write queues and 8 read queues: messages will only be sent to 4 queues (0, 1, 2, 3), and messages will be consumed from 8 queues when consuming, but only (0, 1, 2, 3) There are messages in the queue. If a consumer is assigned a queue (4, 5, 6, 7), nothing will be received.

From this point of view, the best way is number of read queues = number of write queues, so why does RocketMQ need to do this? In order to facilitate the expansion and contraction of the queue.

A topic has created 128 queues on each broker. Now the queues need to be scaled down to 64. What can I do to prevent 100% message loss without restarting the application? Best practice: shrink the write queue 128->64 first, and shrink the write queue from 0 1 2 ...... 127 to 0 1 2 ...... 63. Wait until all messages in 64 65 66...127 are consumed, and then shrink the read queue 128->64. If the write queue and read queue are scaled down at the same time, some messages may not be consumed.

Consumer Group and Queue about load balancing

Since it is load balancing, the discussion is about the cluster consumption model. I said before, Queue is the basic unit of load balancing process of resource allocation .

Therefore, within a Consumer Group, Consumer and Queue have a 1:n relationship:

• A Queue can only be assigned to at most one Consumer.
• A Consumer can be allocated multiple Queues.

Therefore, the number of Consumers should be less than or equal to the number of Queues. If the number of Consumers exceeds the number of Queues, the redundant Consumers will not be able to consume messages.

2. High Availability Architecture

The noun concept of each component of RocketMQ has been introduced before, so now let's talk about how these components are built into the architecture of RocketMQ.

2.1. Name Server Cluster

It is an almost stateless node that can be deployed in a cluster, and the cluster nodes are independent of each other without information exchange. Its function is mainly to update and discover Broker services, through which producers or consumers can find the corresponding Broker IP list for each topic

As mentioned before, the Name Server is independent, and each Name Server will have complete cluster routing information, including information of all Broker nodes, our data information, and so on. So as long as any NamseServer survives, the normal operation of RocketMQ information can be saved without failure.

Therefore, in order to improve availability, the number of nodes of the Name Server should be at least 2 or more. Although it can be deployed directly on the machine where the Broker is located, it is better to deploy it separately if possible.

2.2. Broker cluster

There are many types of Broker clusters that can be built, which are divided according to their functionality:

• single-node mode : just one Master Broker.
• master-slave mode : Each Master Broker is equipped with multiple Slave Brokers. Only Master accepts Topic to create Queue, message writing, etc. Slave just synchronizes the data on Master. However, there are synchronous/asynchronous points. (1) Synchronous, only when the message is synchronized from the Master to the Slave, the message is sent successfully; (2) Asynchronous, the message is sent to the Master even if it is sent successfully, and subsequent messages are asynchronously refreshed from the Master to the Slave.
• Multi-Master mode : It can be multiple Masters or multiple Master-Slave masters and slaves. Multiple Masters can improve message concurrency and high availability, which is why Topic creates fragmented queues on each Master Broker .
• Dledger mode : In the master-slave mode, the Slave hangs up and has little effect, but if the Master hangs up, it will not work. Unless you manually switch a Slave Broker to the new Master Broker. Dledger solves this problem. It can monitor a group of Brokers. When the Master hangs up, it will re-elect a new Master from the remaining Slaves.

To sum up, the most perfect high-availability cluster architecture is: multiple Masters, each Master is configured with multiple slaves, and all master-slave brokers have Dledger enabled.

After reading this, the first reaction is that this is similar to the high-availability cluster architecture of Redis. The previous multi-Master multi-slave is basically the same, and then RocketMQ's Dledger mode corresponds to the Sentinel sentinel mode in Redis. It seems that the distributed development to today, the solution for the high-availability architecture is gradually stable and unified.

2.3. Dledger Mode

DLedger is a CommitLog repository implementation based on the Raft algorithm in OpenMessaging. Starting from RocketMQ 4.5.0 version , RocketMQ introduced the DLedger mode to solve the problem of automatic failover within the Broker group. RocketMQ-on-DLedger Group is now the most common way to deploy RocketMQ clusters.

RocketMQ-on-DLedger Group refers to a group of Brokers with the same name, which requires at least 3 nodes. One leader is automatically elected through Raft, and the remaining nodes are used as followers, and data is replicated between the leader and the follower to ensure high availability.

RocketMQ-on-DLedger Group can automatically switch disaster recovery and ensure data consistency.

RocketMQ-on-DLedger Group can be scaled horizontally, that is, any number of RocketMQ-on-DLedger Groups can be deployed to provide external services at the same time.

When deploying based on RocketMQ-on-DLedger Group, the configuration file of each Broker node needs to add more configuration, of course, it is still for RocketMQ 4.5.0 and above:

enableDLegerCommitLog：是否启动 DLedger
dLegerGroup：DLedger Raft Group 的名字，建议和 brokerName 保持一致
dLegerPeers：DLedger Group 内各节点的地址与端口信息（同一个 Group 内的各个节点配置必须要保证一致）
dLegerSelfId：节点 id, 必须属于 dLegerPeers 中的一个；同 Group 内各个节点要唯一，例如：第一个节点配置为”n0”，第二个节点配置为”n1”，第三个节点配置为”n2”
sendMessageThreadPoolNums：发送线程个数（建议配置成 CPU 核数）

2.4. Comparison with RabbitMQ

At the beginning, our technical director chose RabbitMQ because of the high availability cluster, so we can review what the RabbitMQ cluster looks like.

Ordinary cluster

Start multiple RabbitMQ instances on multiple machines, one per machine. The created Queue will only be placed on one RabbitMQ instance, but each instance synchronizes the metadata of the Queue (metadata can be considered as some configuration information of the Queue, through the metadata, the instance where the Queue is located can be found). When consuming, in fact, if it is connected to another instance, then that instance will pull data from the instance where the Queue is located.

The problem is that if the consumer reads the Queue where it is not the owning instance, it has to pull data from the original instance, which has performance overhead. But if it happens to be the actual instance of Queue, what is the difference between it and a single node, there is still a performance bottleneck of a single node.

Moreover, this cluster mode can only improve part of the concurrency, and there is no high availability. Because other instances only synchronize the metadata of the Queue, if the instance where the Queue is located goes down, the messages in the Queue will still not be pulled.

mirror cluster

Mirror clusters are changes made in the former mode. The created Queue is still only placed on one instance, but each other instance not only synchronizes the metadata of the Queue, but also synchronizes the message data.

In this way, high availability is achieved, the instance to which the Queue belongs is down, and unconsumed messages can still be read from other instances.

But this is equivalent to saving all queued messages on each instance, not to mention performance, just how big is the disk requirement.

RocketMQ, RabbitMQ vs

In contrast, RocketMQ's high-availability architecture is more scientific:

• Topic is sharded on different Master Brokers, and Queues are scattered in different clusters. Although RocketMQ's master-slave synchronization is similar to mirroring, it also synchronizes the Queue's metadata and messages. But after all, it is only the split topic data, and the amount is not so large.
• Separation of reading and writing. If you want to expand write concurrency, expand Master, and if you want to expand read concurrency, expand Slave, which is more flexible.
• Each instance of RocketMQ has its own division of labor, and even has an independent Name Server, so the performance consumption of a single cluster instance is not large. For RabbitMQ, each instance may have a performance bottleneck, which may require higher physical resources on the machine.

3. Other

3.1. External plugins

console console

RocketMQ does not come with a console like RabbitMQ, but it provides an API to the outside world. At present, there are many front-end projects in the community that can access the RocketMQ console. At present, I use rocketmq-console-ng , which is quite good.

access to Prometheus

After all, the console has limited monitoring. The most widely used monitoring solution is Prometheus. RocketMQ also provides access solutions.

RocketMQ-Exporter is mentioned here, which has been officially included by Prometheus, and its address is https://github.com/apache/rocketmq-exporter . It first collects data from the RocketMQ cluster, and then normalizes the collected data into data that meets the requirements of the Prometheus system with the help of the third-party client library provided by Prometheus. Prometheus can periodically pull data from the Exporter.

3.2. Comparison of RocketMQ and RabbitMQ

build operation and maintenance on

During operation and maintenance, RabbitMQ is much simpler. There is only one server in the whole process, and it also includes the console console.

The same situation is relatively complicated on RocketMQ. It is necessary to build Name Server and Broker (for Master and Slave) respectively, and to build the console by yourself.

But complexity has complex benefits, which are reflected in high-availability clusters, so I won't go into details here.

development use on

Compared with Kafka and RocketMQ, the AMQP protocol of RabbitMQ is much more expensive to learn. Whether sending ordinary messages or more complex ones like delayed messages, you need to understand and use the cooperation between exchanges and queues. When used on RocketMQ, it is much simpler and cannot provide an API. The framework encapsulates many implementation details.

Some people say that the RocketMQ framework itself is heavy, but it has heavy benefits. The more the framework encapsulates, the more convenient it is for those who use it. Like sequential messages, deferred messages, transactional messages, etc., almost out of the box. I'm learning to drive recently, and I feel that RabbitMQ is like a manual transmission, while RocketMQ is like an automatic transmission. I don't know if this analogy is appropriate.

3.3. docker compose stand-alone installation

docker-compose.yml

version: '3.5'
services:
  rmqnamesrv:
    image: rocketmqinc/rocketmq:4.4.0
    container_name: rmqnamesrv
    restart: always
    ports:
      - 9876:9876
    environment:
    #内存分配
      JAVA_OPT_EXT: "-server -Xms1g -Xmx1g"
    volumes:
      - /Volumes/rocketmq/namesrv/logs:/root/logs
    command: sh mqnamesrv
    networks:
      rmq:
        aliases:
          - rmqnamesrv
          
  rmqbroker:
    image: rocketmqinc/rocketmq:4.4.0
    container_name: rmqbroker
    restart: always
    depends_on:
      - rmqnamesrv
    ports:
      - 10909:10909
      - 10911:10911
    volumes:
      - /Volumes/rocketmq/broker/logs:/root/logs
      - /Volumes/rocketmq/broker/store:/root/store
      - /Volumes/rocketmq/broker/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf
    command: sh mqbroker  -c /opt/rocketmq-4.4.0/conf/broker.conf
    environment:
      NAMESRV_ADDR: "rmqnamesrv:9876"
      JAVA_OPT_EXT: "-server -Xms1g -Xmx1g -Xmn1g"
    networks:
      rmq:
        aliases:
          - rmqbroker
          
  rmqconsole:
    image: styletang/rocketmq-console-ng
    container_name: rocketmq-console
    restart: always
    ports:
      - 9877:8080
    depends_on:
      - rmqnamesrv
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /Volumes/rocketmq/console/logs:/root/logs
    environment:
      JAVA_OPTS: "-Drocketmq.namesrv.addr=rmqnamesrv:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false"
    networks:
      rmq:
        aliases:
          - rmqconsole
          
networks:
  rmq:
    name: rmq
    driver: bridge