Take you to play with message queues and related selections!

Introduction | This article mainly introduces the basic principles and selection comparisons related to Kafka, RabbitMQ, Pulsar, and RocketMQ. I hope readers who are interested in this direction can provide some experience and help.

First, the message queue

(1) Message queue usage scenarios

The message queue middleware is an important component in the distributed system. It mainly solves problems such as application coupling, asynchronous messages, peak shaving and valley filling. Implement a high-performance, high-availability, scalable, and eventually consistent architecture.

Decoupling: Multiple services monitor and process the same message, avoiding multiple RPC calls.

Asynchronous messages: message publishers do not have to wait for the result of message processing.

Peak shaving and valley filling: Large traffic, write scenarios, and anti-traffic for downstream I/O services. Of course, other solutions need to be used under high traffic.

Message-driven framework: In the event bus, the service drives the service to complete the corresponding action by listening to the event message.

(2) Message queue mode

Peer-to-peer mode, non-repeatable consumption

Multiple producers can send messages to the same message queue. After a message is successfully consumed by a messager, the message will be removed, and other consumers cannot process the message. If the consumer fails to process a message, the message will be re-consumed.

publish/subscribe model

The publish-subscribe model requires registration and subscription, and corresponding messages are consumed according to the registration. Multiple producers can write messages to the same topic, and multiple messages can be consumed by the same consumer. A message produced by a producer can also be consumed by multiple consumers, as long as they have subscribed to the message.

Second, the selection reference

Message order: whether the order of consumption of messages sent to the queue can be guaranteed when consumed.

Scaling: When there is a problem with the performance of the message queue, for example, the consumption is too slow, can the storage capacity be quickly supported; when there are too many consumption queues and system resources are wasted, whether it can support scaling.

Message retention: whether the message will continue to be retained in the message queue after successful consumption.

Fault tolerance: When a message fails to be consumed, is there some mechanism to ensure that the message is a successful one, such as an asynchronous third-party refund message, it is necessary to ensure that the message is consumed before the refund to the user can be determined successfully, so The accuracy of the successful consumption of this message must be guaranteed.

Message reliability: whether there will be lost messages, for example, there are two messages A/B, and finally only the B message can be consumed, and the A message is lost.

Message timing: mainly includes "message survival time" and "delayed message".

Throughput: The highest number of concurrency supported.

Message routing: According to the routing rules, only subscribe messages that match the routing rules, such as messages with both A/B rules, consumers can only subscribe to A messages, but B messages will not be consumed.

(1) Kafka

Kafka is an open source stream processing platform developed by the Apache Software Foundation and written in Scala and Java. The goal of this project is to provide a unified, high-throughput, low-latency platform for processing real-time data. Its persistence layer is essentially a "large-scale publish/subscribe message queue following a distributed transaction log architecture", which makes it very valuable as an enterprise-grade infrastructure for processing streaming data. (Wikipedia)

basic terminology

Producer: message producer. Typically, a message is sent to a specific topic. Typically, messages are written to each partition by polling. The producer can also write the message to the specified partition by setting the message key value. The more uniform the data written to the partition, the better the performance of Kafka.

Topic: Topic is an abstract virtual concept. A cluster can have multiple topics as the identifier of a type of message. A producer sends messages to a topic, and consumers get partitioned messages by subscribing to the topic.

Partition: Partition is a physical concept, and a Topic corresponds to one or more Partitions. New messages are written to the partition in an appending manner, and messages are ordered in the same Partition. Kafka achieves message redundancy and scalability through partitioning, and supports physical concurrent reading and writing, which greatly improves throughput.

Replicas: A Partition has multiple replicas of Replicas. These replicas are stored in brokers, and each broker stores hundreds or thousands of replicas of different topics and partitions. The stored content is divided into two types: master replicas, each partition has a master replica, and all content is written and consumed. All will go through the master copy; the follower copy does not process any client's request, but only synchronizes the contents of the master for replication. If an exception occurs on the master, a follower will soon become the new master.

Consumer: message reader. Consumers subscribe to topics and read messages in a certain order. Kafka guarantees that each partition can only be used by one consumer.

Offset: Offset is a kind of metadata, which is an increasing integer. Kafka adds it to the message as it is written. Offsets are unique within a partition. During the consumption process, the last read offset will be stored in Kafka, the offset will not be lost when the consumer closes, and the restart will continue to consume from the last position.

Broker: Standalone Kafka server. A Topic has N Partitions, and a cluster has N Brokers, then each Broker will store a Partition of the Topic. If a topic has N partitions and the cluster has (N+M) brokers, then N brokers store one partition of the topic, and the remaining M brokers do not store the topic's partition data. If a topic has N partitions and the number of brokers in the cluster is less than N, then one broker stores one or more partitions of the topic. In the actual production environment, try to avoid this situation, which can easily lead to unbalanced data in the Kafka cluster.

system framework

The first topic has two productions, new messages are written to partition 1 or partition 2, and both partitions have backups on broker1 and broker2. After a new message is written, the two follower partitions will synchronize changes from the two master partitions. The corresponding consumer will get messages from the two master partitions according to the current offset and update the offset. The second topic has only one producer, which also corresponds to two partitions, which are distributed on two brokers in the Kafka cluster. When a new message is written, the two follower partitions will synchronize the master changes. Two consumers get messages from different master partitions respectively.

advantage

High throughput and low latency: Kafka can process hundreds of thousands of messages per second, and its latency is at least a few milliseconds;

Scalability: Kafka cluster supports hot expansion;

Persistence and reliability: messages are persisted to local disks, and data backup is supported to prevent data loss;

Fault tolerance: Allows node failures in the cluster, multiple copies of one data, a few machines down, and no data loss;

High Concurrency: Support thousands of clients to read and write at the same time.

shortcoming

Partition ordering: ordering is only guaranteed within the same partition, and global ordering cannot be achieved;

No delay message: The order of consumption is in the order of writing, and delayed messages are not supported;

Repeated consumption: The offset is not submitted due to the downtime and restart of the consumption system;

Rebalance: During the rebalance process, all consumer instances under the consumer group will stop working and wait for the rebalance process to complete.

scenes to be used

Log collection: A large number of log messages are first written to Kafka, and the data service implements the data by consuming Kafka messages.

Message system: decouple producers and consumers, cache messages, etc.

User activity tracking: Kafka is often used to record various activities of web users or app users, such as browsing the web, searching, clicking and other activities. These activity information is published by each server to the topic of Kafka, and then consumers subscribe to these topics by subscribing to these topics. For real-time monitoring and analysis, it can also be saved to the database.

Operational metrics: Record operational and monitoring data, including collecting data from various distributed applications, and producing centralized feedback for various operations, such as alarms and reports. Streaming: such as spark streaming.

(2) RabbitMQ

RabbitMQ is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). RabbitMQ server is written in Erlang language, while clustering and failover are built on open telecom platform framework. All major programming languages have client libraries that communicate with the proxy interface. (Wikipedia)

basic terminology

Broker: Receives client link entities and implements AMQP message queue and routing functions.

Virtual Host: is a virtual concept, the smallest unit of permission control. A Virtual Host contains multiple Exchanges and Queues.

Exchange: Receive messages from message producers and forward messages to a queue. When sending a message, the routing rules are determined according to different ExchangeTypes. ExchangeTypes are commonly used: direct, fanout, and topic.

Message Queue: message queue, stored as consumed messages.

Message: It consists of Header and Body. Header is various attributes added by the producer, including whether the Message is persistent, which MessageQueue receives it, and the priority. Body is the specific message content.

Binding: Binding connects Exchange and Message Queue. When the server is running, a routing table is generated, which records the conditions and BindingKey values of MessageQueue. When Exchange receives the message, it parses the Header in the message to get the BindingKey, and sends the message to the corresponding MessageQueue according to the routing table and ExchangeType. The final matching pattern is determined by ExchangeType.

Connection: The TCP connection between the Broker and the client.

Channel: channel. Only the tcp connection between the broker and the client cannot send messages, and a channel must be created. The AMQP protocol stipulates that AMQP commands can only be executed through Channel. A Connection can contain multiple Channels. The reason why a Channel needs to be established is because each TCP connection is precious. If each client and each thread need to interact with the Broker and need to maintain a TCP connection, the machine consumes resources. It is generally recommended to share the Connection. RabbitMQ does not recommend that client threads share the Channel before, at least to ensure that the same Channel sends small messages through.

Command: AMQP command, the client completes the interaction with the AMQP server through Command.

system framework

A Message reaches the corresponding Exchange through the channel. After Exchange receives the message, it parses the content of the message Header, obtains the BindingKey of the message, forwards the message to the corresponding MessageQueue according to the Binding and ExchangeType, and finally sends the message to the client through the Connection.

ExchangeType

Direct: exact match

Only when the RoutingKey and BindingKey match exactly, the message queue can get the message

Broker provides an Exchange by default, the type is Direct, the name is an empty string, and it is bound to all Queues (distinguished by the Queue name here)

Fanout: subscription, broadcast

This mode will forward the message to the Queue of all routes

Topic: Wildcard Pattern

RoutingKey is a string separated by a period ". " (each independent string separated by a period ". " is called a word), such as "quick.orange.rabbit". BindingKey is the same as RoutingKey.

The two special characters "#" and "" in Bindingkey are used for fuzzy matching, "#" is used to match multiple singles, and "" is used to match a single word (including zero)

advantage

Based on the AMQP protocol: Apart from Qpid, RabbitMQ is the only message server that implements the AMQP standard.

Robust, stable and easy to use. The community is active and the documentation is complete.

Timed messages are supported. Pluggable authentication, authorization, TLS and LDAP support.

It supports querying messages based on message identifiers, and also supports querying messages based on message content.

shortcoming

Erlang development source code is difficult to understand, which is not conducive to secondary development and maintenance.

The interfaces and protocols are complex, and the learning and maintenance costs are high.

Summarize

Erlang has concurrency advantages and better performance. Although the source code is complex, the community is highly active and can solve problems encountered in development. If the business traffic is not large, you can choose RabbitMQ with complete functions.

(3) Pulsar

Apache Pulsar is a top-level project of the Apache Software Foundation. It is a next-generation cloud-native distributed message flow platform that integrates messaging, storage, and lightweight functional computing. Cross-regional data replication across multiple computer rooms, with streaming data storage features such as strong consistency, high throughput, low latency, and high scalability, is regarded as the best solution for real-time message streaming, storage, and computing in the cloud-native era. Pulsar is a pub-sub (publish-subscribe) model message queuing system. (Encyclopedia)

basic terminology

Property: On behalf of the tenant, each property can represent a team, a function, and a product line. A property can contain multiple namesapce, and multi-tenancy is a means of resource isolation, which can improve resource utilization.

Namespace: The basic management unit of Pulsar. At the namaspace level, permissions, message TTL, and retention policies can be set. All topics in a namaspace inherit the same settings. There are two types of namespaces: the local namespace, which is only visible within the cluster, and the global namespace, which is visible to multiple clusters.

Producer: The data producer, responsible for creating messages and delivering them to Pulsar.

Consumer: Data consumer, connects to Pulsar to receive messages and process them accordingly.

Broker: Stateless Proxy service, responsible for receiving messages, delivering messages, cluster load balancing and other operations. It shields the client from the complexity of the server-side read and write process and plays an important role in ensuring data consistency and data load balancing. Broker does not persist metadata. Can be expanded but not reduced.

BookKeeper: Stateful, responsible for persistently storing messages. When the cluster is expanded, Pulsar will add BookKeeper and Segment (that is, the Ledger of Bookeeper), and it does not need to perform Rebalance when expanding the capacity like Kafka. The result of the expansion is that Fragments are distributed in stripes across multiple Bookies, and Fragments of the same Ledger are distributed on multiple Bookies, resulting in reading and writing jumping between multiple Bookies.

ZooKeeper: Stores the metadata of Pulsar and BookKeeper, cluster configuration and other information, and is responsible for coordination between clusters, service discovery, etc.

Topic: used to transmit messages from producer to consumer. Pulsar has a leader broker at the topic level, which is called ownership of the topic, and all R/W for the topic is completed by the broker. Metadata such as the mapping relationship between Topic's Ledger and Fragment are stored in Zookeeper, and Pulsar Broker needs to track these relationships in real time for read and write processes.

Ledger: Segment, the underlying data of Pulsar is stored on BookKeeper in the form of Ledger. is the smallest unit that Pulsar deletes.

Fragment: Each Ledger consists of several Fragments.

system framework

The above frame diagram demonstrates the two situations of capacity expansion and failover respectively. Capacity expansion: Bookie N is newly added due to the increase in business volume, and the subsequently written data segment x and segment y are written into the newly added Bookie. In order to maintain a balance, the expansion result is shown in the green module in the above figure. Failover: When segment 4 of Bookie 2 fails, Pulasr's Topic will immediately reselect Bookie 1 as the service for processing reads and writes. Broker is a stateless service that only serves data calculation and not storage, so Pulsar can be considered as a distributed system based on Proxy.

advantage

Flexible expansion.

Seamless failback.

Delayed messages are supported.

Built-in replication function for cross-region replication, such as disaster recovery.

Two consumption models are supported: stream (exclusive mode) and queue (shared mode).

(4) RocketMQ

RocketMQ is a distributed messaging and streaming data platform with low latency, high performance, high reliability, trillion-level capacity, and flexible scalability. RocketMQ is the third-generation distributed messaging middleware open sourced by Alibaba in 2012. (Wikipedia)

basic terminology

Topic: A topic can have 0, 1, or multiple producers send messages to it, and a producer can also send messages to different topics at the same time. A Topic can also be subscribed by 0, 1, or multiple consumers.

Tag: The secondary type of message, which can provide users with additional flexibility. A message can have no tag.

Producer: message producer.

Broker: Stores messages, a lightweight queue with Topic as the latitude; forwarding messages, a single Broker node maintains a long connection and heartbeat with all NameServer nodes, and regularly registers Topic information to the NameServer.

Consumer: Message consumer, responsible for receiving and consuming messages.

MessageQueue: The physical management unit of messages. A topic can have multiple Queues. The introduction of Queues realizes the ability to scale horizontally.

NameServer: Responsible for the management of the original data, including Topic and routing information, there is no communication between each NameServer.

Group: A group can subscribe to multiple topics. ProducerGroup and ConsumerGroup are a type of producer and a type of consumer respectively.

Offset: Access the storage unit through Offset. All messages in RocketMQ are persistent, and the storage unit has a fixed length. Offset is a Java Long type, and theoretically it will not overflow within 100 years, so it is considered that Message Queue is infinitely long data, and Offset is a subscript.

Consumer: Supports two consumption modes, PUSH and PULL, and supports cluster consumption and broadcast consumption.

system framework

advantage

Publish/subscribe (Pub/Sub) and peer-to-peer (P2P) messaging models are supported.

Sequential Queue: Reliable first-in-first-out (FIFO) and strictly sequential delivery in a queue.

Both pull and push message modes are supported.

Stacking capacity of millions of messages on a single queue.

Supports multiple message protocols, such as JMS, MQTT, etc.

Distributed scale-out architecture.

Satisfies at-least-once message delivery semantics.

Provides a rich Dashboard, including configuration, metrics, monitoring, and more.

The supported clients are currently java, c++ and golang.

shortcoming

The community is generally active.

Delayed message: The open source version does not support arbitrary time precision, only a specific level.

scenes to be used

Born for the financial Internet field, it requires high reliability for scenarios.

About the Author

Chen Dong, Tencent backend development engineer.