Blog recommendation｜Tencent experts in-depth analysis of five major application scenarios of Apache Pulsar

Editor’s Recommendation:

The MQ team of Tencent Data Platform Department conducted in-depth research on Pulsar and optimized a lot of performance and stability. It has been launched on Tencent Cloud Message Queuing TDMQ. This article mainly summarizes some traditional message queue application scenarios supported by Pulsar, and the new features of Pulsar support more scenarios.

The following article is from Tencent Cloud Middleware, author Zhang Chao

This article is transferred from Tencent Cloud Middleware, author Zhang Chao, senior engineer of the MQ team of Tencent Data Platform Department, Apache TubeMQ (incubating) PMC, Kafka-on-Pulsar Maintainer, Apache Pulsar Contributor.
Typesetting: Tango@StreamNative.

About Apache Pulsar

Apache Pulsar is the top-level project of the Apache Software Foundation. It is the next-generation cloud-native distributed message flow platform. It integrates messaging, storage, and lightweight functional computing. It uses a separate architecture design for computing and storage to support multi-tenancy, persistent storage, Multi-computer room and cross-regional data replication, with strong consistency, high throughput, low latency and high scalability and other streaming data storage characteristics.
GitHub address: http://github.com/apache/pulsar/

Message queue overview

What is a message queue

Message Queue (MQ for short) refers to a container or service that stores messages in the transmission of messages. It is an asynchronous method of communication between services. It is suitable for serverless and microservice architectures and is a distributed system that achieves high performance. , High availability, scalability and other important components of advanced special effects.

Common mainstream message queues include ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, Pulsar, etc. In the company, there are TubeMQ, Ckafka, TDMQ, CMQ, CDMQ, Hippo, etc.

Message queue characteristics

distributed

Message queues are distributed, so they can provide asynchronous, decoupling and other functions.

reliability

Message-based communication is reliable, and messages will not be lost. Most message queues provide the function of persisting messages to disk.

asynchronous

Through the message queue, remote synchronous calls can be disassembled into asynchronous calls. For application scenarios that do not need to obtain remote call results, the performance is improved significantly.

Loosely coupled

The messages are stored and distributed directly by the middleware. The message producer only needs to pay attention to how to send the message to the message intermediary server; the consumer only needs to pay attention to how to subscribe from the intermediary server. The producer and the consumer are completely decoupled and do not need to know each other's existence.

Event driven

The complex application system can be reconstructed into an event-driven system. Event Sourcing refers to the various states that an object will go through from creation to death. If the state changes of the object are stored, not only can the current state of the object be obtained from the state change record, but also the change process of the object can be traced back. The message queue can well support such a system design method, and put the event that triggers the object state change into the message queue.

Message queue classification

In the JMS (JAVA Message Service) standard, there are two message models: P2P (Point to Point) and Publish/Subscribe (Pub/Sub).

P2P

The characteristic of P2P is that there is only one consumer for each message. The message producer sends the message to the message queue (Queue), only one consumer can consume the message, and the message is deleted after the consumption is completed. Any consumer can consume this message, but the message will never be repeated by two consumers.

Pub/Sub

The characteristic of Pub/Sub is that messages published to Topic will be consumed by all subscribers. The message producer sends the message to the message topic (Topic), and all consumers who subscribe to this topic can consume the message. The message can be deleted after all subscribers have completed the consumption.

There is time dependence between message producers and consumers, and only consumers who subscribe to this topic in advance can consume it. If you send the message first and then subscribe to the topic, the message before the subscription will not be consumed by this subscriber.

The traditional enterprise message queue ActiveMQ follows the JMS specification and implements the point-to-point and publish-subscribe model, but other popular message queues RabbitMQ and Kafka do not follow the JMS specification.

In the real-time streaming architecture , the message delivery of the message queue can be divided into two types queue (Queue) and stream (Stream)

Queue model

The queue model is mainly to consume messages in an unordered or shared manner. Through the queue model, users can create multiple consumers to receive messages from a single pipe; when a message is sent from the queue, only one of the multiple consumers (any one is possible) receives and consumes the message. The specific implementation of the message system determines which consumer actually receives the message in the end.

The queue model is usually used in conjunction with stateless applications. Stateless applications don't care about ordering, but they do need to be able to acknowledge (ACK) or delete individual messages, as well as the ability to expand consumption parallelism as much as possible. Typical message systems based on the queue model include RabbitMQ and RocketMQ.

Stream model

In contrast, the streaming model requires strict ordering of message consumption or exclusive message consumption. For a pipeline, using the streaming model, there will always be only one consumer to consume and consume messages. Consumers receive messages sent from the pipe in the exact order in which they are written to the pipe.

The flow model is usually associated with stateful applications. Stateful applications pay more attention to the order of messages and their status. The order in which messages are consumed determines the state of a stateful application. The order of the messages will affect the correctness of the application processing logic. Typical message systems based on the stream model include Kafka and TubeMQ.

Application scenarios of traditional message queues

Asynchronous call

Suppose there is a system call link for A to call B and it takes 20ms, B to call C to take 20ms, and C to call D takes 2s, so the entire call takes 2040ms. But in fact, it only takes 40ms for A to call B, and B to call C, and the introduction of the D system directly causes the system performance to drop by about 50 times. At this point, we can consider introducing a message queue, extracting the call of the D system, and making an asynchronous call: system A to system B and then to system C will end directly, system C will send the message to the message queue, system D from Messages are taken from the message queue for consumption, so that the performance of our system is improved by nearly 50 times.

System decoupling

Each business system only needs to process its own business logic and send event messages to the message queue. The downstream business system directly subscribes to the queue or topic of the message queue to obtain events. Message queues can be used to communicate between different microservices after a single application is disassembled into microservices. The advantage of system decoupling is that the iterations of different systems no longer depend on each other, which can effectively shorten the length of the data link and improve the efficiency of data processing.

Peak shaving and valley filling

When large-scale activities bring high traffic, failure to properly protect the system can easily lead to system overload or even crash, while too much restriction will cause a large number of requests to fail and affect user experience. The message queuing service has high-performance message processing capabilities that can handle traffic pulses without being overwhelmed. While ensuring system availability, it also improves user experience through fast and effective request response technology. Its massive message accumulation capability ensures the smooth and stable operation of downstream businesses within a safe water level, avoiding the impact of traffic peaks.

Broadcast notification

A state change of the system requires multiple related systems to be notified, which can be pushed to each subscriber system through message subscription. For example, if the database value changes, all cache systems need to be notified to update. You can send a message to the message queue to change the database value, and then each cache subscribes to related topics, and updates its own cache after receiving the message.

Distributed cache

In big data scenarios, log analysis often needs to process a large number of logs, and it is impossible to store them on a physical machine. The message queue can provide a cluster to store massive messages and cache them in the message queue for further analysis of logs by the real-time analysis system. Kafka and TubeMQ often act as distributed caches in big data processing.

Newsletter

Message queues generally have built-in efficient communication mechanisms, so they can also be used for pure message communication. Such as the realization of point-to-point message queues, or chat rooms.

Application scenarios of Pulsar

Pulsar is a message queue service with a new generation of storage and computing separation architecture. not only suitable for the above-mentioned traditional message queue application scenarios, but some of its new features also bring possibilities for more application scenarios.

The integration of queues and streams—maintaining a set of MQ services is enough

Apache Pulsar abstracts a unified producer-topic-subscription-consumer consumption model, which supports both the queue model and the stream model. In Pulsar's message consumption model, Topic is the channel used to send messages. Each topic corresponds to a distributed log in Apache BookKeeper. Each message published by the publisher is only stored once in the Topic; during the storage process, BookKeeper will copy the message and store it on multiple storage nodes; each message in the Topic can be multiple times according to the consumer's subscription requirements. Use, each subscription corresponds to a consumer group. Although the messages are only stored once on the topic (Topic), users can have different subscription methods to consume these messages:

• Consumers are grouped together to consume messages, and each consumer group is a subscription.
• Each topic can have different consumer groups.
• Each group of consumers is a subscription to the topic.
• Each group of consumers can have their own different consumption methods: Exclusive, Failover or Share.

Through this model, Pulsar combines the queue model and the flow model together to provide a unified API interface. This model will not affect the performance of the message system, nor will it bring additional overhead. At the same time, it also provides users with more flexibility, and it is convenient for the user program to use the message system in the most matching mode.

Multiple MQ protocol compatibility-easy migration of traditional MQ services

In the Pulsar architecture, in order to process Bookie storage messages and prevent message loss, a set of distributed process encapsulation is implemented based on Managed Leger. The Pulsar Protocol Handler processes the TCP requests sent by the producers and consumers in Pulsar, and converts them into operations that can be read. After Pulsar 2.5 version, the Protocol Handler interface is separated separately. Using this framework, you can implement the conversion of custom protocols, such as Kafka, AMQP, etc., which can help the existing MQ business to easily migrate to Pulsar.

Kafka Protocol Handler is currently maintained as an independent project — Kafka On Pulsar (KoP for short). Due to the large amount of memory in the company, Kafka has a lot of business, and the Suping MQ team has done a lot of optimization work for KoP. Tencent now has other teams that are also more in-depth To participate in the KoP project, please refer to Tencent Joining: Kafka-on-Pulsar project welcomes 2 Tencent Maintainers!

Enterprise-level multi-tenant features-data security is guaranteed

As the information hub of the enterprise, Apache Pulsar has supported multi-tenancy since its birth, because the project was originally designed to meet the strict requirements of Yahoo, and there was no open source system available on the market that could provide multi-tenancy. In Pulsar's design, tenants can be distributed across clusters, and each tenant can have a separate authentication and authorization mechanism; tenants are also management units for storage quotas, message TTLs, and isolation policies. Pulsar meets data security in multi-tenant scenarios in the following ways:

• Obtain the required security by performing authentication, authorization, and ACL (Access Control List) for each tenant.
• Enforce storage quotas for each tenant.
• All isolation mechanisms are defined in a strategic way, and the strategy can be changed during operation, thereby reducing operation and maintenance costs and simplifying management.

Cross-regional replication-comes with cross-computer room redundancy capability

In large-scale distributed systems, requirements across multiple data centers are involved. In scenarios with higher requirements for service quality and disaster recovery, the computer room will be planned to be deployed in multiple geographically dispersed data centers. In this type of multi-data center deployment, a cross-regional replication mechanism is usually used to provide additional redundancy to prevent a data center failure, natural intrusion, or other event from causing the service to fail to operate normally. Apache Pulsar added the need for cross-regional replication of Yahoo's more than ten computer rooms around the world at the beginning of its design. Apache Pulsar's cross-regional multi-computer room mutual backup feature is an important part of Pulsar's enterprise-level features. It provides users with convenient operation and management while ensuring the stability and reliability of data.

In the figure above, no matter when Producer P1, P2, and P3 publish messages to Topic T1 in Cluster A, Cluster B, and Cluster C, these messages will be replicated to the entire cluster immediately. Once the replication is complete, Consumers C1 and C2 can consume these messages from their clusters.

Pulsar's cross-regional replication is not only used in cross-data center data backup scenarios, but the ability of cross-regional replication in the PowerFL federal learning platform is also used as a communication service.

Cloud native support-help services go to the cloud

The originality of cloud native means that the software design takes into account the possibility of running in the cloud in the future, thus making full use of the characteristics of cloud resources at the design level, which is typically distributed and elastically scalable. The reason why Pulsar is said to be a cloud-native messaging platform is that its architecture is designed to make full use of distributed, elastically scalable cloud resources. Take Pulsar on Kubernetes as an example. Bookie is a stateful node, but the nodes are peer-to-peer and can be deployed using StatefulSet; while Broker, as a stateless node, can directly use ReplicaSet, and each Pod supports horizontal expansion.

At present, the company already has business using Pulsar on Kubernetes. If bookie uses Local Storage Volume, it will basically have no impact on the performance of Pulsar.

Related Reading

• Translation｜Iterable: From Kafka to Pulsar, we chose the right one!
• Translation | 10 reasons to choose Apache Pulsar over Kafka
• Pulsar and Kafka benchmark test report (full version)

Click the link , and like Pulsar!