In-depth analysis of Apache Pulsar series (1): client message confirmation

Editor's

This article is reproduced from Tencent Cloud Middleware Public Account. Author Lin Lin, Apache Pulsar PMC, Tencent Cloud middleware expert engineer, author of "In-depth Analysis of Apache Pulsar". The first article in this series explains the mode of message confirmation in Pulsar.

Introduction

In Apache Pulsar, in order to avoid repeated delivery of messages, it is a very important step for consumers to confirm messages. When a message is consumed by the consumer, the consumer needs to send an Ack request to the Broker, and the Broker will think that the message is actually consumed. Messages marked as consumed will not be re-delivered to consumers in the future. In this article, we will introduce the mode of message confirmation in Pulsar and how normal message confirmation is implemented on the Broker side.

Mode for acknowledging messages

Before understanding the Pulsar message confirmation mode, we need to understand some pre-knowledge - subscriptions and cursors in Pulsar. There are multiple consumption modes in Pulsar, such as: Shared (shared), Key_shared (key shared), Failover (disaster recovery), etc. No matter which consumption mode the user uses, a subscription will be created. Subscriptions are divided into durable subscriptions and non-durable subscriptions. For durable subscriptions, there will be a persistent Cursor on the Broker, that is, the metadata of the Cursor is recorded in ZooKeeper. Cursor takes subscription (or called consumer group) as a unit and saves where the current subscription has been consumed. Because the subscription models used by different consumers are different, the Ack behaviors that can be performed are also different. Generally speaking, it can be divided into the following Ack scenarios:

1. Single message acknowledgement (Acknowledge)

Unlike some other messaging systems, Pulsar supports a Partition being consumed by multiple consumers. Assuming that messages 1, 2, and 3 are sent to Consumer-A, and messages 4, 5, and 6 are sent to Consumer-B, and Consumer-B consumes faster, Ack message 4 first, then Cursor will record it separately Message 4 is Ack status. If other messages are consumed but not Acked, and both consumers are offline or Ack times out, the Broker will only push messages 1, 2, 3, 5, and 6, and message 4 that has been Acked will not be pushed again. push.

2. Acknowledge Cumulative

Assuming that the Consumer has received messages 1, 2, 3, 4, and 5, in order to improve the performance of Ack, the Consumer can not separate Ack 5 messages, just call AcknowledgeCumulative, and then pass in the message 5, and the Broker will send the message 5 and the previous message. Messages are all marked as Ack.

3. Acknowledge Cumulative

In this message confirmation mode, the calling interface is the same as the confirmation of a single message, but this capability requires the Broker to enable the configuration item AcknowledgmentAtBatchIndexLevelEnabled . When enabled, Pulsar can support some messages in only Ack a Batch. Suppose the Consumer gets a batch message, which contains messages 1, 2, and 3. If this option is not enabled, we can only consume the entire Batch and then Ack, otherwise the Broker will re-deliver all the batches once. After the options described above are turned on, we can use the Acknowledge method to confirm a single message in a batch of messages.

4. Negative Acknowledge

The client sends a RedeliverUnacknowledgedMessages command to the Broker, which clearly informs the Broker that the current Consumer cannot consume this message, and the message will be redelivered.

Not all subscription modes can use these Ack behaviors. For example, AcknowledgeCumulative is not supported in Shared or Key_shared modes. Because in Shared or Key_shared mode, the previous message is not necessarily consumed by the current Consumer. If AcknowledgeCumulative is used, other people's messages will also be confirmed. The relationship between subscription mode and message acknowledgment is as follows:

Implementation of Acknowledge and AcknowledgeCumulative

The Acknowledge and AcknowledgeCumulative interfaces do not directly send a message confirmation request to the Broker, but forward the request to the AcknowledgmentsGroupingTracker for processing. This is the first Tracker in the Consumer we are going to introduce. It is just an interface. There are two implementations under the interface, one is the implementation of persistent subscription, and the other is the implementation of non-persistent subscription. Since the Tracker implementations of non-persistent subscriptions are empty, that is, do nothing, we only introduce the implementation of persistent subscriptions - PersistentAcknowledgmentsGroupingTracker.

In Pulsar, in order to ensure the performance of message confirmation and prevent the Broker from receiving very high concurrent Ack requests, Tracker supports batch confirmation by default. Even the confirmation of a single message will enter the queue first, and then send it to the Broker in batches. We can set the parameter AcknowledgementGroupTimeMicros when creating the Consumer. If it is set to 0, the Consumer will immediately send a confirmation request every time. All single acknowledgement (IndividualAck) requests will be put into a Set named PendingIndividual Acks first. The default is to send a batch of acknowledgement requests every 100ms or if the accumulated acknowledgement requests exceed 1000.

The request for message acknowledgment is ultimately sent out asynchronously. If the Consumer is set to require a receipt (Receipt), it will return a CompletableFuture, and success or failure can be sensed through the Future. By default, no receipt is required, and a completed CompletableFuture is returned directly.

For a single acknowledgment (IndividualBatchAck) in a Batch message, a PendingIndividualBatchIndexAcks is used to save it instead of a Set of ordinary single messages. The Key of this Map is the MessageId of the Batch message, and the Value is a BitSet, which records which messages in this batch need Ack. Using BitSet can greatly reduce the memory occupancy of saving message Id, and 1KB can record whether 8192 messages are confirmed. Since the contents stored in BitSet are all 0 and 1, it can be easily stored outside the heap. BitSet objects are also pooled and can be used cyclically without creating new ones every time, which is very memory-friendly.

As shown in the figure below, only 8 bits are used to indicate the Ack status of the 8 messages in the Batch. The figure below indicates that the Entry with the EntryId of 0, 2, 5, 6, and 7 has been confirmed, and the confirmed position will be set. is 1:

For the cumulative confirmation (CumulativeAck) implementation method is simpler, only the latest confirmation location point is saved in the Tracker. For example, the CumulativeAck position saved in Tracker is now 5:10, which means that the subscription has consumed the message with LedgerId=5, EntryId=10. After that, another Ack of 5:20 is added, so just replace the previous 5:10 with 5:20.

The last is the Flush of the Tracker. All confirmations need to be sent to the Broker by triggering the Flush method. No matter what kind of confirmation, the Flush creates the same command and sends it to the Broker, but the AckType in the parameters will be different. .

Implementation of NegativeAcknowledge

Negative acknowledgment, like other message acknowledgments, does not immediately request the Broker, but forwards the request to the NegativeAcksTracker for processing. Tracker records each message and the time it needs to be delayed. Tracker reuses the time wheel of PulsarClient. The default is a time scale of about 33ms for checking, and the default delay time is 1 minute. It extracts expired messages and triggers redelivery. The main purpose of Tracker is to merge requests. In addition, if the delay time has not expired, the message will be temporarily stored in the memory. If there are a large number of messages on the business side that need to be delayed for consumption, it is recommended to use the ReconsumeLater interface. The only advantage of NegativeAck is that you don't need to specify a time for each message, you can set the delay time globally.

Handling of unacknowledged messages

What if the consumer does not Ack after getting the message? There are two cases. The first is that the business side has called the Receive method, or has called back the consumer who is asynchronously waiting. At this time, the reference of the message will be saved in the UnAckedMessageTracker, which is the third Tracker in the Consumer. . A time wheel is maintained in UnAckedMessageTracker. The scale of the time wheel is AckTimeout and TickDurationInMs , and each scale time is =AckTimeout / TickDurationInMs . The newly tracked message will be placed in the last tick. Each scheduling will remove the first tick at the head of the queue, and add a new tick to the end of the queue to ensure that the total number of ticks remains unchanged. The messages in the queue head ticks will be cleaned up each time they are scheduled, and UnAckedMessageTracker will automatically redeliver these messages.

RedeliverUnacknowledgedMessages means that the client sends a 061f0f87fb454e command to the Broker. Every message pushed to the consumer but not Ack will be recorded in a collection (Pengding Ack) on the Broker side, which is used to avoid repeated delivery. After triggering re-delivery, the Broker will remove the corresponding message from this set, and then these messages can be consumed again. Note that when re-delivery, if the consumer is not in the Share mode, it cannot re-deliver a single message, and can only re-deliver all the messages that the consumer has received but not acked. The following figure is a simple example of a time wheel:

Another situation is that the consumer has pre-fetched, but has not called any Receive methods, and messages will always be accumulated in the local queue. Pre-pulling is the default behavior of the client SDK, which will pre-pull messages to the local. We can control the number of pre-pull messages through the ReceiveQueueSize parameter when creating a consumer. The Broker side will record these messages that have been pushed to the local consumer as PendingAck, and these messages will not be delivered to other consumers, and the Ack will not time out. Unless the current Consumer is closed, the message will be re-delivered. There is a Redelivery Tracker interface on the Broker side, and the temporary implementation is InMemoryRedeliveryTracker. This tracker will record how many times the message has been re-delivered. When each message is pushed to the consumer, it will first query the number of red-delivery times from the tracker's hash table, and push it to the consumer together with the message.

From the above logic, we can know that the Receive QueueSize set when creating a consumer is really careful to avoid a large number of messages from accumulating in the local pre-pull queue of a certain Consumer, while other Consumers have no messages to consume.

PulsarClient can be set to enable ConsumerStatsRecorder . After enabling, the consumer will print out the metrics information of the current consumer at regular intervals, such as: local message accumulation, number of received messages, etc., to facilitate business troubleshooting performance problems.

postscript

There are many design details in Pulsar. Due to the limited space, the author will organize a series of articles for technical sharing, so stay tuned.

Follow public account "Apache Pulsar", for dry goods and news

Join the Apache Pulsar Chinese exchange group 👇🏻