Blog recommendation | This article takes you to understand Pulsar's message retention and expiration strategy

Editor’s Recommendation:

The original author Ran Xiaolong, first published on the public account "Tencent Cloud Middleware", and the publication has been authorized by the original account. If you need to reprint, please go to contact. This article mainly introduces Apache Pulsar's message retention and expiration strategy for your reference.

About Apache Pulsar

Apache Pulsar is the top-level project of the Apache Software Foundation. It is the next-generation cloud-native distributed message flow platform. It integrates messaging, storage, and lightweight functional computing. It uses a separate architecture design for computing and storage to support multi-tenancy, persistent storage, Multi-computer room and cross-regional data replication, with strong consistency, high throughput, low latency and high scalability and other streaming data storage characteristics.
GitHub address: http://github.com/apache/pulsar/

By default, Pulsar Broker will process messages as follows:

• When the message is confirmed by the Consumer, the delete operation will be executed immediately.
• The unacknowledged messages will be stored in the backlog.

However, in many online production environments, this default behavior does not meet our production needs. Therefore, Pulsar provides the following configuration strategies to cover these behaviors:

• Retention strategy: The user can retain the message that the Consumer has confirmed.
• TTL policy: For unconfirmed messages, users can set TTL to make unconfirmed messages reach the confirmed state.

The above two strategies are set at the level of NameSpace.

Retention strategy

There are two ways to set the Retention policy:

• The size of the message, default value: defaultRetentionSizeInMB=0
• The time the message was saved, default value: defaultRetentionTimeInMinutes=0

We can configure these two items in broker.conf or through the command line. As mentioned above, the settings of these two strategies are set at the level of NameSpace, so when we use the command line to configure, use NameSpaces to configure, as follows:

root@e6df71e544ea:/pulsar# ./bin/pulsar-admin namespaces set-retention
The following options are required: --size, -s --time, -t
Set the retention policy for a namespace
Usage: set-retention [options] tenant/namespace
  Options:
  * --size, -s
       Retention size limit (eg: 10M, 16G, 3T). 0 or less than 1MB means no
       retention and -1 means infinite size retention
  * --time, -t
       Retention time in minutes (or minutes, hours, days, weeks eg: 100m, 3h, 2d,
       5w). 0 means no retention and -1 means infinite time retention

As shown above: we can specify the size or time we need to configure -s or -t

After you set the Retention policy, you can use the following commands to view specific information:

$ pulsar-admin namespaces get-retention [your tenant]/[your-namespace]
{
  "retentionTimeInMinutes": 10,
  "retentionSizeInMB": 0
}

Backlog

The backlog is a collection of unconfirmed messages. It has a major premise that the Topic where these messages are located is persisted by the Broker. By default, the Topic created by the user will be persisted. In other words, Pulsar Broker will store all unconfirmed or unprocessed messages in the backlog.

Similarly, we can configure the size of the backlog at the NameSpace level. It should be noted that when configuring the backlog, we need to clarify the following two points:

• Under the current NameSpace, what is the allowable size of each topic
• If the threshold of the backlog is exceeded, what actions will be performed

When the threshold of the set backlog is exceeded, Pulsar provides the following three strategies for users to choose:

You can configure the backlog at the NameSpace level set-backlog-quota

root@e6df71e544ea:/pulsar# ./bin/pulsar-admin namespaces set-backlog-quota
The following options are required: -l, --limit -p, --policy
Set a backlog quota policy for a namespace
Usage: set-backlog-quota [options] tenant/namespace
  Options:
  * -l, --limit
       Size limit (eg: 10M, 16G)
  * -p, --policy
       Retention policy to enforce when the limit is reached. Valid options are:
       [producer_request_hold, producer_exception, consumer_backlog_eviction]

As shown above, set-backlog-quota provides two parameters. -l used to specify the size of the backlog you set, and -p used to specify the strategy that Broker will execute when the threshold of the backlog you set is exceeded.

After you set the backlog, you can use the following command to view the corresponding information:

$ pulsar-admin namespaces get-backlog-quotas [your tenant]/[your namespace]
{
  "destination_storage": {
    "limit" : 2147483648,
    "policy" : "producer_request_hold"
  }
}

If you want to cancel the backlog configuration, you can use the following command:

$ pulsar-admin namespaces remove-backlog-quota [your tenant]/[your namespace]

When there is a backlog of messages, you can clear the backlog through clear-backlog. Clearing the backlog of messages in the backlog is a relatively dangerous operation, so the system will prompt you whether you want to delete the messages in the backlog. clear-backlog provides the -f(--force) to shield the prompt.

$ pulsar-admin namespaces clear-backlog [your tenant]/[your namespace]

Time To Live （TTL）

By default, Pulsar will persist all unacknowledged messages. If there are many unconfirmed messages, this strategy will cause a large amount of messages to be backlogged, resulting in increased disk space. In some scenarios, messages do not need to be persisted. The user's more expected behavior is to directly discard these unconfirmed messages. In this case, you can set the TTL to make the unconfirmed message enter the confirmed state. When the set TTL time expires, the message will be discarded with the corresponding Retention strategy.

A typical use scenario of TTL is that when the Consumer fails for some reasons and cannot consume messages normally, the Producer is still producing messages in the Topic, which will cause a large number of unconfirmed messages to appear in the Topic. You can change these unconfirmed messages to confirmed status by setting TTL.

Similarly, you can set TTL by specifying set-message-ttl at the Namesapce level. The specific commands are as follows:

root@e6df71e544ea:/pulsar# ./bin/pulsar-admin namespaces set-message-ttl
The following option is required: --messageTTL, -ttl
Set Message TTL for a namespace
Usage: set-message-ttl [options] tenant/namespace
  Options:
  * --messageTTL, -ttl
       Message TTL in seconds
       Default: 0

As shown above, set-message-ttl has only one parameter -ttl , the unit is seconds, and the default value is 0.

After you set the TTL policy, you can view the corresponding configuration information through get-message-ttl, as follows:

$ pulsar-admin namespaces get-message-ttl [your tenant]/[your namespace]
60

The difference and connection between TTL, Backlog and Retention

In the above description process, it can be found that TTL only handles one thing, changing the unconfirmed message into the confirmed state, and TTL itself does not involve the corresponding deletion operation, as shown in the following figure:

• In the T1 phase, the 5 messages of m1-m5 are confirmed, and the 5 messages of m6-m10 are not confirmed
• In the T2 phase, set the TTL policy for the three messages m6, m7, and m8
• In the T3 phase, the threshold set by TTL is reached, and the three messages m6, m7, and m8 are confirmed

As you can see from the above figure, for the unconfirmed messages in the backlog, when you set the TTL, the status of the unconfirmed messages will be changed to the confirmed state. The role of TTL here is to move the Cursor of the message from m5 to m8, and the three messages of m6, m7, and m8 become confirmed.

Pulsar is a multiple-subscription messaging system. For a message in a topic, it can only be deleted when all subscribers ack or consume the message.

By default, Pulsar Broker will persist all unconfirmed messages to the backlog. The function of TTL is that you can turn these unconfirmed messages into the confirmed state, and the focus of Retention is that when the message is in the confirmed state, the retention strategy you can carry out on the confirmed message is what. In other words, the backlog is for unconfirmed messages, what is the processing done by Broker. Retention refers to the confirmed message, what is the retention strategy Broker does.

TTL and dead letter queue

The relevant introduction of the dead letter queue will not be repeated here.

In a production environment, sometimes you may encounter poor quality data due to upstream reasons, which must be resolved by the upstream. It is meaningless to continue to try to process other messages. At this time, the user wants to stop processing immediately when an error occurs. Pulsar provides a special topic-dead letter queue.

Both the dead letter queue and TTL can turn unconfirmed messages into confirmed states. The main difference between them is that in the T2 stage in the above figure, TTL just turns unconfirmed messages into confirmed states. The method of dead letter queue is to drop messages into the dead letter queue, m6, m7, m8 These three messages become the confirmed state. The two valid messages m9 and m10 will be processed normally, and the Broker will continue to run. After that, you can check for invalid messages from the dead letter queue and ignore or fix and reprocess them as needed. Users can determine whether unconfirmed messages are changed to confirmed status in the form of TTL or through dead letter queue according to their own needs. The main criterion is to see whether you need to process messages that cannot be consumed.

Usage problem

scene one

Start the Producer to send a message to the Broker, set the TTL, not start the Consumer, and set the Retention policy for half an hour. After reaching the Retention threshold, it is found that the message that sets the TTL has not been removed. Why?

In the above scenario, there is a problem that needs attention. does not start Consumer . As we mentioned above, TTL is the Cousor that sets the message to move forward. If the Consumer is not started, it is equivalent to the Cousor has not been initialized, that is, if there is no Consumer, you don't need to set TTL.

Scene two

I set the Retention policy, but the Retention threshold is reached, and the data in Topic is not deleted. Why is this?

This is an implementation mechanism inside Pulsar. Topic is a logical concept in Pulsar. A topic corresponds to a manage ledger. When you write data, you actually write the data to ledger. Remember in many previous articles. The core of the design of Pulsar mentioned: In Pulsar, all operations are asynchronous , so when Retention reaches the specified threshold, whether to delete the data in the corresponding ledger, this operation is also asynchronous. The delete operation will not be performed on the currently active ledger. Only when the data is filled with the current ledger and the ledger switches, will the retention strategy be truly implemented.

If you want to enforce it, you can use pulsar-admin to forcibly uninstall the current ledger, forcing it to switch to the ledger.

About the author

Ran Xiaolong, R&D Engineer of Tencent Cloud Microservice Product Center, Apache Pulsar Committer, Apache BookKeeper Contributor

related suggestion

• Pulsar namespace strategy analysis
• 1619d9caacbc4f Message Lifecycle: What is the message
• Understanding the working principle of Apache Pulsar

Click the link get the Apache Pulsar hard core dry goods information!