Community Knowledge Base | Frequently Asked Questions FAQ Collection Issue 4: Issues Related to Message Retention and Delay, Broker, Pulsar Permissions, etc.

Usually in the Pulsar communication group, we find that people will encounter similar problems repeatedly in the process of contacting and using Pulsar. In order to solve these "high-frequency questions" more efficiently, and to express our gratitude to friends who raised high-quality questions, we have specially established a FAQ knowledge base to collect and answer your questions.

We will regularly collect and screen high-frequency questions raised in the community, and community experts will screen out high-quality questions to answer. After integration and optimization, we will share them with friends in the community as a priority reference when encountering problems. I hope it can help everyone. Solve problems in using Pulsar.

Let's take a look at the questions in this collection:

message retention

Question 1: In Pulsar's message retention mechanism, Pulsar's message will enter the retention policy after being Acked by a consumer. So when multiple consumers subscribe to a topic and some consumers of a message have Ack, will this message enter the retention policy?

Answer: Only if the message is ACKed in all subscriptions will it be further processed by the retention policy. For the same subscription, it doesn't matter which consumer ACKs, as long as it is ACKed once, the message in this subscription is already ACKed. For details, please refer to: https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#retention-policies

number of retries

Question 2: Pulsar support parameter configuration retry times?

Answer: Not supported. Pulsar implements a backoff retry strategy based on time, which is similar to the number of retries. The client's internal retry strategy is a backoff mechanism, which can be controlled by configuring the backoff parameter of ClientBuilder.

 /**
     * Set the duration of time for a backoff interval.
     *
     * @param duration the duration of the interval
     * @param unit the time unit in which the duration is defined
     * @return the client builder instance
     */
    ClientBuilder startingBackoffInterval(long duration, TimeUnit unit);

    /**
     * Set the maximum duration of time for a backoff interval.
     *
     * @param duration the duration of the interval
     * @param unit the time unit in which the duration is defined
     * @return the client builder instance
     */

From the producer's perspective, the total send timeout can be configured in the ProducerBuilder.

    /**
     * Set the send timeout <i>(default: 30 seconds)</i>.
     *
     * <p>If a message is not acknowledged by the server before the sendTimeout expires, an error will be reported.
     *
     * <p>Setting the timeout to zero, for example {@code setTimeout(0, TimeUnit.SECONDS)} will set the timeout
     * to infinity, which can be useful when using Pulsar's message deduplication feature, since the client
     * library will retry forever to publish a message. No errors will be propagated back to the application.
     *
     * @param sendTimeout
     *            the send timeout
     * @param unit
     *            the time unit of the {@code sendTimeout}
     * @return the producer builder instance
     */
    ProducerBuilder<T> sendTimeout(int sendTimeout, TimeUnit unit)

Pulsar Perf

Question 3: uses Pulsar Perf to produce data, but no backlog.

Answer: Create a Subscription before Pulsar Perf produces data.

Pressure measurement tool

Question 4: How does What tool to use?

Answer: Pulsar stress testing tools currently include Pulsar Perf and OpenMessaging Benchmark. For details, please refer to the following links:

• https://pulsar.apache.org/docs/en/performance-pulsar-perf/
• http://openmessaging.cloud/docs/benchmarks/

message delay

Question 5: Pulsar server support delayed delivery by default?

Answer: Delay message only takes effect in Shared subscription mode. If it does not take effect, you need to first check whether the subscription mode is Shared (shared), Pulsar itself defaults to the Exclusive (exclusive) subscription mode; at the same time, it should be avoided that the delay message takes effect in the KeyShared (key shared) mode, and the semantics of Delay message and Key_Shared are contradictory.

Broker down

Question 6: If the broker is down, how will the message be notified to the next broker?

Answer: The Broker itself is stateless.

• When the broker is down, it will not actively notify other brokers. All bundles distributed on this broker (bundle is a topic collection and the smallest unit for Pulsar topic load balancing) will perform the unload operation. The process of the unload operation:

1. Close the producers, consumers and replicators of all topics in unload;
2. Close Managedledger. When all bundles are unloaded, the broker can exit normally.

• If the broker exits abnormally, all bundles on the broker will be forcibly removed from the corresponding attribution, that is, the bundle does not belong to the broker.

When the producer or consumer client needs to continue to send/receive messages to a topic, it will first execute the lookup request, find the target broker node (currently the node with the lowest load) according to the loadbalance strategy, and onLoad the corresponding bundle to the target broker . After onLoad is completed, the broker can continue to provide read and write services for the topic.

In addition, some temporary node information of the Broker on ZooKeeper will be actively deleted; or it will be disconnected due to timeout, and other Brokers will perform corresponding operations after monitoring.

Compaction current limit

Question 7: The is very serious, and the speed of disk cleaning is far behind the writing speed. Adjusting the BookKeeper parameter gcWaitTime、majorCompactionInterval、minorCompactionInterval has no obvious effect. Is there any other solution?

Answer: Compaction has a current limiting strategy. From your description, Compaction should be slow. You can adjust isThrottleByBytes=true and increase the current limiting threshold of compactionRateByBytes

# Throttle compaction by bytes or by entries.
isThrottleByBytes=false

# Set the rate at which compaction will readd entries. The unit is adds per second.
compactionRateByEntries=1000

# Set the rate at which compaction will readd entries. The unit is bytes added per second.
compactionRateByBytes=1000000

Pulsar permissions

Issue 8: Pulsar permissions related information.

answer:

• Video: https://www.bilibili.com/video/BV1T741147B6?p=10
• Documentation: https://pulsar.apache.org/docs/en/security-overview/

The above is the summary of the 4th issue of the community FAQ. I would like to thank the friends who participated in the daily questions and answers of the community. Let's look forward to the next FAQ content!

related suggestion

• Community Knowledge Base | Frequently Asked Questions FAQ Collection Issue 1
• Community Knowledge Base | Frequently Asked Questions FAQ Collection Issue 2
• Community Knowledge Base | Frequently Asked Questions FAQ Collection Issue 3