Apache Pulsar 2.7.3 version introduction

The original author of this article is StreamNative engineers Cong Bo and Liu Yu. Translator Liu Zilin, Chuanzhi Education Engineer.

About Apache Pulsar
Apache Pulsar is the top-level project of the Apache Software Foundation. It is the next-generation cloud-native distributed message flow platform. It integrates messaging, storage, and lightweight functional computing. It uses a separate architecture design for computing and storage to support multi-tenancy, persistent storage, Multi-computer room and cross-regional data replication, with strong consistency, high throughput, low latency and high scalability and other streaming data storage characteristics. GitHub address: http://github.com/apache/pulsar/
Recently, the Apache Pulsar community released Pulsar 2.7.3 version! The new version includes improvements and bug fixes provided by 32 contributors, and 79 changes have been submitted.

Version highlights:

• Cursor reads follow the setting of the scheduling byte rate limiter and no longer cause unexpected results. [1]
• Ledger scrolling tasks are performed as expected. [2]

This blog introduces the most noteworthy progress of version 2.7.3. For a complete list of all performance upgrades and bug fixes, please refer to the Pulsar 2.7.3 release notes[3].

Bug fixes and performance upgrades

Broker

PR-9826[4]: cursor reading follows the limit of the scheduling byte rate limiter.

question : Neither the namespace nor the topic policy considers the byte rate limit when limiting the distribution rate.

Solution : Fixed the behavior of scheduling byte rate limiter settings. Cursor reads will follow this setting and will not cause unexpected results.

PR-11226[5]: Ledger rolling plan task is executed as expected.

Issue : Before this PR, ledger performed a rolling task before reaching the maximum rolling time, which caused the ledger to be unable to roll in time.

Solution : Fix the time of the ledger rolling scheduling, the task can only run after the ledger is successfully created.

PR-11136[6]: When restarting the broker, the topic-level retention policy can work normally.

Issue : Before this PR, when a topic-level retention policy was set for a topic and then the broker was restarted, the topic-level retention policy did not take effect.

Solution : Fixed the behavior of this policy so that it replays all policy messages after starting policyCacheInitMap, and adds a retention policy check test when restarting the broker.

PR-10977[7]: calling lastMessageId API will no longer cause memory leaks.

issue : Before this PR, there was a memory leak when calling the lastMessageId API, which caused the broker process to be stopped by Kubernetes.

Solution : Added a missing entry.release() call to PersistentTopic.getLastMessageId to ensure that the broker does not run out of memory.

[8]: 16125c04c71eb6 ZooKeeper reads are cached by the broker.

Question : When performing management operations to obtain the tenant's namespace, ZooKeeper uses the ZooKeeper client to read it, not from the broker cache.

Solution : Fix the caching problem when ZooKeeper obtains the namespace list for tenants.

PR-10512[9]: The monitoring thread that calls LeaderService.isLeader() is no longer blocked.

Issue : When LeaderService changes to leadership state, it will be locked by a synchronized block, which also prevents other threads from calling LeaderService.isLeader().

Solution : By modifying ClusterServiceCoordinator and WorkerStatsManager to check whether it comes from the leader of MembershipManager, the deadlock condition of the monitoring thread is fixed so that it is not blocked by LeaderService.isLeader().

PR-10414[10]: hasMessageAvailable can successfully read the message.

problem: is filtered by acknowledgmentsGroupingTracker because of the message, and the message cannot be read when hasMessageAvailableAsync returns true.

solution: fixes the race condition by modifying the acknowledgmentsGroupingTracker to filter duplicate messages, and then clean up the messages when the connection is opened.

Proxy

PR-8048[11]: Proxy supports automatic creation of partition topics.

Issue : Proxy did not create a partition because it uses the current ZooKeeper metadata.

Solution : By selecting and obtaining from the available brokers, instead of using the current ZooKeeper metadata to change the proxy to process PartitionMetadataRequest.

Pulsar admin

PR-11140[12]: adds a flag to indicate whether to create a metadata path on the replicated cluster.

Issue : When creating a partition topic on the replicated namespace, the metadata path /managed-ledgers is not created on the replicated cluster.

Solution : Added an identifier (createLocalTopicOnly) to indicate whether to create a metadata path for the partition topic in the replication cluster.

PR-11131[13]: prohibits setting policies for non-existent topics.

Question 16125c04c72127: Due to the redirection loop in the topic strategy, users can set strategies for non-existent topics or

Solution : This fix adds an authoritative identifier to the topic strategy to avoid redirection loops. Users cannot set topic policies for non-existent topics or single-partition topics. If you set a policy for the partition of topic 0, it will be redirected to the broker.

PR-10806 [14]: service discovery no longer hard-codes topic as persistence.

Issue : When using the lookup service to discover a partition's non-persistent topic, it will return 0 instead of the number of partitions. The Pulsar client will try to connect to the topic by connecting to a common topic.

Solution : Implement topicName.getDomain().value() instead of hard-coding persistent. Now users can successfully use service discovery for non-persistent topics in a partition.

PR-10744[15]: other connectors can now use Kinesis Backoff class

Issue : The Kinesis sink connector Backoff class in the Pulsar client implementation project combined with the dependency org.apache.pulsar:pulsar-client-original increased the size of the connector.

Solution : Added a new class Backoff to the function io-core project so that Kinesis sink connector and other connectors can use this class.

Client

PR-10506[16]: Cannot send a FLOW request with zero permission.

Issue : When the broker receives a zero-permission FLOW request, it will throw an exception and close the connection. This will cause frequent reconnections and result in repeated or out-of-sequence messages.

Solution : Added a verification function to verify the permission of the FLOW request before sending it. If the request is a zero permission, then the FLOW request cannot be sent.

Functions and connectors

PR-10769[17]: Kinesis sink connector confirms the success message.

Issue : The Kinesis sink connector has no confirmation message after sending successfully.

Solution : Added the confirmation after the message is sent successfully for the Kinesis sink connector.

Docker

PR-10531[18]: When running with Kubernetes, the function name cannot exceed 52 characters.

Question : When running with Kubernetes, if the submitted function length is valid (less than 55 characters), a StatefulSet that cannot generate pods will be created.

Solution : Change the maximum length of the function name of the Kubernetes runtime from 55 characters to 53 characters. With this fix, the length of the function name cannot exceed 52 characters.

rely

PR-10907[19]: After starts TLS, the connection between pulsar-admin and proxy is stable.

Issue : Because Jetty 9.4.39 introduces an SSL cache error, pulsar-admin is unstable in TLS connection, which causes frequent failures of uploading large function jar packages.

Solution : Upgrade Jetty to 9.4.42.v20210604, so that when TLS is enabled, the connection between pulsar-admin and proxy is the most stable.

participate

New version use

Everyone is welcome to download [20] and use the new version! If you encounter problems during use, you can raise questions and communicate with the community by raising issue[21] or communicating in the WeChat group.

Join the Apache Pulsar community

The growth of the Pulsar project comes from the community, and it is also rooted in the community. The preparation and release of new versions from time to time are inseparable from the contributions of community partners. Are you willing to be one of them?

Participating in open source can be recognized by the company and the community inside and outside, and make friends with like-minded people from various fields; at the same time, it can also increase personal influence and promote personal development. Participating in open source is not exclusive to code farmers. Community, documentation and other aspects can allow everyone to play their skills.

As a global open source project, as of now, Apache Pulsar has 435 contributors, 9.4K+ Star, and 2.3 K+ Fork. We have provided you with a guide for participation, and welcome more and more small partners to help the continuous development and advancement of the Apache Pulsar project.

• Apache Pulsar official contribution guide [22]

• Join the Apache Pulsar volunteer family

Translator information
Hi~ My name is Liu Zilin, a slash young man who can code~

Apache Pulsar 2.7.3 version introduction

About Apache Pulsar

Version highlights:

Bug fixes and performance upgrades

Broker

Proxy

Pulsar admin

Client

Functions and connectors

Docker

rely

participate

New version use

Join the Apache Pulsar community

Translator information

Recommended reading

ApachePulsar

引用和评论

深入解析 Apache BookKeeper 系列：第二篇 — 写操作原理

解决阿里云ECS服务器拉取不下docker镜像的问题

美国居民IP节点

docker 打包 opencv-python，libGL.so.1 报错解决方法

记录修复 cd 系统报错整个过程

Docker入门系列之三：如何将dockerfile制作好的镜像发布到Docker hub上

docker 踩坑