Peripheral ecology｜RoP releases version 0.2.0: The architecture is newly upgraded, and the message accuracy reaches 100%

About the Author

Ran Xiaolong-R&D engineer of Tencent Cloud middleware team, Apache Pulsar committer, RoP author and Maintainer, Apache BookKeeper contributor, Apache Pulsar Go client, Apache Pulsar Go Functions, StreamNative/pulsarctl author.

Summary

A few days ago, the Tencent Cloud middleware team and the StreamNative team officially released the RoP 0.2.0 version. This version is newly upgraded in architecture. Users can completely avoid problems such as message loss, repeated consumption of messages, and only partial consumption of Partition data. .

Definition of RoP

Similar to KoP , MoP and AoP , RoP is a pluggable protocol processing plug-in.

After adding the RoP protocol processing plug-in to an existing Pulsar cluster, users can migrate existing RocketMQ applications and services to Pulsar without modifying the code, while still being able to use the powerful functions of Pulsar, such as:

• Separation of computing and storage
• Multi-tenant
• Cross-regional replication
• Hierarchical sharding
• Lightweight computing framework - Pulsar Functions
• ......

RoP 0.2.0 released

On May 17, 2021, the Tencent Cloud middleware team contributed the beta version of RoP 0.1.0 to the community. RoP (RocketMQ on Pulsar) introduces the RocketMQ protocol processing plug-in into the Pulsar Broker, so that Pulsar can support the native RocketMQ protocol. RocketMQ users can seamlessly migrate to Apache Pulsar.

Today, we released RoP 0.2.0, which is a new upgrade in architecture, and has been greatly improved in terms of functionality and stability. Provides ACL authentication and verification functions, which can better ensure the security of user data, and at the same time allow users to expand Partitioned Topic, obtain better concurrent writing capabilities, and improve RocketMQ's native control port interface. The service can be better processed and monitored.

Latest function optimization

In version 0.2.0, the Tencent Cloud middleware team made a new design based on the 0.1.0 architecture, refactored MessageID and message routing models to ensure the accuracy of RoP messages in different scenarios.

There are mainly the following three points of optimization content:

1. Support RoP ACL function

The ACL mechanism is a built-in capability of the RocketMQ community, which can well authenticate and authenticate user data. RoP 0.2.0 version reuses RocketMQ's own Hook implementation, and uses Pulsar's own authentication mechanism to realize the function of authenticating and authenticating user data.

The use of RoP ACL still continues the use of RocketMQ. You only need to define the ACL_ACCESS_KEY and ACL_SECRET_KEY fields, and then use RocketMQ's ACLRPCHook function to load it, which can ensure that users change the client's business code logic as little as possible.

The specific code example is as follows:

private static final String ACL_ACCESS_KEY = "eyJrZXlJZCI6InJvY2tldG1xLW13bmI3bWFwMjhqZSIsImFsZyI6IkhTMjU2In0."
+ "eyJzdWIiOiJyb2NrZXRtcS1td25iN21hcDI4amVfdGVzdCJ9.BDOjqqY25a6apnZTMZCqg0I0pxVFcqz7fvZbaTqkf5U"; // token
private static final String ACL_SECRET_KEY = "rop";
public static void producer() throws MQClientException {
DefaultMQProducer producer = new DefaultMQProducer("rocketmq-mwnb7map28je|nit", "ProducerGroupName",
getAclRPCHook());
...
}
static RPCHook getAclRPCHook() {
return new AclClientRPCHook(new SessionCredentials(ACL_ACCESS_KEY, ACL_SECRET_KEY));
}

• ACL_ACCESS_KEY: Token created by the user at the namespace level.
• ACL_SECRET_KEY: fixed value, this field will not be used when parsing inside RoP.

2. Refactor MessageID

RocketMQ is similar to Kafka in that both use 64-bit Offset to uniquely identify a message, but in Pulsar, 64-bit LedgerID and 64-bit EntryID are used to uniquely identify a message. To solve this problem, in RoP 0.1.0, we use the following form to construct the MessageID object:

• PartitionID: 8 bits, can allow a Topic to create up to 256 Partitions
• LedgerID: 32 bits
• EntryID: 24 bits

Using the above method may cause the message accuracy of MessageID to be lost. After the system has been running for a period of time, new LedgerIDs cannot be created, resulting in the situation that the services of the entire cluster are unavailable to the outside world. This problem is the same dilemma as the earlier KoP version, so in RoP 0.2.0, we adopted the same processing method as , using 161b07b223f4b4 PIP 70: Introduce lightweight broker entry metadata . In the protocol header, a 64-bit index/publish-time field is added, so that a 64-bit field can be added to each message for use without the need for protocol analysis on the client side.

PIP-70 is loaded using a plug-in, so when the service starts, we need to do the following configuration:

brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor

Note: Broker Entry Metadata is only supported in the version of Pulsar 2.8.0, so you need to make sure that the version of Pulsar Broker is 2.8.0 and above.

It should be noted that RocketMQ and Kafka use different ways of offset. There are two Offsets in RocketMQ, one is Queue Offset, which is used to indicate the position of the message in the MessageQueue. The MessageQueue is essentially an array. When a message comes in, the subscript of the array will be +1. One is CommitLog Offset used to indicate the location where the message is stored in CommitLog. Message storage is completed by the cooperation of ConsumeQueue and CommitLog. ConsumeQueue is a logical queue, CommitLog is actually storing message files, and ConsumeQueue is storing addresses that point to physical storage. Each MessageQueue under Topic has a corresponding ConsumeQueue file, and the content will also be persisted to disk.

Therefore, in the implementation of MessageID reconstruction, unlike Kafka, there is only one global Offset to identify the uniqueness of the message. In RoP, these two Offset situations need to be processed separately, as follows:

• RESERVED_BITS: 1 byte of reserved bits to avoid the occurrence of a negative number in the first byte, etc., which may cause an error in the offset calculation.
• RETRY_TOPIC_TAG_BITS: The 1-byte tag is used to identify whether this topic is a Retry type topic
• PULSAR_PARTITION_ID_BITS: 10-byte Partition Num, used to record how many Partitions there are under a PartitionedTopic, the maximum support is 1024 Partitions.
• OFFSET_BITS: 52 bytes are used to identify the Offset of the message.

3. Refactor the message routing model

In the version of RoP 0.1.0, in the implementation of message routing, RocketMQ and Pulsar first find the corresponding Owner Broker node through the Topic lookup operation, and then return the address of the Broker. But in this action, an important issue was ignored, that is, RocketMQ is different from Kafka and Pulsar, and its Queue is not globally unique.

The RocketMQ routing protocol mainly includes two parts:

1. The IP address information of the Broker service;
2. The total number of topic partitions corresponding to a certain Broker and the readable and writable information of the partitions.

In the RocketMQ routing protocol, there is no unique ID for the partition that globally identifies the topic (for example, in Pulsar/Kafka, the partition ID is unique in the cluster); while in RocketMQ, the partition routing information is identified by the broker plus the broker. The order is from 0→N Index to identify the Topic partition.

Therefore, in the RocketMQ protocol, the client only needs to obtain the total number of partitions on the Broker corresponding to the Topic, and can obtain the ID of the partition on the Broker through calculation; all requests are constructed based on [Broker-Tag] + [Broker-Topic-Seq] Unique routing query primitive to request service. Simply put: RocketMQ's partition is stateful, and it is bound to a specific Broker; once a partition is assigned to a Broker, it is related to it for life and cannot be migrated. The client resolves the partition routing information through calculation; for example, a TopicA has 5 partitions, which fall on BrokerA and BrokerB respectively, BrokerA has 3, and BrokerB has 2; then the protocol record is (BrokerA, 3) (BrokerB , 2), the client obtains all the partition data through calculation:

• BrokerA-TopicA-0，BrokerA-TopicA-1，BrokerA-TopicA-2
• BrokerB-TopicA-0，BrokerB-TopicA-1

Due to the above routing relationship, there is no way to GET_ROUTEINTO_BY_TOPIC with Pulsar's search protocol through the 061b07b223f6bb protocol request. The essential reason is that like Kafka/Pulsar, its Partition information is globally unique. After the Topic routing strategy is executed, it can accurately return who the Owner Broker corresponds to a particular Topic Partition. But RocketMQ's Topic route returns two fields, one is Broker Name, and the other is the number of Queue. The specific QueueID is calculated based on the number returned by the Broker in a fixed increment from 0. Therefore, in the routing mapping of Topic, RocketMQ and Pulsar's own routing protocols cannot be mapped one by one. In order to solve this problem, in RoP 0.2.0, a layer of Proxy is abstracted to maintain the mapping relationship between Topic and Broker. In order to achieve this goal, there are mainly the following things to consider:

1. Where are these mapping relationships stored?
2. How to distribute routing relationships?
3. What to do when the routing relationship changes?

In response to the first question and comprehensive consideration, we chose to store the routing mapping relationship in the ZooKeeper cluster, because the current RoP service itself also needs to rely on the ZooKeeper cluster and will not introduce new components; secondly, ZooKeeper's own consistency capabilities It satisfies the needs of this scene very well.

In response to the second question, while creating partition topics on the RoP interface, we search for the Broker nodes where each partition is located in turn, and write the mapping relationship to the ZooKeeper cluster based on the node information where the initial topic is located. The advantages of this are:

• Reuse the results calculated by Pulsar's own partition allocation mechanism, which is simple to implement.
• After the initial allocation, the virtual node and the physical node are on the same node with good performance.
• Optimal performance can be achieved if the ability to rebalance the routing relationship is coordinated.

In response to the third problem, we can reduce the impact of a single node failure on the system by increasing the Master-Slave mode. ZooKeeper metadata is as follows. You only need to add Broker-related information to realize the mutual master-slave relationship between each node. When the master node is unavailable, the slave node can continue to provide services. Since the current Offset information is stored in the Compact Topic, and all nodes subscribe at the same time, the metadata of each node can be guaranteed to be consistent, and the master-slave switch can be realized. The following is the route mapping relationship in the RoP cluster deployed in the test environment:

Therefore, in order to ensure that the RoP cluster has better fault tolerance, it is recommended to use an even number of nodes in the deployment of the RoP cluster. You can determine how many slave nodes the current Master node has as its backup nodes through the following parameter configuration:

RoPBrokerReplicationNum=2

Assuming there are 6 Broker nodes and RoPBrokerReplicationNum=2, it means that only three Master Broker nodes provide external services at this time. But for Pulsar, Broker nodes are peer-to-peer. When a topic is created, it may be allocated to any node. Therefore, for requests that are not on the Owner Broker node, a proxy is created on the RoP Proxy layer. First, perform a search operation on the topic, and then forward the request to the Owner Broker node to return.

future plan

In order to better implement the concepts of open source collaboration and open source co-construction, the above functions have been contributed back to the community. In addition to the arbitrary delayed message function of the commercial version of RocketMQ, the Tencent Cloud middleware team also developed related plug-ins based on the native features of Pulsar to support it. In addition to supporting multiple levels of delayed messages, RoP's delayed message function also has the ability to support arbitrary delayed messages.

After that, the Tencent Cloud middleware team will continue to develop RoP-related functions while ensuring the stability of the RoP project, such as message tracing, message query and backtracking, and monitoring capabilities to further improve the functions of RoP and the surrounding ecology.

RoP project address: https://github.com/streamnative/rop

Special thanks

Thanks to Han Mingze and Zhang Yonghua from the Tencent Cloud middleware team for verifying and supporting the technical details of this article.

Follow the public account "Apache Pulsar" for dry goods and news

joined the Apache Pulsar Chinese exchange group 👇🏻

Click link to download the latest version of RoP!