Introduction to For a long time, RocketMQ's easy-to-deploy, high-performance, and highly available architecture has supported a large number of business scenarios inside and outside the group for decades. Today, in order to meet the new challenges of today's cloud-native era, we have launched the RocketMQ 5.0 new architecture.
Author | Ling Chu
introduction
For a long time, RocketMQ's easy-to-deploy, high-performance, and highly available architecture has supported a large number of business scenarios inside and outside the group for decades. Today, in order to meet the new challenges of today's cloud-native era, we have launched the RocketMQ 5.0 new architecture.
In the 5.0 new architecture, we have updated the entire RocketMQ network topology model, focusing on stripping the higher-level business logic from the broker to the stateless proxy, so that independent computing nodes can take on future upgrade and release tasks without loss, and At the same time, the broker is liberated to undertake pure storage tasks, paving the way for a stronger message storage engine in the future. For the communication layer, for standardization and multi-language considerations, we discarded the RemotingCommand protocol that RocketMQ has used for many years and adopted gRPC to implement the communication logic between the client and the server.
For the user side, we hope to upgrade as little as possible to disturb customers, maintain a lightweight logic, easy to maintain, and good observability, so that we can achieve "doing things right at one time".
At present, to ensure that the interface is fully compatible, the commercial version of the Java SDK based on RocketMQ 5.0 has been released in the public cloud, and the open source version will be released soon. The SDK will support both the cloud version of the proxy architecture on the cloud and the open source version of Broker. The following will describe the iteration and evolution of the SDK under the RocketMQ 5.0 new architecture.
Fully asynchronous
1. The original intention of asynchronous
Because many network IOs are involved, RocketMQ opens up two sets of APIs, synchronous and asynchronous, for message sending for users to use. The old architecture maintains two sets of similar business logic for synchronization and asynchronous from API, which is very unfavorable for iteration. With this in mind, the new architecture SDK hopes to unify them at the bottom.
Taking message sending as an example, a complete message sending link includes obtaining:
- Get the route corresponding to the topic;
- Select the corresponding partition according to the route;
- Send the message to the specified partition. If it fails to be sent to the partition, the next partition will be sent again until the maximum number of retries is reached; if the sending is successful, the sending result will be returned.
Obtaining the route corresponding to the topic from the remote end is a heavy IO operation, and sending a message itself is also a heavy IO operation. In the past implementation of sending, even if it is sent asynchronously, the acquisition of the route is also synchronous. The acquisition of the route itself is not included in the user's sending time. The user can independently set the timeout period of message sending, and because The sending of the message itself is synchronous, and the precise control of the timeout time cannot be achieved. After using the asynchronous Future, it can be done very conveniently by controlling the timeout time of the Future.
2. Asynchronously unify all implementations
Essentially all heavy IO operations in RocketMQ can be unified through asynchronous. Thanks to the Future-based stub provided by gRPC itself, we connect the Future of the network layer layer by layer to the final business layer. When the user needs a synchronous API, wait synchronously; when the user needs an asynchronous API, add a callback to the outermost Future for monitoring.
In fact, the idea based on Future design is implemented throughout the entire client. For example, message consumption is also accomplished through the only implementation based on Future:
/** * Deliver message to listener. * * @param messageExtList message list to consume * @param delay message consumption delay time * @param timeUnit delay time unit. * @return future which contains the consume status. */public ListenableFuture<ConsumeStatus> consume(List<MessageExt> messageExtList, long delay, TimeUnit timeUnit) { // Omit the implement}In view of the failure of sequential message consumption, which requires local suspend for a period of re-delivery, the consumption interface adds a delay parameter. However, regardless of whether it is a normal message or a sequential message, only the Future containing the consumption status will be returned. The upper layer then ACK/NACK the message for the Future containing the consumption state. In particular, for the scenario where the server delivers a specific message to the client for consumption verification, the current Future interface is also called, and then the consumption result is packaged and the server responds to the consumption result.
The sending and consumption process of RocketMQ itself is full of a lot of asynchronous logic, and the use of Future makes a large number of interface implementations streamlined and unified. Especially in our IDL based on the new gRPC architecture protocol, in order to keep it simple, we all use unary rpc (non-streaming), so that we can all use gRPC's Future stub to complete communication requests.
Observability enhancement
The above picture is from an important blog post by Peter Bourgon in 2017, which systematically and in detail explains the characteristics and definitions of metrics, tracing and logging, and the relationship between them.
- Metrics: Aggregate statistical information of similar data for early warning and monitoring.
- Tracing: Associate and analyze metadata on the same call chain to determine abnormal and blocking behaviors on the specific call chain.
- Logging: Record discrete events to analyze program behavior.
In the cloud native era, observability is one of the core competitiveness of cloud products. Therefore, the keynote of enhanced observability was determined at the beginning of the development of the entire new architecture. While the client logic of the old architecture is complex, the lack of observability has also led to a lack of intuitive and convenient means when facing customer work orders. Therefore, in the new architecture, we focus on the three important aspects of Tracing, Logging and Metrics. All-round observability is improved.
1. Full link Tracing
Tracing is embodied in message middleware. The most basic thing is to monitor and record the entire life cycle of each message itself, such as sending, pulling, consumption, ACK/NACK, transaction submission, storage, and deletion. It is the most basic in RocketMQ. The basic implementation is the message track.
The old message trajectory uses a private protocol for encoding and decoding, and the observation of the message life cycle is also limited to the sending, consumption, and transaction-related stages. It is not unified with the open source specification, nor does it have the ability to share context between the trace of the message itself and the trace of the user link.
In the new implementation, the latest CNCF OpenTelemetry community protocol specification is embraced, and an OTLP exporter is embedded in the client to send tracing data to the proxy in batches. The proxy side scheme is more diverse. It can be used as a collector to process data. Integration can also be forwarded to other collectors. The proxy side will also have corresponding tracing data, which will be merged with the tracing data reported by the client for processing.
Due to the open source standard OTLP exporter and protocol, it is possible for users to define the corresponding collector address. In the commercial version, we collect and integrate the tracing data on the user client side and the tracing data on the server side for managed storage. In the open source version, users can also customize their collector address to report the tracing data to their own platform for analysis and processing.
For the entire message life cycle, we redesigned all span topology models. Take the simplest message sending, receiving, consumption and ACK/NACK process as an example:
in:
- Prod: Produce, which means the message is sent, that is, the start time is when the message starts to be sent, and the end time is the result of receiving the message (the internal retry of the message will record a separate span);
- Recv: Receive, which means the reception of the message, that is, the start time is the client's request to accept the message, and the end time is the reception of the corresponding response;
- Await: Indicates that the message arrives at the client until the message starts to be consumed;
- Proc: Process, represents the message consumption process;
- Ack/Nack means the process of the message being Ack/Nack.
In this process, the relationship between each Span is as follows:
The commercial version of ONS also supports the new version of Trace on the control side, giving a more detailed display of the message production time, specific consumption status, consumption time, waiting time, and consumption times that users care about.
Observe the topological relationship between the producer and consumer span through the trace service of SLS (the link relationship is not shown, so there is no receive-related span in the figure):
OpenTelemetry's specifications related to messaging span are constantly iterating in the community, which involves specific tracing topologies, span attribute definitions (that is, attribute semantic conventions) and so on. We also initiated a preliminary Pull Request with RocketMQ related content to the community OpenTelemetry specification in the first time, and it was included and affirmed by the community. Thanks to the detailed and standardized definition of the OpenTelemetry specification, we have added a large amount of information to the tracing data, including but not limited to program runtime, operating system environment and version (ie resource semantic conventions), which is conducive to online problem discovery and troubleshooting.
Regarding tracing context propagation, we have adopted W3C standards to serialize and deserialize the trace context and pass it back and forth between the client and the server. In the next version, we will also provide an interface for users to customize the trace context so that users You can easily associate RocketMQ with your own tracing data.
In the new architecture, we expose a lot of hook points for different nodes in the message life cycle. The logic of tracing is also implemented based on these hook points, so it can remain relatively independent. After the complete new architecture is pushed to open source, the entire tracing related logic will also be extracted into a special instrumentation library and contributed to the openTelemetry community.
2. Accurate and diverse Metrics
Tracing is more about observing the trend of messages from the perspective of the call chain. More often, we hope that there can be aggregated Metrics and corresponding dashboards that can be observed from a more macro perspective for common data. If tracing can help find and locate problems better and faster, then Metrics provides important multi-dimensional observation and early warning methods.
After collecting enough tracing data, the server will perform a secondary aggregation of the data to calculate the percentile of the data sent, waiting, and consumption time of the user, which can make a good judgment on many glitch problems. .
3. Standardized Logging
In our development practice, we strictly define the log content according to the levels of Trace, Debug, Info, Warn, and Error. For example, the Trace level will respond to each RPC request and response, and each message is recorded from entering the client to the error. Once the log of the level is printed, it must be worthy of our attention and our customers. While removing a large amount of redundant information, key nodes, such as load balancing, transmission failure retry and other critical links also complete a large amount of information, and the information density of a single-line log is greatly increased.
In addition, with regard to the implementation of the log module, RocketMQ was originally developed by itself. Compared with external implementations such as logback and log4j2, the function is relatively single and the cost of secondary development is relatively high. Logback was not used in the selection. Basically, I just wanted to avoid conflicts with the user log module. After investigating many solutions, I chose the shade logback method to replace it. The shade here not only replaces the package name and coordinates, but also modifies the official log configuration file name and many internal environment parameters of logback.
For example, the default configuration file:
| <span class="lake-fontsize-1515">library</span> | <span class="lake-fontsize-1515">default configuration file</span> |
| "lake-fontsize-1515">standard logback</span> | <span class="lake-fontsize-1515">logback.groovy/logback-test.xml/logback.xml</span> |
| < span class="lake-fontsize-1515">logback for rocketmq</span> | <span class="lake-fontsize-1515">rocketmq.logback.groovy/rocketmq.logback-test.xml/rocketmq.logback. xml</span> |
`
// redirect JUL logging to slf4j.// see https://github.com/grpc/grpc-java/issues/1577SLF4JBridgeHandler.removeHandlersForRootLogger();SLF4JBridgeHandler.install();
`
# Update of consumption model
The consumption model of RocketMQ's old architecture is very complicated. The message in the topic itself is stored according to the MessageQueue, and the client pulls, caches, and delivers the message according to the MessageQueue during consumption.
ProcessQueue has a one-to-one correspondence with MessageQueue in RocketMQ, and is basically one of the most complex structures in the client-side consumer logic. In the client of the old architecture, after the message is pulled, the message will be cached in the ProcessQueue. When it needs to be consumed, the corresponding message will be taken out of the ProcessQueue for consumption. When the consumption is successful, the message will be removed from the ProcessQueue. . The sending of the retry message and the update of the location are interspersed in this process.
## 1. Design ideas
In the new client, the introduction of the pop consumption mode makes the logic of processing retry messages and location updates separately removed. The user’s consumption behavior becomes
1. Pull messages
2. Consumer news
3. ACK/NACK message to the far end
Because the pulled message will be cached in the client memory first, so the size of the message cache must be calculated during the consumption and pull process to protect the program, so each ProcessQueue in the new client maintains two Two queues: cached messages and pending messages. After the message arrives at the client, it will be placed in the cached messages first, and will be moved from the cached messages to the pending messages when it is ready for consumption, and will be removed from the pending messages when the message consumption is over and is Ack.
The client of the new architecture streamlines the implementation of ProcessQueue, and the encapsulation is also better. For consumers, there are actually only four core interfaces.
`
public interface ProcessQueue { / Try to take messages from cache except FIFO messages. @param batchMaxSize max batch size to take messages. @return messages which have been taken. / List<MessageExt> tryTakeMessages(int batchMaxSize); / Erase messages which haven been taken except FIFO messages. @param messageExtList messages to erase. @param status consume status. / void eraseMessages(List<MessageExt> messageExtList, ConsumeStatus status); / Try to take FIFO message from cache. @return message which has been taken, or {@link Optional#absent()} if no message. / Optional<MessageExt> tryTakeFifoMessage(); / Erase FIFO message which has been taken. @param messageExt message to erase. @param status consume status. */ void eraseFifoMessage(MessageExt messageExt, ConsumeStatus status);}
`
For ordinary consumers (non-sequential consumption), ProcessQueue#tryTakeMessages will fetch messages from Cached messages (after fetching messages, they will automatically move from Cached messages to Pending messages), and carry the corresponding consumption results after message consumption is over. Call ProcessQueue#eraseMessages. For sequential consumers, the only difference is that the corresponding method calls are replaced with ProcessQueue#tryTakeFifoMessage and ProcessQueue#eraseFifoMessage.
And ProcessQueue#tryTakeMessages and ProcessQueue#tryTakeFifoMessage itself already contains the logic of consumption current limiting and sequential consumption in order to ensure the sequence to lock the queue, that is to say: once ProcessQueue#tryTakeMessages/ProcessQueue#tryTakeFifoMessage can get the message, then the message must be Meet the conditions of being consumed. After the consumer obtains the consumption result, he brings the consumption result and executes ProcessQueue#eraseMessage and ProcessQueue#eraseFifoMessage. The erase process will complete the ACK/NACK of the message and the logic of unlocking the queue during sequential consumption.
After simplification, the upper-level consumer logic basically only needs to be responsible for submitting consumer tasks to the consumer thread. Any logic that can be said to be'Process' is closed loop in the new ProcessQueue.
# Compatibility and quality assurance
The entire SDK of the new architecture relies on protocol buffers, gRPC-java, openTelemetry-java and many other libraries. While simplifying RocketMQ's own code, it also brings some compatibility issues. RocketMQ itself maintains compatibility with Java 1.6, however:
* gRPC-java no longer supports Java 1.6 after version 1.15 in 2018;
* openTelemetry-java only supports Java 8 and above.
During this period, we also investigated the status quo of related SDKs from other vendors such as AWS and Azure, and found that giving up support for Java 1.6 is already a standard practice in the industry. However, due to the old customers sticking to Java 1.6, we have also made some changes:
* The code of protocol buffers has been replaced with the equivalent of Java 1.6 and passed all the single tests of protocol buffers;
* The gRPC code has been replaced with Java 1.6 equivalents and passed all gRPC single tests;
* For openTelemetry, a large number of functional tests have been carried out while performing equivalent substitutions;
In terms of single test, the current client guarantees more than 75% line coverage, but there is still a long way to go compared to good open source projects. We will continue to improve this point in subsequent iterations.
# finally
RocketMQ 5.0 is the largest version of the architecture upgrade since open source. There are still many details of the specific implementation process that have not been disclosed. As the space cannot be exhaustive, everyone is welcome to put forward more valuable opinions in the community during the follow-up open source process.
## related links
* Pull Requesthttps://github.com/open-telemetry/opentelemetry-specification/pull/1904
* Metrics, Tracing, and Logginghttps://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html
* Apache rocketmq apishttps://github.com/apache/rocketmq-apis
* OpenTelemetry specificationhttps://github.com/open-telemetry/opentelemetry-specification
Alibaba Cloud Native Messaging Middleware Team is recruiting, and everyone is strongly welcome to recommend and recommend!
If interested, please contact:
@凌楚(yangkun.ayk@alibaba-inc.com)
@尘央(xinyuzhou.zxy@alibaba-inc.com)
click on the link below to see more recruitment details!
https://job.alibaba.com/zhaopin/position\_detail.htm?spm=a2obv.11410903.0.0.674944f6oxzDCj&positionId=134677
> Copyright Notice: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users, and the copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。