Alibaba Cloud Message Queuing RocketMQ 5.0 brand new upgrade: message, event, stream integration processing platform

From "message" to "message, event, flow"

As the communication infrastructure of contemporary applications, message queues are the core reliance of microservice architecture applications. Through asynchronous decoupling capabilities, users can build distributed, high-performance, flexible and robust applications more efficiently.

From the perspective of data value and business value, the value of message queues continues to deepen. The core business data flowing in the message queue involves different links and scenarios such as integrated transmission, analysis, calculation, and processing. With continuous evolution, we can foresee that message queues will inevitably generate new value and create new "chemical reactions" in scenarios such as data channels, event integration driving, analysis and calculation.

RocketMQ was born in Alibaba's internal e-commerce system. Today, its core architecture has undergone several key evolutions:

As early as 2007, when Taobao's e-commerce system was split into services, the first generation of messaging service Notify was born, which was the earliest prototype of RocketMQ. Notify uses a relational database as storage, using push mode. In the high-frequency trading scenario such as Alibaba Taobao, it has a very wide range of applications.

During the period 2007-2013, with the development of Ali Group's business, it was not only necessary to call transaction scenarios asynchronously, but also to support a large amount of data transmission and data synchronization. At this time, MetaQ and RocketMQ3.0 versions were derived internally. These two versions began to explore self-developed storage engines, adopted self-developed proprietary message storage, supported single-machine massive Topics, and prospectively removed the external components such as Zookeeper rely. Today, ten years later, we see that various keeper has become the mainstream of the development of the entire messaging field.

After the first three generations of internal business polishing, Alibaba actively participated in open source and contributed RocketMQ3.0 to the open source community, and graduated from the Apache incubator in 2017, becoming China's first non-Hadoop ecosystem top Apache community project. Since then, RocketMQ has also begun to serve Alibaba Cloud enterprise customers. Adhering to the trinity development strategy of open source, commercial and internal, the 4.x version released in 18 years focused on optimization in terms of high reliability and low latency, built a new low-latency storage engine and multi-scenario disaster recovery solutions, and provided a wealth of news characteristic. This also makes RocketMQ the preferred solution for financial-level business messaging.

The community released the RocketMQ5.0-preview version last month, officially announcing the arrival of 5.0. RocketMQ5.0 will no longer be limited to the basic scene of message decoupling, but will provide integrated processing capabilities of messages, events, and streams through the advantages of a unified kernel and storage.

Looking back over the more than ten years of RocketMQ development, a good community environment and commercial support have made it easy for a large number of enterprise developers to follow up on business characteristics and demands for selection and verification. In terms of community active influence, the RocketMQ community project has harvested 15,000+ Stars, with 400+ active contributors, and 30+ surrounding active projects such as multi-language, ecological connection, etc., which are well received by community developers. In terms of application scale, RocketMQ, as a financial-level business messaging solution, has accumulated tens of thousands of corporate customers in many industries such as Internet games, online education, financial securities, banking, government and enterprise energy, and automobile travel. At the same time, it is responsible for the core business link within Alibaba, and circulates trillions of message traffic every day, surpassing the zero peak of the previous double elevens. In terms of industry evaluation, RocketMQ has also won many awards.

Official announcement: Alibaba Cloud's new generation of RocketMQ "message, event, stream" integrated processing platform

Today, Alibaba Cloud Message Queuing RocketMQ version 5.0 is released, which we call a one-stop "message, event, and stream" integrated processing platform.

There are two new highlights of the new version of the core. The first is the expansion and layout of the message core scene. RocketMQ 5.0 is no longer limited to the message decoupling scene, and the new layout event-driven and message streaming processing scene; the second is the one-stop integrated processing Technology architecture and trends.

The technical architecture of "message, event, stream" one-stop integrated processing can realize a message storage, support stream computing of messages, asynchronous delivery, and integration-driven multiple scenarios, which greatly reduces the technology of business personnel to operate and maintain multiple systems Complexity and operation and maintenance costs. It can be said that the unified RocketMQ5.0 product stack can handle uniformly whether it is microservice instruction calls, asynchronous notifications, CDC change logs, behavioral embedded data, resource operation and maintenance, and audit events.

Major announcement one: a new upgrade of the RocketMQ infrastructure

First of all, the most important upgrade is the comprehensive renewal of the technical architecture of Alibaba Cloud RocketMQ.

The new RocketMQ5.0 version sinks the general storage logic, centrally solves the technical problems of multiple copies of message storage, low latency, massive queue partitioning, and separates the upper-level message processing and complete stateless computing layer to complete the protocol. Adaptation, authority management, consumption status, and observable operation and maintenance system support. Thanks to the architecture design of separation of storage and calculation, the full link from SDK access to online operation and maintenance has brought a comprehensive improvement:

1. The openness of the lightweight SDK and the improvement of the full-link observable system: It supports both the 4.x communication protocol and the new gRPC communication protocol, and built-in OpenTelemetry embedding support. The new version of the SDK adds more than 10 indicator embedding points.
2. Message-level load balancing: The new version of the SDK no longer participates in the load balancing of the actual storage queue, and the message load balancing will be more lightweight, with a single message as the smallest unit of scheduling.
3. Multi-network access support: The new version supports a single instance to expose the public network, intranet and other access forms at the same time, which is convenient for customers to access multiple networks.
4. Massive hierarchical storage: The new version opens the ability to store historical messages in hierarchical storage. The messages are low-cost and have no size limit, and they can be stored for up to 30 days. The cold and hot data are designed to be separated, which greatly reduces the impact of consumption history messages on the performance of the instance.

Major announcement 2: RocketMQ Streaming cloud best practice-message ETL

In addition to the improvement of messaging infrastructure capabilities, Alibaba Cloud RocketMQ has launched a lightweight messaging ETL function in Streaming scenarios.

Messages generated by users in scenarios such as database changes, terminal data reporting, and back-end burying logs. Typical consumption scenarios are data cleaning and conversion, and they are stored in external storage, offline analysis, and online analysis systems at the same time. Traditional implementation schemes need to build heavyweight real-time computing services such as Flink or self-built consumer applications for message processing. Using the commercial version of RocketMQ ETL function, simple console configuration can achieve message cleaning and conversion. RocketMQ ETL function has three advantages:

1. Lightweight and non-dependency: As the native messaging function of Alibaba Cloud, there is no need to deploy external computing services or consumer programs when used, and the solution is lighter.
2. Low development threshold: Built-in common cleaning conversion templates to meet most of the message content processing needs, and support users to quickly write custom functions to support special business logic. The overall development cost is very low, and the business can be launched in one hour.
3. Serverless elasticity: There is no need to estimate the capacity in advance, and the Serverless serverless mode is adopted to achieve on-demand elastic scaling.
    

Major Announcement 3: Best Practices on EDA Cloud-Event Center EventBridge

The last release point of RocketMQ this time is the layout and evolution of event-driven business scenarios. As early as 2018, the Gartner assessment report listed EDA (Event-Driven-Architecture) as one of the ten strategic technology trends, and event-driven architecture will become the mainstream of microservices in the future. We first define:

Event-driven, in essence, is a re-upgrade of message-driven, and it is the next inevitable stage of the deep evolution of enterprise IT architecture.

The difference and correlation between event-driven architecture and message-driven architecture mainly focus on the following three points:

1. EDA puts more emphasis on deep-level decoupling: message-driven is the decoupling of calls between different components in the same business and organizational system at the technical architecture level, and its information encapsulation and processing are all expected and predefined. Event-driven adaptation is a broader business and organization system. Event-based decoupling does not require expectations and behavior definitions between upstream and downstream. The upstream and downstream uniformly follow standardized specifications, which is a deeper decoupling.
2. EDA puts more emphasis on connection capabilities: message-driven are more calls within a single system, while event-driven often involves the collaboration of different regions, account entities, and three-party SaaS. A major feature of event-driven is the strong ecological connectivity.
3. EDA puts more emphasis on serverless low-code development: analogous to the collaborative relationship between messaging and microservices, the general trend of serverless business architecture in the future will promote the business development model to gradually shift to low-code configuration. Another major feature of event-driven is low-code development. Based on rich tool capabilities, the business side does not need to write a large amount of production and consumption code like message-driven.

Therefore, the EventBridge product of Alibaba Cloud Unified Event Center brings the following capabilities:

1. Unified and standardized event integration ecology: As an Alibaba Cloud event center, it integrates business events from more than 80 cloud products and supports more than 800 event types. Users can use EventBridge to manage all cloud product resource changes, operation and usage events at one time, avoiding docking The repetitive work of multiple product interfaces.
2. Global event interconnection network: to implement the attribute capability of event-driven strong connection, this time the global event interconnection network was released, which is the first batch to support the interconnection of five major regional events in China. Enterprise customers can realize cross-account, cross-regional, and cross-network event aggregation and circulation through simple configuration.
3. Serverless low-code development: built-in more than ten kinds of event targets and processing templates, covering most business scenarios, simple configuration by customers, low-code, and event driving and processing can be completed without deploying services.

Facing the future: Resolutely promote the development of the integration of "messages, events, and streams"

The release of RocketMQ5.0 marks that Alibaba Cloud Messaging has officially moved from the messaging field to a new situation in which the scene of "message, event, and flow" is integrated. In the future, the evolution of Alibaba Cloud's messaging products will continue to focus on the core scenarios of messages, events, and streams. The message infrastructure itself will also continue to move forward towards serverless flexibility, strong disaster tolerance, observable and free operation and maintenance, and provide customers with high-performance, high-reliability, and strong disaster-tolerant high SLA services; and in Streaming Based on the customer’s business demands, the joint ecological product will continue to launch more message processing computing services; create a future-oriented enterprise integration model, and work with ecological partners and open source communities to vigorously promote the further development of event-driven.

Click the "link below" 👇 to watch the important review of the enterprise-level Internet architecture sub-forum.
https://yunqi.aliyun.com/2021/agenda/session184

For more information about the message queue RocketMQ5.0, please search WeChat account (AlibabaCloud888) to add a cloud native assistant! Join the group to get cloud-native first-hand information~