RocketMQ message integration: multi-type business messages-common messages

introduction

Since its inception, Apache RocketMQ has served 100% of the internal business of Alibaba Group and tens of thousands of enterprise customers of Alibaba Cloud after more than ten years of large-scale business stability. As a financial-grade reliable business messaging solution, RocketMQ has been focusing on the construction of asynchronous communication capabilities in the field of business integration since its inception. This article will start with the demands of business integration scenarios, introduce the core capabilities and advantages of RocketMQ as a business message integration solution, and introduce the use of RocketMQ common message types from the perspectives of functional scenarios, application cases, and best practices.

Follow the official account of [Alibaba Cloud Cloud Native] and reply to the keyword [ordinary news] to get the full PPT download address!

Speaking of business integration scenarios, the initial usage scenario of RocketMQ is a typical representative. RocketMQ was born in Ali's e-commerce system. The e-commerce system often needs to do various promotion activities. Under such complex demand scenarios, it has extremely high capabilities for the throughput performance, end-to-end delay, peak-shaving and valley-filling of the messaging system. Require.

Summarize today's core issues in one sentence, what are the characteristics and requirements of messages running in the core transaction business link, and what are the differences between messages running in scenarios such as offline analysis. Let’s discuss with you all below~

Business Integration vs Data Integration

Integration goals are different

When designing the business core architecture, it is often necessary to complete the business logic design for the upper-level requirements. Taking the e-commerce transaction scenario as an example, through the splitting of microservices, the entire link may be split into many links. When different applications are integrated through messages, more attention is paid to the flow of user orders. Whether this business logic will be processed normally, this is business integration.

In contrast, data integration is data-centric, and more attention is paid to the data generated by business integration to analyze the value of these business data. Data integration does not care where the data comes from, but only cares about the attributes of the data itself and the relationship between the data.

different focus

In business integration, with the expansion and complexity of enterprise business logic, the coupling between the caller and the callee will gradually increase, and the topology of the link will become more and more complex. It is often the case that the upstream of one message is the downstream of another, a service may be both a sender and a consumer, and so on.

In the data integration scenario, the above links are not concerned, but the diversity of data is more concerned. That is to say, when doing data integration analysis, it is more to extract and aggregate these data from various heterogeneous data sources, and then aggregate the data of these heterogeneous systems for cleaning, and finally aggregate them into structured data or reports for analysis. Data integration is more concerned with the heterogeneity and diversity of data.

real-time difference

A simple understanding of business integration is an online logic, or a strong real-time logic. In this field of business integration, whether synchronous or asynchronous, there are certain requirements for the response coordination mechanism between the calling and the called. For example, the processing of an order must be completed in milliseconds, otherwise the user experience will be very poor.

However, in the field of data integration, it is more likely to be near real-time or even offline non-real-time scenarios. That is to say, data is crawled through batch, real-time streaming or near-real-time streaming scenarios and then analyzed. The specific link is for users. It is not visible, which is also the difference between the focus of data integration and business integration.

The core demands of business integration on messaging systems

Message queue is one of the main patterns of enterprise business integration, which is an asynchronous communication pattern. Asynchronous mode provides low-coupling, high-reliability, and observable asynchronous communication capabilities. So what will be the effect of using messages in the business integration link? Here is a little list.

The above figure is a typical upper-layer application link, a single link from application A to lower-layer application B. By sending an initialization or structuring message, it is sent to the event channel as a call event. This channel is the message system. Such as RocketMQ, RabbitMQ, etc. After being stored in the time channel, the distribution component of the filtering route is matched to the downstream, and then pushed for processing. At the same time, there will be some systems for observability, operation and maintenance, and monitoring to support the reliable operation of this link.

There are many complete functional requirements. Here are four core demands of business integration for the messaging system:

1) Multi-type message transmission: support the integration demands of various business scenarios, mainly including ordinary messages, timing messages, transaction messages, sequential messages, etc.;

2) Rich routing distribution capabilities: support a variety of distribution routing conditions, including Tag filtering, message attribute filtering, one-to-many, one-to-one distribution, etc.;

3) Various interaction modes: support various interaction modes for sending and receiving messages, support synchronous and asynchronous sending, support active consumption, passive push consumption, support streaming response, single response;

4) Observable system: supports Metrics, Trace, Events analysis, supports single-link and full-link trajectory tracking, supports Metrics analysis and monitoring alarms, supports system operation events, and business event disclosure processing.

As a very typical business messaging solution, RocketMQ provides perfect messaging functions, rich client interfaces, and a perfect observable system and stability guarantee mechanism in response to the above-mentioned requirements for business integration.

Next, we will gradually dismantle RocketMQ's multi-type messages. This article mainly introduces ordinary messages.

Introduction to the principle of ordinary messages

Function introduction

Of the various message types, ordinary messages are the simplest and most important. Ordinary message is the basic message type of RocketMQ, which provides high-throughput, expansion, low-latency, and asynchronous communication capabilities. Other advanced message types basically superimpose unique control characteristics on the basis of this common message type, or a specific way of use.

The following picture is a typical topology of ordinary messages. Like the typical scenario of message queue, the producer sends messages, and sends ordinary messages to the server for storage. After storage, the messages are matched according to the subscription relationship, and finally pushed to the server. Downstream consumers do consumption.

Characteristics of ordinary messages

1) Atomicity: There is no association between messages, and the sending and receiving processing logic is atomic;

2) Scalability: ordinary message capacity and capability can be expanded, supporting multi-queue storage, horizontal splitting, and concurrent consumption;

3) Low latency: The common message link is short, the interaction is simple, the state is simple, the link is minimal, and the communication with low latency in milliseconds.

message life cycle

Ordinary messages will go through multiple states and processes from the initial transmission to the final processing. Understanding the life cycle of messages can help us determine how to quickly locate and solve online problems.

In short, the life cycle of a message can be abstracted into five states:

• Initialization: The status of ordinary messages that have been constructed and initialized by the producer and are to be sent to the server;
• To be consumed: The message is transmitted to the server, visible to the downstream, waiting for the consumer to obtain the processing status;
• Consuming: The message is acquired by the consumer and processed according to the business logic. At this time, the server will wait for the consumption to complete. If the event submitted by the consumer is not received after a certain period of time, the message will be retried for processing;
• Consumption submission: The consumer completes the message processing and submits the response event to the server, and the server marks that the current message has been processed (including consumption success and failure). RocketMQ supports all message retention by default. At this time, the message data will not be deleted immediately, but the logical mark is completed. Before the message is physically deleted, the consumer can still retrospectively reprocess the message;
• Message deletion: RocketMQ cleans up the oldest message data in a rolling manner according to the message storage time mechanism, and deletes the message from the physical file.

Common messaging application scenarios and cases

After a simple understanding of the principle and basic introduction, where are ordinary messages mainly used? Ordinary messages are the most widely used and the largest message type in RocketMQ. It mainly focuses on decoupling calls between services, as well as some scenarios such as batch data collection and transmission.

scenes to be used

1) Microservice call decoupling

• Asynchronous decoupling: Ordinary messages implement asynchronous calls to microservices, shortening business flow and response time.
• Traffic peak shaving and valley filling: Mass accumulation of ordinary messages can solve the stability risk of insufficient downstream processing capacity of traffic peaks.

2) Real-time data transmission

• High-throughput transmission: ordinary messages can achieve unlimited horizontal expansion, high data transmission throughput, and solve the problem of collection and reporting.
• Real-time transmission: ordinary messages are transmitted and delivered in real time, and downstream can consume in time for calculation and analysis.

Case introduction

1) Introduction to the scene

The trading platform is a system involved in the process of buyers and sellers completing the exchange of money and goods online according to the agreed contract. The interaction between the trading platform and multiple subsystems such as payment, logistics, ordering, and operation mostly uses RocketMQ ordinary messages for asynchronous decoupling. The reliable processing of messages is the core of e-commerce promotion guarantee.

2) Core pain points

The order state machine is complex, and the link time needs to be shortened: the order has a long life cycle, involves the flow of multiple downstream subsystems, takes a long time to synchronize calls, and has poor user experience.

Massive orders are processed in the big promotion scenario, and the downstream pressure is high: In the big promotion scenario, the order flow is large, and the processing capacity of each subsystem is insufficient, causing the system to crash.

Distributed scenario order change persistence and downstream call transactionality: Order status flow needs to ensure that database status changes and downstream calls succeed or fail at the same time, that is, transactionality.

Quick start sending and receiving messages

Having said so many scenarios and cases, let's take a look at how the code is used.

send normal message

The process of sending a message is very simple, but there are a few things to keep in mind:

• Message initialization should be as complete as possible: normal message initialization includes topic, tag tag, index key, and payload. It can be set according to the actual situation.
• Message sending needs to capture results and exceptions: After message sending, you need to get the response results. If it fails, you need to catch exceptions and retry.

Consume ordinary messages

RocketMQ supports a variety of consumption methods, including active acquisition methods and passive consumption listener push methods.

The passive consumption method only needs to register the consumption listener, and then the listener handles this logic internally, and finally returns the consumption result. If the consumption fails and you want RocketMQ to re-cast, it will return a failed result; throwing an exception also returns a failure. Similar to this result, returning to the server completes the entire consumption process.

The method of active acquisition will be more flexible. The business party can actively call to acquire messages, and can acquire messages at their own rate and concurrently. After the processing is completed, the RocketMQ server will reply to the consumption results.

Product Trailer: New Generation RocketMQ Version 5.0 Instances

As a final notice, the RocketMQ version of Alibaba Cloud's message queue will release a new version in August. The new version has the characteristics of greater flexibility, lower cost, and easier operation and maintenance. Please stay tuned.

Click here to enter the official website for more details~