Accumulation--This article takes you to understand Alibaba Cloud RocketMQ Lightweight Message Queue (MNS)

Author: Zhou Xinyu & Chen Tao & Li Kai

Alibaba Cloud RocketMQ Lightweight (MNS) message queue is a lightweight, reliable, scalable and fully managed distributed message queue service. MNS can help application developers to transmit data and notification messages more lightly on the distributed components of their applications, and build loosely coupled systems. There is no management overhead, and the model is simple, ready to use out of the box, automatic horizontal expansion, and does not require complex logical operation and maintenance such as managing partitions. It is a serverless message queue product.

MNS focuses on the core capability building of benchmark message queues. After years of iteration and polishing, MNS has been trying to keep it simple and easy to use on the client side despite being extremely complex internally. Focusing on the propositions of light weight and integration, it has been striving to build an easier-to-use system. message queue product.

core advantages

MNS provides a series of lightweight capabilities around lightweight message queues. Reduce the complexity of message products, reduce operation and maintenance overhead, and provide easier-to-use message queue services for enterprises and developers.

lightweight model

MNS provides two lightweight models, 1:1 and 1:N. You can choose the required model according to the scene. The learning cost is low and you can get started quickly. No need to understand complex conceptual models, no need to learn, and you can get started quickly in minutes.

The lightweight MNS model is mainly reflected in the operation of resources. The creation and deletion are configured with a minimalist API. At the same time, it belongs to the same SDK as the API of the data link, which is convenient for users to create resources like sending and receiving messages. On the contrary, many APIs of cloud product resources have adopted the new management and control SDK, and the threshold for use is relatively high.

In addition, MNS models and resources are in one-to-one correspondence. For example, using the queue model only requires creating a Queue, which is very easy to get started and use, while other products of the same type will have instances, topics, queues, partitions, etc. for the same model. multiple resources.

Lightweight Protocol

Using HTTP RESTful standard protocol, it is easy to access, has strong cross-network capability, and naturally has the ability to support multi-language access. The official SDK covers major mainstream languages, including C++, C#, .Net, Python, PHP and Java, and community contributions are also useful for Go, Node, etc.

At the same time, the access protocol is simple and universal. Even if you do not want to rely on cloud products to provide SDK, you can easily access the API through simple coding, with low access cost and high development efficiency.

Light billing

MNS is serverless lightweight billing. Compared with paying by instance, MNS provides a more lightweight serverless billing method. Pay according to the amount of requests and resource occupancy, and pay as much as you use, without paying for additional reserved resources; at the same time, it provides 20 million free API calls per month. This fully postpaid pricing method can save a lot of idle resources for enterprises.

Lightweight operation and maintenance

MNS can be elastically scaled according to the application, so there is no need to worry about capacity planning and provisioning. The number of messages in each queue is not limited, and the standard queue can provide almost unlimited throughput (within the QPS range); it provides serverless elastic services, which are efficient and fast, and do not need to reserve resources. It is more suitable for product integration and transient queue scenarios.

At the same time, MNS provides comprehensive message monitoring, message alerting and other capabilities, supports high reliability and high stability, and eliminates the complexity and R&D overhead of management and operation and maintenance.

basic model

MNS is a consumption system that combines the two consumption models of point-to-point (P2P) and publish-subscribe (Pub/Sub). The "queue" of MNS is the implementation of the P2P consumption model, and the "topic" is the implementation of the Pub/Sub consumption model.

MNS is also a rare cloud product in the industry that provides two hybrid consumption models at the same time. Users can use "queues" or "topics" to meet most of their needs for a messaging system.

Queue (P2P model)

The queue model provides a highly reliable and highly concurrent one-to-one consumption model, that is, each message in the queue can only be consumed by one consumer. The queue is like a revolving sushi restaurant. There are multiple sushi chefs (producers) in a sushi restaurant making exquisite sushi. Each piece of sushi is unique, and customers (consumers) can take their favorite sushi from the conveyor belt for consumption (consumption).

Queue is the top-level resource of MNS and the storage entity of messages. It has a complete life cycle of Alibaba Cloud resources and a complete permission control mechanism. There are three core concepts surrounding queues:

• Producer: The producer is responsible for sending messages to the specified queue, and a queue accepts distributed producers to deliver messages concurrently.
• Queue: The physical resource corresponding to the queue model is also the storage entity of the message. MNS provides the distributed storage capability of the queue through a distributed storage cluster.
• Consumer: The consumer is responsible for obtaining messages from the specified queue through the polling mechanism. A queue accepts distributed consumers to consume messages concurrently, while ensuring that the same message will only be consumed by one consumer within the visible time. .

Topics (Pub/Sub model)

The topic model provides a one-to-many publish-subscribe model and supports message notification. A topic is like a newspaper to which multiple readers go to the post office to subscribe. When the paper launches its latest issue, readers (including Post Office partners) have the following options to get it:

• Have the post office deliver (push) all the newspapers to the home (specific address).
• Go to the nearest newsstand (subscription point) to get the newspaper by yourself (the newspaper will be delivered to each newsstand by the post office courier first).

Topic is another top-level resource of MNS. It also has the ability to store messages. The corresponding Topic resource has a complete life cycle of Alibaba Cloud resources and a complete permission control mechanism. There are also three core concepts surrounding the theme:

• Producer: In addition to sending messages to queues, producers can also send messages to topics, which also accept distributed producers to deliver messages concurrently.
• Topic: The physical resources corresponding to the topic model also have the ability to store messages.
• Subscription: Multiple subscriptions can be created for each topic. Each subscription will receive the complete message on the topic. The downstream of the subscription can be multiple types of subscribers. Users often use the ability to subscribe to the message in the topic. Fan out to multiple queues.

Features

As a mature message product, MNS provides a variety of functional features for topics and queues, which can penetrate into the application architecture, help users reduce the complexity of business process development, and allow businesses to focus on business logic development. For example, the support of multiple message types is convenient for users to quickly implement business logic related to timing and priority.

Support for multiple message types

The MNS queue supports three message types: normal message, delayed message, and priority message.

Ordinary message is the most basic message type. When an ordinary message is sent to MNS, the message will wait in the queue for the user to pull and consume the message.

For delayed messages, users can specify the expected delay time of the message when sending, and when the set delay time is reached, the message can be consumed. At the same time, after the delayed message is sent, in addition to some basic properties of the message, a message handle can also be obtained. When we want to cancel the delayed message in advance, we can use this message handle to delete. After Delete, the message will not be delivered again. By delaying messages, users can easily implement a distributed delay scheduling system that achieves ultra-high scalability and availability while having high-precision delays.

For priority messages, we can specify the priority of the message when sending the message, and the priority value ranges from 1 to 16. The smaller the priority value, the higher the priority of this message. It should be noted that priority messages do not guarantee the order of messages. Messages with higher priority are more likely to be consumed, which does not mean that messages with lower priority will be consumed after the consumption of messages with higher priority is completed. to be consumed.

Efficient long rotation training mechanism

For the message queue MNS, users need to actively pull messages. If the pull frequency is too low, it will cause message delay; correspondingly, if the pull frequency is too high, it will cause excessive Multiple invalid API calls.

To solve this problem, the MNS queue provides a long-round training mechanism, which can reduce invalid API calls to a certain extent, while ensuring the timeliness of messages. When a user pulls a message, the waitseconds for this request to pull the message can be set, and the client will hang a long polling request to the server at the set time. If there is a message that can be consumed during the long rotation training, it will return immediately; if not, it will return after the long rotation period at most. However, it should be noted that although we can use long round training to reduce invalid API calls, a certain degree of concurrency of message pulling is still required as a whole to ensure the timeliness of messages.

Flexible message lifecycle management

For a message in the queue, it will have three states: visible, invisible, and deleted. When a message is sent to the queue, the message is in a visible state, waiting for consumers to pull it. After the message is pulled out by the consumer, the message will enter a period of "invisible time". During this "invisible time", the message will not be pulled out again. But if the message is not deleted during the "invisibility time", then the message will be visible again and can be pulled from the queue again by the consumer.

In addition, when a consumer consumes a message and suddenly there is a problem during processing, such as downtime, then this message will be visible again after the "invisible time" ends and can be consumed by other consumers consumer again.

There may be a contradiction here. Users may expect that after a consumer goes down, messages that have not been processed correctly can be consumed by other consumers as soon as possible, but because their own business processing time may sometimes be long, it may Exceeding the default "invisibility time" configured on the queue results in repeated consumption of messages. Then, in order to solve this scenario, MNS also provides users with the "invisible time" function that can modify a message in the message dimension.

When the user receives a message and finds that the processing time of the message is longer than the "invisible time" configured in the queue, the user can actively call through the message handle to modify the invisible time of the message. For example, in the above figure, the invisible time of this message is extended, leaving more processing time for the business. Of course, like the previous message consumption, when the message is processed, the interface for deleting the message also needs to be called to confirm the successful consumption of the message.

News multi-subscription push

MNS topics provide the ability to broadcast messages one-to-many. We can create multiple subscriptions for a topic and route to different Endpoints. The user only needs to send the message to the topic, and this one message can be pushed to multiple subscriptions.

Second, on the basis of the ability to broadcast messages one-to-many, in subscription, MNS also supports subscribing to messages with specific tags. When creating a subscription, we can specify the tag of the message that the subscription cares about, so that when MNS pushes the message, it will automatically filter the message, and only push the tag that the subscription cares about.

To ensure the reliability of message delivery, users can configure a push policy for each subscription, and can specify a backoff retry policy or an exponential decay retry policy.

For the backoff retry strategy, when the message push fails, it will retry 3 times, and the interval between each retry is a random value of 10 to 20 seconds. For exponentially decaying retries, the overall retry time is 1 day, and each retry interval is exponentially incremented (1 sec, 2 sec, 4 sec, 8 sec, ...).

Finally, we can also configure the format of the pushed message. We can specify the push format as XML, JSON or SIMPLIFIED when creating a subscription. For XML and JSON, the content of the message pushed to the subscription will not only include the message sent to the topic, but also add attributes such as MessageID, TopicName, SubscriptionName, etc. For SIMPLIFIED, the message sent to the topic is forwarded directly without adding additional attributes.

access protocol

MNS provides external services through HTTP protocol, including nearly 30 RESTful APIs, but core business scenarios often use about 3 APIs to complete research and development, which is very easy to get started.

In addition, users can either directly encapsulate it according to the API, or directly use the officially provided SDK for integration. Users only need to rely on one SDK to complete services from opening, creating resources, and sending and receiving messages.

In terms of multi-language SDKs, MNS provides SDKs in 8 mainstream languages, including Java, Python, C#, Node, PHP, C++, Go, etc. It also supports the JMS SDK that uses the Java field standard to access MNS.

Analysis of applicable scenarios

Compared with open source message queues such as RabbitMQ, ActiveMQ, and Kafka, RocketMQ Lightweight MNS abandons very complex conceptual models and bloated protocols. Through simple models, common HTTP RESTful standard protocols can efficiently complete typical scenarios in the message field. . Provides a message integration solution that is completely free of operation and maintenance, significantly reduces development and maintenance costs, and improves the efficiency of new business development.

Message Broadcasting (Fanout) Scenario Analysis

• Background Information

The lightweight message queue MNS provides two models of queue (Queue) and topic (Topic), which can basically meet most application scenarios. The queue provides a one-to-one shared message consumption model and adopts the client's active pull (Pull) mode.

The topic model provides a one-to-many broadcast message consumption model, and adopts the server-side active push (Push) mode.

The advantage of the push mode is that the instant performance is better, but the client address needs to be exposed to receive the message push from the server. In some cases, some information, such as the corporate intranet, cannot expose the push address, and it is hoped to use the pull method instead. Although the message service MNS does not directly provide this consumption model, it can combine topics and queues to implement a one-to-many pull message consumption model.

• solution

By creating a subscription, the topic pushes messages to the queue, and consumers pull messages from the queue. This can not only achieve one-to-many broadcast messages, but also avoid exposing the consumer's address.

Analysis of Large Message Transmission (Claim Check) Scenario

• Background Information

The maximum message size limit of the message service MNS queue is 64 KB, which can basically meet the requirements of the message as a control flow information exchange channel under normal circumstances. However, in some special scenarios, when the message data is relatively large, only the method of message slicing can be used.

The following is a solution for delivering messages larger than 64 KB through OSS without message slicing.

• solution

Before the producer sends a message to the message service MNS, if it finds that the message body is larger than 64 KB, it will upload the message body data to OSS first.

The producer sends the Object information corresponding to the data to the message service MNS.

The consumer reads the message from the MNS queue of the message service and determines whether the message content is the object information of the OSS.

If it is judged that the message content is the object information of the OSS, the corresponding object content is downloaded from the OSS and returned to the upper-layer program as the message body.

Simple transaction message scenario analysis

Background information: In some business scenarios, transaction consistency between local operations and message sending needs to be ensured, that is, if the message is sent successfully, the local operation is successful. If the message is sent successfully and the local operation fails, then the successfully sent message needs to be rolled back. The operation process is shown in the figure:

• solution

The message is sent successfully, and the operation steps are as follows when the transaction operation is successful:

• The producer sends a transaction preparation message to the transaction message queue;
• The producer sends the operation log message to the operation log queue, and the log contains the message handle of the step 1 message;
• The producer executes the local transaction operation successfully;
• The producer requests to modify the message delay time to make the message visible to consumers;
• The producer confirms the operation log to the operation log queue and deletes the log message;
• The consumer receives transaction messages from the transaction message queue;
• Consumers process transactional messages;
• Consumer requests to delete transaction messages.

Message Filter Scenario Analysis

• Background Information

In some scenarios, messages need to be pushed to different push targets according to the message content. To achieve this function, you can create multiple topics and set corresponding push targets for each topic, but this will increase additional costs and increase complexity of operation and maintenance. In order to avoid this situation, the message queue MNS provides the function of message filtering label. You can create only one topic and set different message filtering labels when creating a subscription. Combined with the message filtering labels of messages, MNS can push messages to different push targets.

• solution

In the example scenario shown in the figure, 3 subscriptions with different message filtering tags are created on topic Topic 1, Subscription 1, Subscription 2 and Subscription 3. The push targets of these three subscriptions are Queue 1, Queue 2, and Queue 3, respectively.

• The message filter label of the message is the same as the message filter label of the subscription. The message filtering process is as follows:

The message service MNS pushes Message 1 to the queue Queue 1;

The message service MNS pushes Message 2 to the queue Queue 2.

• Subscriptions do not have message filtering tags. The message filtering process is as follows:

The message service MNS pushes Message 1 to the queue Queue 3;

The message service MNS pushes Message 2 to Queue 3;

The message service MNS pushes Message 3 to the queue Queue 3.

Customer Stories and Best Practices

Online Transaction - Application Decoupling - Peak Shaving and Valley Filling (Typical Scenario)

• scene description

The core system of Taobao/Tmall main site is the transaction system. Each transaction order data will be associated with hundreds of downstream business systems, including logistics, shopping cart, points, direct charging, Alimama, stream computing analysis, etc. The whole system is huge and complex, and the architecture design is slightly unreasonable, which will directly affect the continuity of the main station business.

• Asynchronous decoupling

In such a complex and huge system, if the upstream and downstream business systems are tightly coupled, the unavailability of any subsystem will directly lead to the unavailability of the core trading system;

The upstream and downstream business systems are loosely coupled through message queues, so even if the downstream subsystem (such as the logistics system) becomes unavailable, it will not affect the normal operation of the core transaction system;

• shaving peaks and filling valleys

The message queue can not only play the role of decoupling between systems, but also play a role in reducing peaks and filling valleys in large-scale activities such as big promotions and spikes.

Every year in the early morning of Double 11 on Tmall, ensuring the business processing capacity of the core trading system is always the top priority. The processing capacity of hundreds of thousands of transaction orders per second is increasing linearly year by year; The business processing capacity of payment systems is limited, and failure to perform traffic buffering will directly lead to the collapse of these systems. Therefore, using the powerful message accumulation capability of the message queue can play a good role in reducing peaks and filling valleys, transferring all the system pressure to the message queue, thereby releasing the pressure of logistics, payment and other systems to ensure the normal operation of the business.

Live Recording - Convergence - Transcoding (Serverless Scenario)

Scenario description: Live video recording, confluence, and transcoding are just-needed scenarios in the field of online education and online live broadcast. Through MNS, FC Serverless resources can be scheduled online to help customers achieve complex scenarios such as asynchronous live stream recording, confluence transcoding, etc.

online game scene

• scene description

There are a large number of data flows in strong interaction scenarios in the game scene, which requires extremely high stability and end-to-end delay of the message middleware, and the self-built message middleware is under high pressure and instability. There are a large number of command transmissions and timing tasks in the game scene, and self-built logic cannot solve hot issues.

Through MNS, the conversion from game gateway instructions to background message protocols can be realized. At the same time, MNS also supports asynchronous decoupling and splitting between background logic servers, trade unions, and battle servers.

Summarize

MNS provides a series of lightweight capabilities around lightweight message queues. Reduce the complexity of message products, reduce operation and maintenance overhead, and provide easier-to-use message queue services for enterprises and developers.

As a lightweight version of Alibaba Cloud RocketMQ, it fills the shortcomings of RocketMQ in lightweight and ease of use. Although the MNS version has been iterated and polished with RocketMQ for many years, and the interior is extremely complex, it has been trying to keep it simple and easy to use on the client side. It is the product mission and vision of MNS to reduce the complexity of message products and reduce operation and maintenance costs for enterprises and developers to provide easier-to-use message queue services.

---

MNS product training camp activities are hot registration

In order to help you have a more comprehensive understanding of Alibaba Cloud Message Queue MNS from the shallower to the deeper, and at the same time hope that Message Queue MNS can help you solve daily work and production problems, the Message Queue MNS product training camp course is specially launched. A simple and visual introduction of the product, as well as a "first show" hands-on practical learning course.

By participating in this message queue MNS training camp, you can learn and gain:

• Basic concepts and features of message queue MNS
• Best Practices and Cases of Message Queuing MNS
• Based on MNS, 0 basis to easily build Web Client

In addition to the learning-level gains, during the event, the top 20 students who have completed all the camping tasks and passed the exam can get it (if the scores are the same, they will be ranked in the order of the exam time), and they can get the Xiaomi power bank for free. Event time: August 10th - August 31st (during working days).

Click here to register now~