Parse RocketMQ business messages -- "sequential messages"

Author: Shao Shu

introduction

Since its inception, Apache RocketMQ has served the internal business of Alibaba Group and tens of thousands of enterprise customers of Alibaba Cloud after more than ten years of large-scale business stability. As a financial-grade reliable business messaging solution, RocketMQ has been focusing on the construction of asynchronous communication capabilities in the field of business integration since its inception. This article will continue the scenario of business message integration, and introduce the sequential message function of RocketMQ from the perspective of functional principles, application cases, best practices, and actual combat.

Introduction

A sequential message is a message provided by the RocketMQ version of the message queue that has strict requirements on the order of message sending and consumption. For a specified topic, the messages of the same MessageGroup are published and consumed according to the strict first-in-first-out (FIFO) principle, that is, the messages published first are consumed first, the messages published later are consumed, and the server stores and consumes in strict accordance with the sending order. . The order of messages of the same MessageGroup is guaranteed, and the order of messages between different MessageGroups is not required. Therefore, two points need to be achieved, the order of sending and the order of consumption.

Functional principle

Here's a question first. In daily contact, many RocketMQ users will think that since sequential messages can achieve order on the basis of ordinary messages, it seems to be an enhanced version of ordinary messages, so why not use order for all What about news? Next, we will focus on this problem and compare ordinary messages and sequential messages.

Sequentially sent

In a distributed environment, it is very difficult to ensure the global order of messages. For example, two RocketMQ Producer A and Producer B send message a and message b to the RocketMQ server without communication. Due to the distributed system Due to the limitation, we cannot guarantee the order of a and b. Therefore, the industry message system usually guarantees the order of partitions, that is, the order of messages with the same attribute, which we call MessageGroup. As shown in the figure, ProducerA sends two messages A1, A2 with MessageGroup property A and B1, B2 with MessageGroup property B, while ProducerB sends two messages C1, C2 with MessageGroup property C.

At the same time, for the same MessageGroup, in order to ensure the sequence of its sending, it is relatively simple to construct a single-threaded scenario, that is, different MessageGroups are responsible for different Producers, and for each Producer, sequential messages are sent synchronously of. The benefits of synchronous sending are obvious. After the client receives the sending result of the previous message and then sends the next message, the sending order can be accurately guaranteed. If asynchronous sending or multi-threading is used, it is difficult to guarantee this.

Therefore, it can be seen that although in the underlying principle, sequential message sending is no different from ordinary message sending, in order to ensure the ordering of sequential message sending, the synchronous sending method actually reduces the maximum throughput of messages compared with ordinary messages. .

sequential consumption

Different from sequential messages, there are actually no restrictions on the consumption of ordinary messages. The messages pulled by consumers are consumed asynchronously and concurrently, while sequential messages need to ensure that for the same MessageGroup, only one client is consuming at the same time. The consumer cannot consume the next message of the same MessageGroup before the message is confirmed to be consumed (or enters the dead letter queue), otherwise the order of consumption will not be guaranteed. Therefore, there is a consumption bottleneck, which depends on the user's own business processing logic. In extreme cases, when there are too many messages in a MessageGroup, it may lead to accumulation of consumption. Of course, it also needs to be clear that the context here refers to the same MessageGroup, and there is no sequential association between messages of different MessageGroups, which can be consumed concurrently. Therefore, the order mentioned in the whole text is actually a partial order.

summary

Regardless of sending or consuming, we group messages by means of MessageGroup, that is, the basic unit of concurrency is MessageGroup, and different MessageGroups can be sent and consumed concurrently, thus achieving scalability to a certain extent, supporting multi-queue storage and horizontal splitting , concurrent consumption, and is not affected. Looking back at ordinary messages, from the perspective of sequential messages, it can be considered that the basic unit of concurrency of ordinary messages is a single message, that is, each message has a different MessageGroup.

Let's go back to the question at the beginning:

Since sequential messages can implement order on the basis of ordinary messages, it seems to be an enhanced version of ordinary messages, so why not use sequential messages for all?

Now everyone may have a basic impression of this problem. Of course, the order of messages is very good, but there is a price to achieve order.

The following is a table that briefly compares sequential and normal messages.

Best Practices

Reasonable setting of MessageGroup

MessageGroup will have many wrong choices. Take an e-commerce platform as an example. An e-commerce platform uses the merchant ID as a MessageGroup, because some large-scale merchants will produce more orders. Due to the limitation of downstream consumption capacity, this part The orders corresponding to the merchants experienced a serious accumulation. The correct approach should be to use the order number as a MessageGroup, and in terms of the business logic behind it, the same order has sequential requirements. That is to say, the best practice for selecting MessageGroup is: the life cycle of MessageGroup is preferably short, and the number of different MessageGroups should be as equal and uniform as possible.

Synchronous send and send retries

As described in the previous section, synchronous transmission and transmission retries are used to ensure the ordering of transmissions.

consumption idempotent

The message transmission link will have a small amount of repetition in abnormal scenarios, and business consumption needs to be idempotent to avoid the risk of repeated processing.

Applications

• User registration needs to send a verification code, and the user ID is used as the MessageGroup, then the messages sent by the same user will be consumed in the order of publication.
• When creating an e-commerce order, the order ID is used as the MessageGroup, then the order creation message, order payment message, order refund message, and order logistics message related to the same order will be consumed in the order in which they are released.

actual combat

send

As you can see, the sending case sets the MessageGroup and uses synchronous sending. The sending code is as follows:

 public class ProducerFifoMessageExample {
    private static final Logger LOGGER = LoggerFactory.getLogger(ProducerFifoMessageExample.class);

    private ProducerFifoMessageExample() {
    }

    public static void main(String[] args) throws ClientException, IOException {
        final ClientServiceProvider provider = ClientServiceProvider.loadService();

        // Credential provider is optional for client configuration.
        String accessKey = "yourAccessKey";
        String secretKey = "yourSecretKey";
        SessionCredentialsProvider sessionCredentialsProvider =
            new StaticSessionCredentialsProvider(accessKey, secretKey);

        String endpoints = "foobar.com:8080";
        ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
            .setEndpoints(endpoints)
            .setCredentialProvider(sessionCredentialsProvider)
            .build();
        String topic = "yourFifoTopic";
        final Producer producer = provider.newProducerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the topic name(s), which is optional. It makes producer could prefetch the topic route before 
            // message publishing.
            .setTopics(topic)
            // May throw {@link ClientException} if the producer is not initialized.
            .build();
        // Define your message body.
        byte[] body = "This is a FIFO message for Apache RocketMQ".getBytes(StandardCharsets.UTF_8);
        String tag = "yourMessageTagA";
        final Message message = provider.newMessageBuilder()
            // Set topic for the current message.
            .setTopic(topic)
            // Message secondary classifier of message besides topic.
            .setTag(tag)
            // Key(s) of the message, another way to mark message besides message id.
            .setKeys("yourMessageKey-1ff69ada8e0e")
            // Message group decides the message delivery order.
            .setMessageGroup("youMessageGroup0")
            .setBody(body)
            .build();
        try {
            final SendReceipt sendReceipt = producer.send(message);
            LOGGER.info("Send message successfully, messageId={}", sendReceipt.getMessageId());
        } catch (Throwable t) {
            LOGGER.error("Failed to send message", t);
        }
        // Close the producer when you don't need it anymore.
        producer.close();
    }
}

Consumption

The consumption code is as follows:

 public class SimpleConsumerExample {
    private static final Logger LOGGER = LoggerFactory.getLogger(SimpleConsumerExample.class);

    private SimpleConsumerExample() {
    }

    public static void main(String[] args) throws ClientException, IOException {
        final ClientServiceProvider provider = ClientServiceProvider.loadService();

        // Credential provider is optional for client configuration.
        String accessKey = "yourAccessKey";
        String secretKey = "yourSecretKey";
        SessionCredentialsProvider sessionCredentialsProvider =
            new StaticSessionCredentialsProvider(accessKey, secretKey);

        String endpoints = "foobar.com:8080";
        ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
            .setEndpoints(endpoints)
            .setCredentialProvider(sessionCredentialsProvider)
            .build();
        String consumerGroup = "yourConsumerGroup";
        Duration awaitDuration = Duration.ofSeconds(30);
        String tag = "yourMessageTagA";
        String topic = "yourTopic";
        FilterExpression filterExpression = new FilterExpression(tag, FilterExpressionType.TAG);
        SimpleConsumer consumer = provider.newSimpleConsumerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the consumer group name.
            .setConsumerGroup(consumerGroup)
            // set await duration for long-polling.
            .setAwaitDuration(awaitDuration)
            // Set the subscription for the consumer.
            .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
            .build();
        // Max message num for each long polling.
        int maxMessageNum = 16;
        // Set message invisible duration after it is received.
        Duration invisibleDuration = Duration.ofSeconds(5);
        final List<MessageView> messages = consumer.receive(maxMessageNum, invisibleDuration);
        for (MessageView message : messages) {
            try {
                consumer.ack(message);
            } catch (Throwable t) {
                LOGGER.error("Failed to acknowledge message, messageId={}", message.getMessageId(), t);
            }
        }
        // Close the simple consumer when you don't need it anymore.
        consumer.close();
    }
}

Today, through the introduction of RocketMQ sequential messages, I hope to help you have a deeper understanding of the principles and applications of sequential messages, and I also hope that RocketMQ's sequential messages can help you solve business problems more effectively. If you are interested in RocktMQ business news, you are also welcome to scan the QR code below to join the DingTalk group to communicate~

Click here to enter the official website for more details~