"RabbitMQ" | Solve the problem of message delay and accumulation

Hello everyone, I am a side dish.
A man who hopes to be about architecture ! If you also want to be the person I want to be, otherwise click on your attention and be a companion, so that Xiaocai is no longer alone!

This article mainly introduces the common problems of

If necessary, you can refer to

If it helps, don’t forget to 16187d4eb4fe13 ❥

WeChat public account has been opened, , students who have not followed please remember to pay attention!

• message reliability problem : How to ensure that the sent message is consumed at least once?
• Delayed Message Question : How to achieve delayed delivery of messages?
• message accumulation problem : How to solve the problem of message accumulation of millions of levels and unable to consume in time?

In the previous article, we have explained how to solve the problem of message loss, that is, to ensure the reliability of the message, then the other two problems are equally important. In this article, we will describe the solutions to the other two problems~!

Message loss solution: "RabbitMQ" | Message loss is the same thing

One, delayed message

Delayed message literally means to delay the reception of the message, so how to make the message arrive later? This is the problem we have to think about and solve. Before we understand the delay queue, we need to understand the two concepts in RabbitMQ

• Dead letter switch
• TTL

1) Dead letter switch

dead letter , that is, discarding a dead message, under what circumstances can an ordinary message become dead letter ? Need to meet the following three conditions:

1. Consumers use basic.reject or basic.nack declare consumption failure, and set the message's requeue parameter to false
2. The message is an expired message, no one consumes after the timeout
3. The queue messages to be delivered are full, and the oldest messages will become dead letters

The dead letter switch is the attribution of the dead letter.

If a queue is configured dead-letter-exchange attribute, specifies a switch, then the dead letter queue will be delivered to the switch, and this switch is called badmail switch - DLX ( Dead Letter the Exchange )

Step : When the producer normally delivers to the queue (simple.queue), if the consumer consumes a message from the queue (simple.queue) but declares reject , then the queue is bound to the dead letter switch (dl.queue) ), then the message that becomes a dead letter at this time will be delivered to this dead letter queue (dl.queue).

From the normal queue --> dead letter queue process, we must declare two key pieces of information

• The name of the dead letter switch
• The routing key bound between the dead letter switch and the dead letter queue

These two pieces of information are also the basic configuration of our message delivery.

Next, we simply simulate the scenario generated by condition 1

1. First declare a dead letter exchange and dead letter queue

Our side is directly generated using a simple annotation method

Through the RabbitMQ console interface, you can see that it has been successfully generated

2. Declare the normal use of switches and queues

Then at this time we can create a normal exchange and queue, and specify the dead letter exchange

You can also view the creation status through the console

Whether there is a dead letter switch statement, we can DLX and DLK flags in the queue

3. Simulate rejection

Then we now simulate the scenario where the client rejects the message through the code

1) Message sending

2) Message reception

Check the console, the results are as follows:

2021-11-06 23:56:52.095  INFO 2112 --- [ntContainer#0-1] c.l.m.c.listener.SpringRabbitListener    : 正常业务交换机 | 接收到的消息 : [hello]
2021-11-06 23:56:52.118  INFO 2112 --- [ntContainer#1-1] c.l.m.c.listener.SpringRabbitListener    : 死信交换机 | 接收到的消息 : hello

This shows that our dead letter exchange has successfully played a role

2）TTL

Above, we have successfully recognized the use of dead letter switch, but this does not seem to have much to do with the 16187d4eb505c2 delay queue TTL(Time-To-Live) is used to process delayed messages~!

In the concept of TTL, if a message in a queue is not consumed after the TTL ends, the message will automatically become a dead letter, and there are two types of TTL timeout situations:

1. The queue where the message is located has a time to live
2. The message itself sets the time to live

We also perform the above simulation scenario of condition 2

1. Declare dead letter exchange and dead letter queue (the above has been completed)

2. Declare the delay queue and specify the dead letter exchange

The same console view the creation results, and we found that there are not only DLX and DLK signs, but also an additional TTL , indicating that the queue is a delayed queue

3. Simulate consumption overtime

We send a message to the delay queue, and there is no consumer to consume, wait for 1 minute to see if we can enter the dead letter queue

We have sent a message to the delay queue and successfully found in the console one minute later that this message has entered the dead letter exchange

2021-11-07 00:01:30.854  INFO 32752 --- [ntContainer#1-1] c.l.m.c.listener.SpringRabbitListener    : 死信交换机 | 接收到的消息 : test ttl-message

The above is to configure the queue timeout time, the message itself can also be configured with timeout time, when message and queue have timeout time, then the shortest TTL shall prevail, and the message timeout configuration is as follows:

As shown in the figure above, we can use Message to transfer message information and set the timeout time. We set 5000 ms. After waiting for the transmission to be successful, the console also successfully printed the message consumed by the dead letter switch after 5000 ms:

2021-11-07 00:03:09.048  INFO 39996 --- [ntContainer#1-1] c.l.m.c.listener.SpringRabbitListener    : 死信交换机 | 接收到的消息 : this is a ttl message

3) Delay queue

We used the dead letter switch to indirectly realize delay queue , but it doesn’t have to be so troublesome in RabbitMQ. RabbitMQ has already packaged the plug-in for us, we only need to download and install it~

RabbitMQ plug-in download address

When we enter the address, we can find that there are many plug-ins, search for delay keywords to find the plug-ins we need to download

Once downloaded directly uploaded to RabbitMQ plugins directory - plugins , side dishes here is the use docker temporary installation test, so the plug-in directory has been mounted out:

docker run -itd --name rabbitmq -v plugins:/plugins -p 15672:15672 -p 5672:5672 rabbitmq:management

Therefore, I can upload the plug-in directly to the plugins directory in the container~

Then enter the container and execute the following command to open the plug-in

rabbitmq-plugins enable rabbitmq_delayed_message_exchange

And when we create the switch in the console, we can see that type are more options for the 06187d4eb509a7 type

Successful execution to this step means that RabbitMQ's delay queue function has been turned on

Then we can use DelayExchange . First, we need to understand the code to create a delay switch:

Way 1

Way 2

When we have everything we need, we can send a message

When sending a message, the x-delay parameter must be carried in the message header to specify the delay time

After this configuration, we can see in the console that after 10 seconds, delay.queue received the corresponding message, and then was consumed by the corresponding consumer

3) Summary

From the dead letter switch to the TTL to delay queue, we learned step by step how to implement the function of delaying messages, and then we made a small summary:

Question 1: What kind of news will become a dead letter?

1. The message was rejected by the consumer or returned to nack
2. Message timed out and not consumed in time
3. Message queue is full

Question 2: The way the message timed out

1. Set the TTL attribute to the queue
2. Set the TTL attribute to the message

Question 3: How to use delay queue

1. Download and enable RabbitMQ delay queue plugin
2. Declare a switch and set the delayed attribute to true
3. When sending a message, add the x-delay header, the value is the timeout period

Question 4: Use scenarios of delayed queues

1. Delay in sending SMS notifications
2. The order is automatically cancelled
3. Automatic inventory rollback

Second, the lazy queue

about the delay queue, let's continue to understand the 16187d4eb50d74 lazy queue

Before we talk about the lazy queue, let’s first raise a question~

How RabbitMQ solves the problem of message accumulation

Under what circumstances will the message accumulation problem occur?

1. When the producer's production speed is far from the consumer's consumption speed
2. When consumers fail to restart in time

So how to solve this problem? The usual ideas are as follows:

1. After the consumer's machine restarts, add more consumers for processing
2. Develop a thread pool inside the consumer processing logic, and use multi-threading to improve processing speed
3. Expand the capacity of the queue and increase the stacking limit

These methods are theoretically no problem to solve the problem of message accumulation, but the processing methods are not elegant or even flexible~ Then in addition to the above solutions, we can use a queue type that comes with RabbitMQ - Lazy queue

What is a lazy queue? We recognize several characteristics of lazy queues:

• After receiving the message, save it directly to disk instead of memory
• The consumer will read from the disk and load it into the memory when it wants to consume the message
• It supports the storage of millions of messages

In the final analysis, it is to use the disk buffer mechanism, and the disadvantage of this mechanism is that message will be reduced, and the performance is limited by the disk IO. After understanding the characteristics and shortcomings, we will take a look at how to create a lazy queue

Way 1

Way 2

Way 3

This method is to modify a running queue to a lazy queue directly based on the command line modification

rabbitmqctl set_policy Lazy "^lazy-queue$" '{"queue-mode":"lazy"}' --apply-to queues  

The meanings of several command parameters are as follows:

• rabbitmqctl : command line tool
• set_policy : add a policy
• Lazy : Strategy name, can be customized
• ^lazy-queue$ : Use regular expressions to match the name of the queue
• '{"queue-mode":"lazy"}' : Set the queue to lazy mode
• --apply-to queues : the target of the strategy, all queues

Although the disadvantage of this lazy queue method is that the timeliness of the message will be reduced, it is not unacceptable in some scenarios, not to mention its advantages are also obvious:

• Based on disk storage, the upper limit of messages is high
• No intermittent page-out, stable performance

---

Up to this point, we have talked about the common problems of RabbitMQ. For us, ordinary development scenarios may encounter these problems, but no encounter does not mean there is no, so we still need to know more to prevent problems!

Don't talk empty talk, don't be lazy, and be a X as an architecture with Xiaocai~ Follow me to be a companion, so Xiaocai is no longer alone. See you below!

If you work harder today, you will be able to say less begging words tomorrow!
I am Xiaocai, a man who becomes stronger with you. 💋
WeChat public account has been opened, , students who have not followed please remember to pay attention!