"RabbitMQ" | That's all about message loss

Hello everyone, I am a side dish.
A man who hopes to be about architecture ! If you also want to be the person I want to be, otherwise click on your attention and be a companion, so that Xiaocai will no longer be alone!

This article mainly introduces the message loss problem of

If necessary, you can refer to

If it helps, don’t forget to 16175f393541e6 ❥

The WeChat public account has been opened, , students who have not followed please remember to pay attention!

Yes, it finally started RabbitMQ

RabbitMQ interview questions that are common in interviews are also more common. The common ones are as follows:

• Message reliability problem : How to ensure that the sent message is consumed at least once?
• Delayed message problem : How to achieve delayed delivery of messages?
• High Availability Problem : How to avoid unavailability problems caused by single-point MQ failures?
• Message accumulation problem : How to solve the problem of message accumulation of millions of levels and unable to consume in time?

These few questions have to make your head hurt for a while. Have you read a lot of blog posts on the Internet to introduce solutions in this area, but you have read it and forgot. Actually, it is because of the lack of practical operation. This side dish is the focus Tell me how RabbitMQ solves the problem of message loss~

1. Message reliability issues

The message reliability problem may be interpreted as how to prevent message loss? So why is the message lost? We can first look at the entire process of message delivery:

From the figure, we can analyze the possible message loss from three stages:

• publisher send message to exchange
• exchange distributed to queue
• queue delivered to customer

Now that we know which stages may cause data loss, we can prevent it from the source~!

Engineering structure

The engineering structure is very simple, it is a simple Spring Boot project, which has two modules: consumer and producer

1. Lost sent by the producer

RabbitMQ provides publisher confirm mechanism to avoid the problem of message loss in the process of sending to MQ. After the message is sent to MQ, a confirmation result will be returned to the producer to indicate whether the message is confirmed successfully. There are two types of requests for this confirmation result:

• publisher-confirm

The type is sender confirmation , there are two cases

1. The message is successfully delivered to the switch, and the return is ack
2. The message is not delivered to the switch, and the return is nack

• publisher-return

The type is sender receipt , there are two situations

1. The message is delivered to the exchange, and is successfully distributed to the queue, returning ack
2. The message is delivered to the exchange, but it is not successfully delivered to the queue, and the return is nack

Note: When the confirmation mechanism sends messages, it is necessary to set a globally unique ID for each message to distinguish different messages and avoid ack conflicts

Next, we use code to illustrate the specific operation method

1) configuration file

Let's first look at the configuration file of the producer

The connection information of the first few configurations RabbitMQ is nothing to talk about, let’s look at some unfamiliar configurations

• publisher-confirm-type

Turn on sending confirmation, here can support two types

1. simple : synchronously wait for the confirm result until the timeout
2. correlated : asynchronous callback, define ConfirmCallback, MQ will call back this ConfirmCallback when the result is returned

• publisher-returns

Open public-return , which is also based on the CallBack mechanism, but defines ReturnCallback

• template.mandatory

Define the strategy when routing fails.

• true : call ReturnCallback
• false : directly discard the message

2) defines the callback event

Each RabbitTemplate can only be configured with one ReturnCallback

3) send message

Before executing the sending code, we make sure that we have created (a direct switch direct-exchange , a queue direct-queue , and the bound key is direct

Under normal circumstances, we must execute the code to send successfully, you can see the console green output

And we also successfully received the message in the message queue:

There is no problem up to this step, then we need to manually create some problems for it~ We can modify the switch name. At this time, when the switch is not found when sending a message, the switch will definitely return nack , and then see if it works. Enter the judgment in our code:

Although the code execution is green, but because rabbitMQ can't find the correct switch, the message sending fails, which is the process in the following figure:

This one is the publish -> exchange we successfully captured it, so exchange -> queue is whether we can capture it normally? We can modify the routing key so that the switch cannot route the corresponding queue

It can be found that when the switch is not routed to the corresponding queue, it has successfully triggered our custom callback function, and then looking at the rabbitMQ console, we can find that the message has been successfully delivered to the switch

At this point, we have used two simple error simulations to enable the program to smoothly enter our predefined callbacks. If we encounter a failure to send, we can customize the message retransmission mechanism in the failed callback. Minimize the problem of message loss

4) Summary

We can use the publisher-confirm and publisher-return two error capture mechanisms to avoid message loss on the link producer -> exchange -> queue

• publisher-confirm

  1. The message is successfully sent to exchange, and ack is returned
  2. The message is not successfully sent to the exchange, and nack is returned
  3. If there is an exception during the message sending process, and the receipt is not received, it will enter failureCallback callback

• publisher-return

  1. The message is successfully sent to the exchange, but not routed to the queue, call the custom callback function returnCallback

2. The message storage is lost

What does the message storage loss mean? In fact, it is persistence. When the message has been successfully sent to the queue , at this time, if the consumer does not consume in time, rabbitMQ just crashes and restarts, then you will find that the message is lost at this time.

This is because MQ stores messages in memory by default, and we can ensure that messages in MQ are not lost by turning on the persistence function

In fact, when we create a switch or queue through the GUI provided by RabbitMQ, we can find this option of persistence.

If durability set to durable , we can find that no matter how to restart MQ, the switch and queue still exist after restarting.

But many times our switch and queue are not created on the GUI, but created by application code.

• switch persistence

• Queue persistence

• Message persistence

By default, the messages sent by AMQP are persistent and do not need to be specified specifically

3. Consumer consumption is lost

The mechanism adopted by RabbitMQ is when the confirmation message is immediately deleted after being consumed by the consumer

So how to confirm that the message has been consumed by the consumer? Then you have to rely on the receipt to confirm. After the consumer gets the message, it needs to send the ack receipt to RabbitMQ to indicate that it has processed the message. Among them, ack has three confirmation modes in AMQP:

• manual : Manual ack, you need to call api to send ack after the business code ends
• auto : automatic ack, spring monitors whether the listener code is abnormal, if there is no abnormality, it returns ack, otherwise it returns nack
• none : Turn off ack, MQ will delete the message immediately after the message is delivered

The above three methods are by modifying the configuration file:

1）manual

This method requires the user to manually confirm by himself, which is more flexible

If the execution logic is normal at this time, the message will be deleted on RabbitMQ, but if the executed logic throws an exception and does not enter the manual confirmation link, RabbitMQ will keep the message:

2）auto

This method will automatically confirm the message when there is no abnormality

We changed the confirmation method to auto in the configuration file for testing:

Under normal circumstances, there is no problem receiving messages, so we also create some abnormal situations:

We manually made some exceptions and found that while the message was not deleted by RabbitMQ, the console kept reporting errors, and we were endlessly trying to re-consume. This would inevitably be a bit of a crash if we put it online.

When the consumer has an exception, the message will continue to requeue (re-enter the queue) to the queue, and then resend to the consumer, and then abnormal again, requeue again, infinite loop, will lead to the soaring of MQ message processing

The reason why this happens is because of RabbitMQ's message failure retry mechanism, but in many cases we may not want to retry all the time, just after a few attempts, if it fails, we will give up processing. At this time, we need to configure Configuration failure retry mechanism in the file:

After opening the configuration, we restart the project to observe

It can be seen from the console that after 3 retries, SpringAMQP will throw an exception AmqpRejectAndDontRequeueException , indicating that the local retry mechanism has taken effect. And when we go back to the RabbitMQ console, we can see that the corresponding message has been deleted, indicating that SpringAMQP returned ack at the end, which caused the message to be deleted by MQ

However, this approach does not elegant, delete messages directly after retrying too violence , then there is no better way? The answer is yes!

We can use the MessageRecovery interface provided by AMQP to achieve this. There are three different implementations of the interface:

• RejectAndDontRequeueRecoverer : After the retries are exhausted, directly reject and lose the message. The default method, the above is to use this method
• ImmediateRequeueMessageRecoverer : After the retries are exhausted, nack is returned and the message is re-entered
• RepublishMessageRecoverer : After the retries are exhausted, the failure message will be delivered to the specified switch

The three methods can be adopted according to different scenarios. After analyzing it, it is not difficult to find that the third RepublishMessageRecoverer is more elegant~ When the retry fails, the message will be delivered to a designated queue for storing abnormal messages, and the subsequent manual processing will be concentrated. ! The specific usage is as follows:

After customizing the exception handling, we restart the project to view the console:

It can be found that after 3 retries, our exception message entered our custom exception queue

3）none

There is nothing to say about this method~ MQ will delete the message regardless of whether the message is abnormal or not!

4. Summary

If I ask you again in the interview at this time, how to ensure the reliability of RabbitMQ news? Then you have to chat

How to ensure that the message is not lost?

1) What are the lost scenes first?

Message loss may occur when sent (not delivered to exchange / not routed to queue), message is not persisted and MQ is down, consumer receives message and fails to consume it correctly

2) Then how to prevent

• Enable the producer confirmation mechanism to ensure that the producer's message can reach the queue

Confirmation mechanisms include publisher-confirm and publisher-return

When it is not delivered to switch we can publisher-confirm returned ack and nack

When switch not successfully routed to queue , we can publisher-return to confirm the custom callback function, each RabbitTemplate can configure only one ReturnCallback

• Enable the persistence function to ensure that the message will not be lost in the queue before being consumed

The persistence function is divided into switch persistence, queue persistence and message persistence. We all need to set durable to true

• Turn on the consumer confirmation mechanism at a minimum level of auto

There are three types of consumer confirmation mechanisms: manual (manual confirmation), auto (automatic confirmation), none (close ack)

• Failure retry mechanism

We manually set MessageResoverer to RepublishMessageRecoverer , and transfer the failed messages to the exception queue for manual processing

---

After answering this set of combo punches, the interviewer still has to silently admit that you have something?

Of course, this is just one of RabbitMQ's problems, we will continue to solve several other problems in the next article~

Don't talk about it, don't be lazy, and be a X as an architecture with Xiaocai~ Follow me to be a companion, so that Xiaocai is no longer alone. See you below!

If you work harder today, you will be able to say less begging words tomorrow!
I am Xiaocai, a man who becomes stronger with you. 💋
WeChat public account has been opened, , students who have not followed please remember to pay attention!