Understand RabbitMQ in one article

This article has been included in the github repository. This repository is used to share a summary of Java-related knowledge, including Java basics, MySQL, Spring Boot, MyBatis, Redis, RabbitMQ, computer networks, data structures and algorithms, etc. Welcome everyone to mention pr and star!

github address: https://github.com/Tyson0314/Java-learning

If github is not accessible, you can visit the gitee repository.

gitee address: https://gitee.com/tysondai/Java-learning

Article directory:

Introduction

RabbitMQ is a message queue developed by erlang. Message queues are used for asynchronous collaboration between applications.

basic concept

Message: consists of message header and message body. The message body is opaque, while the message header consists of a series of optional attributes, including routing-key, priority, delivery-mode (whether persistent storage), etc.

Publisher: The producer of the message.

Exchange: Receive messages and route them to one or more Queues. The default exchange is the default direct-connected exchange. The name is an empty string. Each new queue will be automatically bound to the default exchange. The bound routing key name is the same as the queue name.

Binding: Associate Exchange and Queue through Binding, so that Exchange knows which Queue to route the message to.

Queue: Store messages, the characteristic of the queue is first in, first out. A message can be distributed to one or more queues.

Virtual host: Each vhost is essentially a mini version of RabbitMQ server, with its own queue, switch, binding, and permission mechanism. The vhost is the basis of the AMQP concept and must be specified when connecting. The default vhost of RabbitMQ is /. When multiple different users use the services provided by the same RabbitMQ server, multiple vhosts can be divided, and each user creates exchange and queue in his own vhost.

Broker: Message queue server entity.

When to use MQ

For some operations that do not need to take effect immediately, they can be split out, executed asynchronously, and implemented using message queues.

Taking a common order system as an example, the business logic after the user clicks the order button may include: deducting inventory, generating corresponding documents, and sending SMS notifications. MQ can be used in this scenario. The SMS notification is placed in MQ for asynchronous execution. After the main process of placing an order (such as deducting inventory and generating corresponding documents) is completed, a message is sent to MQ, so that the main process is quickly completed, and another thread consumes the MQ message.

Pros and cons

Disadvantages: The use of erlang is not conducive to secondary development and maintenance; the performance is worse than Kafka. In the case of persistent messages and ACK confirmation, the single machine throughput of production and consumption messages is about 10,000 to 20,000, and the single machine throughput of Kafka is 100,000. level.

Advantages: a management interface, easy to use; high reliability; rich functions, support for message persistence, message confirmation mechanism, and multiple message distribution mechanisms.

Exchange type

Exchange distributes messages according to different types of distribution strategies. There are currently four types: direct, fanout, topic, and headers. The headers mode is routed according to the headers of the message. In addition, the headers switch and the direct switch are exactly the same, but the performance is much worse.

Exchange rules.

type name	Type description
fanout	Route all messages sent to the Exchange to all Queues bound to it
direct	Routing Key==Binding Key
topic	Fuzzy matching
headers	Exchange does not rely on the matching rules of routing key and binding key to route messages, but matches based on the header attribute in the message content sent.

direct

The direct switch will route the message to the queue where the binding key and routing key exactly match. It is an exact match, unicast mode.

fanout

All messages sent to the fanout exchange will be routed to all queues bound to the exchange. The fanout type is the fastest to forward messages.

topic

The topic switch uses routing key and binding key for fuzzy matching, and if the matching succeeds, the message is sent to the corresponding queue. Both routing key and binding key are character strings separated by periods ".". There can be two special characters "*" and "#" in the binding key, which are used for fuzzy matching, and "*" is used to match a word. "#" is used to match multiple words.

headers

The headers switch performs routing based on the headers attribute in the message content sent. Specify a set of key-value pairs when binding Queue and Exchange; when a message is sent to Exchange, RabbitMQ will get the message headers (also in the form of a key-value pair), and compare whether the key-value pairs completely match Queue and The key-value pair specified during Exchange binding; if it matches exactly, the message will be routed to the Queue, otherwise it will not be routed to the Queue.

Message is lost

Message loss scenarios: messages from producer to RabbitMQ Server are lost, messages stored in RabbitMQ Server are lost, and messages from RabbitMQ Server to consumer are lost.

Message loss is solved from three aspects: producer confirmation mechanism, consumer manual confirmation of the message and persistence.

Producer confirmation mechanism

The producer sends a message to the queue, and cannot ensure that the sent message reaches the server successfully.

Solution:

1. Transaction mechanism. After a message is sent, the sender will be blocked, waiting for RabbitMQ's response, before it can continue to send the next message. Poor performance.
2. Turn on the producer confirmation mechanism, as long as the message is successfully sent to the switch, RabbitMQ will send an ack to the producer (even if the message is not received by the Queue, it will send an ack). If the message is not successfully sent to the switch, a nack message will be sent to indicate that the sending has failed.

In Springboot, confirm mode is set through the publisher-confirms parameter:

spring:
    rabbitmq:   
        #开启 confirm 确认机制
        publisher-confirms: true

Provide a callback method on the production side. When the server confirms one or more messages, the producer will call back this method and perform follow-up processing on the message based on the specific results, such as resending, logging, etc.

// 消息是否成功发送到Exchange
final RabbitTemplate.ConfirmCallback confirmCallback = (CorrelationData correlationData, boolean ack, String cause) -> {
            log.info("correlationData: " + correlationData);
            log.info("ack: " + ack);
            if(!ack) {
                log.info("异常处理....");
            }
    };

rabbitTemplate.setConfirmCallback(confirmCallback);

Unreachable message

The producer confirmation mechanism only ensures that the message arrives at the switch correctly. Messages that fail to be routed from the switch to the Queue will be discarded, resulting in message loss.

For non-routable messages, there are two processing methods: Return message mechanism and backup switch.

Return message mechanism

The Return message mechanism provides a callback function ReturnCallback, which will be called back when the message fails to be routed from the switch to the Queue. You need to set mandatory to true to monitor unreachable messages.

spring:
    rabbitmq:
        #触发ReturnCallback必须设置mandatory=true, 否则Exchange没有找到Queue就会丢弃掉消息, 而不会触发ReturnCallback
        template.mandatory: true

Monitor route unreachable messages through ReturnCallback.

    final RabbitTemplate.ReturnCallback returnCallback = (Message message, int replyCode, String replyText, String exchange, String routingKey) ->
            log.info("return exchange: " + exchange + ", routingKey: "
                    + routingKey + ", replyCode: " + replyCode + ", replyText: " + replyText);
rabbitTemplate.setReturnCallback(returnCallback);

When a message fails to be routed from the switch to the Queue, it will return return exchange: , routingKey: MAIL, replyCode: 312, replyText: NO_ROUTE .

Backup switch

The alternate-exchange of the backup exchange is a normal exchange. When you send a message to the corresponding exchange, if it does not match the queue, it will be automatically transferred to the queue corresponding to the backup exchange so that the message will not be lost.

Consumer manual message confirmation

It is possible that the consumer may crash before the MQ service has time to process the message, causing the message to be lost. Because the messager uses automatic ack by default, once the consumer receives the message, he will notify the MQ Server that the message has been processed, and MQ will remove the message.

Solution: The consumer is set to manually confirm the message. After the consumer has processed the logic, he replies ack to the broker, indicating that the message has been successfully consumed and can be deleted from the broker. When the messager fails to consume, it responds with nack to the broker, and decides whether to re-enter the queue or remove it from the broker according to the configuration, or enter the dead letter queue. As long as the consumer's acknowledgment is not received, the broker will keep the message, but it will not requeue or distribute it to other consumers.

Consumers set manual ack:

#设置消费端手动 ack
spring.rabbitmq.listener.simple.acknowledge-mode=manual

After the message is processed, manually confirm:

    @RabbitListener(queues = RabbitMqConfig.MAIL_QUEUE)
    public void onMessage(Message message, Channel channel) throws IOException {

        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        long deliveryTag = message.getMessageProperties().getDeliveryTag();
        //手工ack；第二个参数是multiple，设置为true，表示deliveryTag序列号之前（包括自身）的消息都已经收到，设为false则表示收到一条消息
        channel.basicAck(deliveryTag, true);
        System.out.println("mail listener receive: " + new String(message.getBody()));
    }

When the message consumption fails, the consumer responds to the broker with nack. If the consumer sets requeue to false, the broker will delete the message or enter the dead letter queue after nack, otherwise the message will re-enter the queue.

Persistence

If the RabbitMQ service is abnormally restarted, the message will be lost. RabbitMQ provides a persistence mechanism to persist messages in memory to the hard disk. Even if RabbitMQ is restarted, the messages will not be lost.

Message persistence needs to meet the following conditions:

1. Message settings are persistent. Before publishing the message, set the delivery mode to 2, which means that the message needs to be persisted.
2. Queue settings are persistent.
3. The switch settings are persistent.

When publishing a message to the switch, Rabbit will write the message to the persistent log before sending a response to the producer. Once a message is consumed from the queue and confirmed, RabbitMQ will remove the message from the persistent log. Before consuming the message, if RabbitMQ restarts, the server will automatically rebuild the switch and queue, and load the message in the persistent log to the corresponding queue or switch to ensure that the message will not be lost.

Mirror queue

When MQ fails, the service will be unavailable. Introduce RabbitMQ's mirroring queue mechanism to mirror the queue to other nodes in the cluster. If a node in the cluster fails, it can automatically switch to another node in the mirror to ensure service availability.

Usually each mirror queue contains a master and multiple slaves, corresponding to different nodes. All messages sent to the mirror queue are always sent directly to the master and all slaves. All actions except publish will only be sent to the master, and then the master will broadcast the result of the command execution to the slave. The consumption operation from the mirror queue is actually executed on the master.

Repeat consumption

There are two reasons for message duplication: 1. Message duplication during production, 2. Message duplication during consumption.

The producer sends a message to MQ. When the MQ confirms, network fluctuations occur and the producer does not receive the confirmation. At this time, the producer will resend the message, causing MQ to receive duplicate messages.

After the consumer succeeded in consumption, network fluctuations occurred when MQ was confirmed, and MQ did not receive the confirmation. In order to ensure that the message is not lost, MQ will continue to deliver the previous message to the consumer. At this time, the consumer received two identical messages. Since duplicate messages are caused by network reasons, it cannot be avoided.

Solution: When sending a message, let each message carry a global unique ID. When consuming the message, first determine whether the message has been consumed, so as to ensure the idempotence of the message consumption logic. The specific consumption process is:

1. After receiving the message, the consumer first queries whether the message exists in redis/db according to the id
2. If it does not exist, it will be consumed normally and written to redis/db after consumption.
3. If it exists, it proves that the message has been consumed and discarded directly

Consumer current limit

When the RabbitMQ server accumulates a large amount of messages, the messages in the queue will flood into the consumer, which may cause the consumer server to crash. In this case, it is necessary to limit current on the consumer side.

Spring RabbitMQ provides the parameter prefetch to set the number of messages processed by a single request. If the messages processed by the consumer at the same time reach the maximum value, the consumer will block and will not consume new messages. New messages will not be consumed until there is a message ack.

Turn on current limit on the consumer side:

#在单个请求中处理的消息个数，unack的最大数量
spring.rabbitmq.listener.simple.prefetch=2

Native RabbitMQ also provides two parameters, prefetchSize and global. Spring RabbitMQ does not have these two parameters.

//单条消息大小限制，0代表不限制
//global：限制限流功能是channel级别的还是consumer级别。当设置为false，consumer级别，限流功能生效，设置为true没有了限流功能，因为channel级别尚未实现。
void basicQos(int prefetchSize, int prefetchCount, boolean global) throws IOException;

Dead letter queue

The queue for storing failed messages.

Reasons for message consumption failure:

• The message was rejected and the message was not re-entered (requeue=false)
• Message timed out and not consumed
• Maximum queue length reached

Set the exchange and queue of the dead letter queue, and then bind:

    @Bean
    public DirectExchange dlxExchange() {
        return new DirectExchange(RabbitMqConfig.DLX_EXCHANGE);
    }

    @Bean
    public Queue dlxQueue() {
        return new Queue(RabbitMqConfig.DLX_QUEUE, true);
    }

    @Bean
    public Binding bindingDeadExchange(Queue dlxQueue, DirectExchange deadExchange) {
        return BindingBuilder.bind(dlxQueue).to(deadExchange).with(RabbitMqConfig.DLX_QUEUE);
    }

Add two parameters to the normal queue to bind the normal queue to the dead letter queue. When message consumption fails, the message will be routed to the dead letter queue.

    @Bean
    public Queue sendSmsQueue() {
        Map<String,Object> arguments = new HashMap<>(2);
        // 绑定该队列到私信交换机
        arguments.put("x-dead-letter-exchange", RabbitMqConfig.DLX_EXCHANGE);
        arguments.put("x-dead-letter-routing-key", RabbitMqConfig.DLX_QUEUE);
        return new Queue(RabbitMqConfig.MAIL_QUEUE, true, false, false, arguments);
    }

Producer's complete code:

@Component
@Slf4j
public class MQProducer {

    @Autowired
    RabbitTemplate rabbitTemplate;

    @Autowired
    RandomUtil randomUtil;

    @Autowired
    UserService userService;

    final RabbitTemplate.ConfirmCallback confirmCallback = (CorrelationData correlationData, boolean ack, String cause) -> {
            log.info("correlationData: " + correlationData);
            log.info("ack: " + ack);
            if(!ack) {
                log.info("异常处理....");
            }
    };


    final RabbitTemplate.ReturnCallback returnCallback = (Message message, int replyCode, String replyText, String exchange, String routingKey) ->
            log.info("return exchange: " + exchange + ", routingKey: "
                    + routingKey + ", replyCode: " + replyCode + ", replyText: " + replyText);

    public void sendMail(String mail) {
        //貌似线程不安全 范围100000 - 999999
        Integer random = randomUtil.nextInt(100000, 999999);
        Map<String, String> map = new HashMap<>(2);
        String code = random.toString();
        map.put("mail", mail);
        map.put("code", code);

        MessageProperties mp = new MessageProperties();
        //在生产环境中这里不用Message，而是使用 fastJson 等工具将对象转换为 json 格式发送
        Message msg = new Message("tyson".getBytes(), mp);
        msg.getMessageProperties().setExpiration("3000");
        //如果消费端要设置为手工 ACK ，那么生产端发送消息的时候一定发送 correlationData ，并且全局唯一，用以唯一标识消息。
        CorrelationData correlationData = new CorrelationData("1234567890"+new Date());

        rabbitTemplate.setMandatory(true);
        rabbitTemplate.setConfirmCallback(confirmCallback);
        rabbitTemplate.setReturnCallback(returnCallback);
        rabbitTemplate.convertAndSend(RabbitMqConfig.MAIL_QUEUE, msg, correlationData);

        //存入redis
        userService.updateMailSendState(mail, code, MailConfig.MAIL_STATE_WAIT);
    }
}

Consumer complete code:

@Slf4j
@Component
public class DeadListener {

    @RabbitListener(queues = RabbitMqConfig.DLX_QUEUE)
    public void onMessage(Message message, Channel channel) throws IOException {

        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        long deliveryTag = message.getMessageProperties().getDeliveryTag();
        //手工ack
        channel.basicAck(deliveryTag,false);
        System.out.println("receive--1: " + new String(message.getBody()));
    }
}

When there is a dead letter in the normal queue, RabbitMQ will automatically republish the message to the set dead letter switch, and then be routed to the dead letter queue. You can monitor the messages in the dead letter queue for corresponding processing.

other

pull mode

The pull mode is mainly to get the message through the channel.basicGet method. The sample code is as follows:

GetResponse response = channel.basicGet(QUEUE_NAME, false);
System.out.println(new String(response.getBody()));
channel.basicAck(response.getEnvelope().getDeliveryTag(),false);

Message expiration time

When sending a message on the production side, you can set an expiration time for the message, in milliseconds (ms)

Message msg = new Message("tyson".getBytes(), mp);
msg.getMessageProperties().setExpiration("3000");

You can also specify the ttl of the queue when you create the queue. The calculation starts from the time the message enters the queue, and the messages that exceed this time will be removed.

Reference link

RabbitMQ basics

Springboot integrates RabbitMQ

RabbitMQ message persistence

RabbitMQ send mail code

online rabbitmq question

Finally, I will share a github repository with you. more than 200 classic computer books , including C language, C++, Java, Python, front-end, database, operating system, computer network, data structure and algorithm, machine learning, programming Life and so on, you can star, next time you find a book, search directly on it, and the warehouse is continuously updating~

github address: https://github.com/Tyson0314/java-books

If github is not accessible, you can visit the gitee repository.

gitee address: https://gitee.com/tysondai/java-books