RabbitMQ from the simplest to the deepest summary (two)

Write at the top

It’s been a long time since the last time I posted an article. In fact, I haven’t stopped writing during this period. I’m just busy looking for work and school closing. I re-organized the blog and will continue to update the article later. Updated~

• This article is a bit long, so it is divided into two
• PS: The article on Java Knowledge Questions and Answers on Github has not stopped writing, and it will be updated recently.

6. Advanced supplement

6.1 Expiration time setting (TTL)

The expiration time (TTL) is to message or the queue 160d16e5c407e7, which can be received by consumers only within the time range, and the message will be automatically deleted after the expiration time.

Note: We mainly talk about message expiration. In the first method of message expiration, the way to set the queue expiration will also be mentioned by the way.

1. Through the queue property setting, all messages in the queue have the same expiration time
2. Set the message individually, and the TTL of each message can be different

When the two methods are used at the same time, the value with the smaller TTL of the expiration time of the two methods shall prevail. Once the survival time of a message in the queue exceeds the set TTL value, it is called Dead Message and is delivered to the dead letter queue, and consumers will no longer receive the message (the dead letter queue is what we will talk about in the next point)

6.1.1 Expiration time applied to all messages

• Configuration class

@Configuration
public class RabbitMqConfiguration {

    public static final String TOPIC_EXCHANGE = "topic_order_exchange";
    public static final String TOPIC_QUEUE_NAME_1 = "test_topic_queue_1";
    public static final String TOPIC_ROUTINGKEY_1 = "test.*";

    @Bean
    public TopicExchange topicExchange() {
        return new TopicExchange(TOPIC_EXCHANGE);
    }

    @Bean
    public Queue topicQueue1() {
        // 创建参数 Map 容器
        Map<String, Object> args = new HashMap<>();
        // 设置消息过期时间 注意此处是数值 5000 不是字符串
        args.put("x-message-ttl", 5000);
        // 设置队列过期时间
        args.put("x-expires", 8000);
        // 在最后传入额外参数 即这些过期信息
        return new Queue(TOPIC_QUEUE_NAME_1, true, false, false, args);
    }

    @Bean
    public Binding bindingTopic1() {
        return BindingBuilder.bind(topicQueue1())
                .to(topicExchange())
                .with(TOPIC_ROUTINGKEY_1);
    }
}

1. creates the parameter Map container : the type is required in the Queue parameter, and it must be as required.
2. set message expiration time : The message expiration time set here will be applied to all messages.
3. set the queue expiration time
4. pass in additional parameters : set the expiration time configured above and pass in via Queue.

• Producer

@SpringBootTest(classes = RabbitmqSpringbootApplication.class)
@RunWith(SpringRunner.class)
public class RabbitMqTest {
    /**
     * 注入 RabbitTemplate
     */
    @Autowired

    @Test
    public void testTopicSendMessage() {
        rabbitTemplate.convertAndSend(RabbitMqConfiguration.TOPIC_EXCHANGE, "test.order.insert", "This is a message !");
    }
}

Do not configure the consumer, then you can see the effect in the web manager

6.1.2 Expiration time applied to individual messages

• Just keep the original configuration in the configuration, there is no need to configure the expiration time
• Configure the individual expiration time of the message in the producer

@SpringBootTest(classes = RabbitmqSpringbootApplication.class)
@RunWith(SpringRunner.class)
public class RabbitMqTest {
    /**
     * 注入 RabbitTemplate
     */
    @Autowired

    @Test
    public void testTopicSendMessage2() {
        MessagePostProcessor messagePostProcessor = new MessagePostProcessor(){
            public Message postProcessMessage(Message message){
                // 注意此处是 字符串 “5000”
                message.getMessageProperties().setExpiration("5000");
                message.getMessageProperties().setContentEncoding("UTF-8");
                return message;
            }
        };
        rabbitTemplate.convertAndSend(RabbitMqConfiguration.TOPIC_EXCHANGE, "test.order",
                "This is a message 002 !",messagePostProcessor);
    }
}

6.2 Dead letter queue

The official original text of a dead letter is Dead letter, which is a message mechanism in RabbitMQ. When you are consuming messages, if the following situations occur in the queue and the messages in the queue, it means that the current message has become a "dead letter". If configured Dead letter queue, these data will be transmitted to it, if there is no configuration will be directly discarded.

1. Message rejected
2. Message expired
3. The queue reaches the maximum length

However, the dead letter queue is not a very special existence. We only need to configure a switch, configure it in the consuming queue, and re-send the dead letter to the previously configured switch, and then be routed to the switch bound to the switch. In the queue, this queue is also a dead letter queue, so from the point of view of creation, it is no different from a normal queue.

6.2.1 Application Scenario

For example, in some important business queues, messages that are not consumed correctly are often not discarded, because if we want to restore the data after discarding, it is often necessary for the operation and maintenance personnel to obtain the original message from the log, and then re-deliver the message. Configuring the dead letter queue is equivalent to giving a temporary storage location for incorrectly consumed messages. When you need to restore it in the future, you only need to write the corresponding code.

6.2.2 Implementation

• Define a switch and queue to handle dead letters

@Configuration
public class DeadRabbitMqConfiguration{

    @Bean
    public DirectExchange deadDirect(){
        return new DirectExchange("dead_direct_exchange");}

    @Bean
    public Queue deadQueue(){
        return new Queue("dead_direct_queue");}
    @Bean
    public Binding deadBinds(){
        return BindingBuilder.bind(deadQueue()).to(deadDirect()).with("dead");
    }
}

• Specify the dead letter queue in the normal consumption queue

@Configuration
public class RabbitMqConfiguration {

    public static final String TOPIC_EXCHANGE = "topic_order_exchange";
    public static final String TOPIC_QUEUE_NAME_1 = "test_topic_queue_1";
    public static final String TOPIC_ROUTINGKEY_1 = "test.*";

    @Bean
    public TopicExchange topicExchange() {
        return new TopicExchange(TOPIC_EXCHANGE);
    }

    @Bean
    public Queue topicQueue1() {
        // 设置过期时间
        Map<String, Object> args = new HashMap<>();
        args.put("x-message-ttl", 5000);
        // 设置死信队列交换器
        args.put("x-dead-letter-exchange","dead_direct_exchange");
        // 设置交换路由的路由key fanout 模式不需要配置此条
        args.put("x-dead-letter-routing-key","dead");
        return new Queue(TOPIC_QUEUE_NAME_1, true, false, false, args);
    }

    @Bean
    public Binding bindingTopic1() {
        return BindingBuilder.bind(topicQueue1())
                .to(topicExchange())
                .with(TOPIC_ROUTINGKEY_1);
    }
}

6.3 Memory and Disk Monitoring

6.3.1 Memory Alarm and Control

In order to prevent the server from crashing due to insufficient memory, RabbitMQ sets a threshold. When the memory usage exceeds the threshold, RabbitMQ will temporarily block all client connections and stop receiving new messages.

There are two ways to modify this threshold

1. By command (choose one of the two)

   • The command method will be invalid after Broker restarts

# 通过百分比设置的命令 <fraction> 处代表百分比小数例如 0.6
rabbitmqctl set_vm_memory_high_watermark <fraction>
# 通过绝对值设置的命令 <value> 处代表设置的一个固定值例如 700MB
rabbitmqctl set_vm_memory_high_watermark absolute <value>

2. By modifying the configuration file rabbitmq.conf

   • The configuration file will be loaded every time it is started and is permanently valid

# 百分比设置 默认值为 0.4 推荐 0.4-0.7 之间
vm_memory_high_watermark.relative = 0.5
# 固定值设置
vm_memory_high_watermark.absolute = 2GB

6.3.2 Memory paging

Before the client connection and the producer are blocked, it will try to page the messages in the queue to the disk. This idea is actually very common in the operating system to satisfy the normal processing of the message to the greatest extent.

When memory paging occurs, both persistent and non-persistent messages will be transferred to the disk, and since the persistent message originally has a persistent copy on the disk, the persistent message will be removed first .

By default, when the memory reaches 50% of the threshold, paging is performed.

Can be modified by setting vm_memory_high_watermark_paging_ratio

# 值小于 1, 如果大于 1 就没有意义了
vm_memory_high_watermark_paging_ratio = 0.6

6.3.3 Disk Warning

If endless page changes, it is also very likely to run out of disk space and cause the server to crash. Therefore, RabbitMQ provides a disk warning threshold. When it is lower than this value, an alarm will be issued. The default is 50MB, which can be passed Modification of command mode

# 固定值
rabbitmqctl set_disk_free_limit <disk_limit>
# 百分数
rabbitmqctl set_disk_free_limit memory_limit <fraction>

6.4 Reliable delivery of messages

When the producer sends a message to RabbitMQ, the sending may fail due to various reasons such as the network. Therefore, RabbitMQ provides a series of mechanisms to ensure the reliable delivery of the message, which can be roughly divided into two parts: the producer and the consumer.

6.4.1 Mechanism in the producer

As the sender of the message, the producer needs to ensure that its message is sent successfully. RabbitMQ provides two ways to ensure this. -

1. confirm confirm mode
2. return mode

6.4.1.1 confirm confirm mode

After the producer sends the message, it will asynchronously wait to receive an ack response. After receiving the returned ack confirmation message, it will call the confirmCallback interface for processing according to whether the ack is true or false.

• Configuration class

spring:
  rabbitmq:
    # 发送确认
    publisher-confirm-type: correlated

• Implement the confirm method of the ConfirmCallback interface

@Component
public class ConfirmCallbackService implements RabbitTemplate.ConfirmCallback {

    /**
     * @param correlationData 相关配置信息
     * @param ack             exchange交换机 是否成功收到了消息。true 成功，false代表失败
     * @param cause           失败原因
     */
    @Override
    public void confirm(CorrelationData correlationData, boolean ack, String cause) {
        if (ack) {
            //接收成功
            System.out.println("消息成功发送到交换机");
        } else {
            //接收失败
            System.out.println("消息发送到交换机失败，失败原因: " + cause);
            // TODO 可以处理失败的消息，例如再次发送等等
        }
    }
}

• Declare queues and exchanges

@Configuration
public class RabbitMqConfig {
    @Bean()
    public Queue confirmTestQueue() {
        return new Queue("confirm_test_queue", true, false, false);
    }

    @Bean()
    public FanoutExchange confirmTestExchange() {
        return new FanoutExchange("confirm_test_exchange");
    }

    @Bean
    public Binding confirmTestFanoutExchangeAndQueue() {
        return BindingBuilder.bind(confirmTestQueue()).to(confirmTestExchange());
    }
}

• Producer

@SpringBootTest(classes = RabbitmqSpringbootApplication.class)
@RunWith(SpringRunner.class)
public class RabbitMqTest {
    /**
     * 注入 RabbitTemplate
     */
    @Autowired
    
     /**
     * 注入 ConfirmCallbackService
     */
    @Autowired
    private ConfirmCallbackService confirmCallbackService;

    @Test
    public void testConfirm() {
        // 设置确认回调类
        rabbitTemplate.setConfirmCallback(confirmCallbackService);
        // 发送消息
        rabbitTemplate.convertAndSend("confirm_test_exchange", "", "ConfirmCallback !");
    }
}

6.4.1.2 return mode

When the Exchange fails to send to the Queue, a returnsCallback will be called. We can implement this interface and then deal with this failure.

• Enable sending callback in the configuration file

spring:
  rabbitmq:
    # 发送回调
    publisher-returns: true

• Implement the returnedMessage method of ReturnsCallback

//  public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) 已经属于过时方法了
@Component
public class ReturnCallbackService implements RabbitTemplate.ReturnsCallback {
    @Override
    public void returnedMessage(ReturnedMessage returned) {
        System.out.println(returned);
    }
}

• Declare queues and exchanges (Direct mode)

@Configuration
public class RabbitMqConfig {

    @Bean()
    public Queue returnsTestQueue() {
        return new Queue("return_test_queue", true, false, false);
    }

    @Bean()
    public DirectExchange returnsTestExchange() {
        return new DirectExchange("returns_test_exchange");
    }

    @Bean
    public Binding returnsTestDirectExchangeAndQueue() {
        return BindingBuilder.bind(returnsTestQueue()).to(returnsTestExchange()).with("info");
    }
}

• Producer

@SpringBootTest(classes = RabbitmqSpringbootApplication.class)
@RunWith(SpringRunner.class)
public class RabbitMqTest {
    /**
     * 注入 RabbitTemplate
     */
    @Autowired
    
    /**
     * 注入 ConfirmCallbackService
     */
    @Autowired
    private ConfirmCallbackService confirmCallbackService;
    
    /**
     * 注入 ReturnCallbackService
     */
    @Autowired
    private ReturnCallbackService returnCallbackService;

    @Test
    public void testReturn() {
        // 确保消息发送失败后可以重新返回到队列中
        rabbitTemplate.setMandatory(true);
        // 消息投递到队列失败回调处理
        rabbitTemplate.setReturnsCallback(returnCallbackService);
        // 消息投递确认模式
        rabbitTemplate.setConfirmCallback(confirmCallbackService);
        // 发送消息
        rabbitTemplate.convertAndSend("returns_test_exchange", "info", "ReturnsCallback !");
    }
}

• Modify different routing keys to test the results.

6.4.2 Mechanisms in the consumer

6.4.2.1 ack confirmation mechanism

ack means the confirmation of receiving the message, the default is automatic confirmation, but it has three types

Introduction to the acknowledge-mode option

• auto: automatic confirmation, the default option
• manual: manual confirmation (assignment according to capacity needs to be set to manual confirmation)
• none: do not confirm, automatically discard after sending

Among them, automatic confirmation means that once the message is received by the consumer, it will automatically confirm the receipt and delete the message from the queue.

However, in actual business processing, the correctly received message may not be processed correctly due to business problems. However, if the manual confirmation method is set, you need to call channel.basicAck( after the business processing is successful. ), sign manually. If an exception occurs, call the channel.basicNack() method to automatically resend the message.

• Configuration file

spring:
  rabbitmq:
    listener:
      simple:
          # 手动确认
        acknowledge-mode: manual 

• consumer

@Component
@RabbitListener(queues = "confirm_test_queue")
public class TestConsumer {

    @RabbitHandler
    public void processHandler(String msg, Channel channel, Message message) throws IOException {
        long deliveryTag = message.getMessageProperties().getDeliveryTag();
        try {
            System.out.println("消息内容: " + new String(message.getBody()));

            System.out.println("业务出错的位置：");
            int i = 66 / 0;
            
            // 手动签收 deliveryTag标识代表队列可以删除了
            channel.basicAck(deliveryTag, true);
        } catch (Exception e) {
            // 拒绝签收
            channel.basicNack(deliveryTag, true, true);
        }
    }
}

6.5 Cluster & 6.6 Distributed Transaction (to be updated)

Since these two points are not short in length, I really don't want to write things in a simple way and put them in a separate article later and publish them.

For the construction of the cluster, please refer to: https://blog.csdn.net/belonghuang157405/article/details/83540148