Golang implements RabbitMQ's delay queue

Before reading this article, you should already understand some concepts of RabbitMQ, such as queues, exchanges and so on.

Introduction to Delay Queues

Messages in a queue are delayed for a period of time before being consumed by consumers. Such a queue can be called a delayed queue.

The application scenarios of the delayed queue are very wide, such as: cancel the order if the payment is not made within 30 minutes after placing the order; send a notification at a certain time, etc.

Delay queue implementation via dead letter

Through the introduction of Golang's implementation of RabbitMQ's dead letter queue , we can easily implement a delay queue.

1. Cancel the consumer of the normal queue;
2. Set TTL when sending a message;

Through the above two points, the messages of the normal queue will never be consumed, but wait for the message TTL to expire, enter the dead letter queue, and let the dead letter consumers consume, so as to achieve the effect of delaying the queue.

It seems that there is no problem with the above, but after testing it, you will find that message will not "die as scheduled" .

When a message with a TTL of 60s is produced first, and then a message with a TTL of 5s is produced, the second message will not expire and enter the dead letter queue after 5s, but needs to wait until the TTL of the first message expires. The message goes to the dead letter queue together. This is because RabbitMQ only determines if the first message in the queue is expired.

Implementing delayed queues via plugins

Architecture

For the above problem, there is a natural solution, which is to solve it through the rabbitmq_delayed_message_exchange plugin of RabbitMQ. This article will not go into details about the installation of RabbitMQ and plugins. You can refer to to install or use Docker to install.

The principle of this plugin is to temporarily store the message in the mnesia (a distributed data system) table at the switch , delay delivery to the queue, and wait until the message expires before delivering it to the queue.

With a simple understanding of the principle of the plug-in, we can design the delay queue in this way.

accomplish

Key points implemented by the producer:

1. When declaring the switch, it is not of type direct , but of type x-delayed-message , which is the type provided by the plugin;

2. The switch needs to add "x-delayed-type": "direct" parameter settings;

3. When publishing a message, set x-delay parameter in the Headers to control the expiration time of the message from the switch;

err = mqCh.Publish(constant.Exchange1, constant.RoutingKey1, false, false, amqp.Publishing{
    ContentType: "text/plain",
    Body:        []byte(message),
    //Expiration: "10000", // 消息过期时间（消息级别）,毫秒
    Headers: map[string]interface{}{
        "x-delay": "5000", // 消息从交换机过期时间,毫秒（x-dead-message插件提供）
    },
})

The complete code of the producer:

// producter.go
package main

import (
    "fmt"
    "github.com/streadway/amqp"
    "learn_gin/go/rabbitmq/delayletter/constant"
    "learn_gin/go/rabbitmq/util"
    "strconv"
    "time"
)

func main() {
    // # ========== 1.创建连接 ==========
    mq := util.NewRabbitMQ()
    defer mq.Close()
    mqCh := mq.Channel

    // # ========== 2.设置队列（队列、交换机、绑定） ==========
    // 声明队列
    var err error
    _, err = mqCh.QueueDeclare(constant.Queue1, true, false, false, false, amqp.Table{
        // "x-message-ttl": 60000, // 消息过期时间（队列级别）,毫秒
    })
    util.FailOnError(err, "创建队列失败")

    // 声明交换机
    //err = mqCh.ExchangeDeclare(Exchange1, amqp.ExchangeDirect, true, false, false, false, nil)
    err = mqCh.ExchangeDeclare(constant.Exchange1, "x-delayed-message", true, false, false, false, amqp.Table{
        "x-delayed-type": "direct", 
    })
    util.FailOnError(err, "创建交换机失败")

    // 队列绑定（将队列、routing-key、交换机三者绑定到一起）
    err = mqCh.QueueBind(constant.Queue1, constant.RoutingKey1, constant.Exchange1, false, nil)
    util.FailOnError(err, "队列、交换机、routing-key 绑定失败")

    // # ========== 4.发布消息 ==========
    message := "msg" + strconv.Itoa(int(time.Now().Unix()))
    fmt.Println(message)
    // 发布消息
    err = mqCh.Publish(constant.Exchange1, constant.RoutingKey1, false, false, amqp.Publishing{
        ContentType: "text/plain",
        Body:        []byte(message),
        //Expiration: "10000", // 消息过期时间（消息级别）,毫秒
        Headers: map[string]interface{}{
            "x-delay": "5000", // 消息从交换机过期时间,毫秒（x-dead-message插件提供）
        },
    })
    util.FailOnError(err, "消息发布失败")
}

Since queues and exchanges are established on the producer side, consumers do not need special settings, and the code is directly attached.

Consumer complete code:

// consumer.go
package main

import (
    "learn_gin/go/rabbitmq/delayletter/constant"
    "learn_gin/go/rabbitmq/util"
    "log"
)

func main() {
    // # ========== 1.创建连接 ==========
    mq := util.NewRabbitMQ()
    defer mq.Close()
    mqCh := mq.Channel

    // # ========== 2.消费消息 ==========
    msgsCh, err := mqCh.Consume(constant.Queue1, "", false, false, false, false, nil)
    util.FailOnError(err, "消费队列失败")

    forever := make(chan bool)
    go func() {
        for d := range msgsCh {
            // 要实现的逻辑
            log.Printf("接收的消息: %s", d.Body)

            // 手动应答
            d.Ack(false)
            //d.Reject(true)
        }
    }()
    log.Printf("[*] Waiting for message, To exit press CTRL+C")
    <-forever
}

end！

Source code Mr-houzi/go-demo