This article takes you to understand the simplest message queue implementation

When looking at the company's redis queue recently, I found that the bottom layer is go-zero of queue . In this article, take a look at queue , and hope to learn about the minimal design practice of mq

use

Combined with other mq use experience, the basic use process:

1. Create producer or consumer
2. Start mq
3. Production news/consumption news

Corresponds to queue , which is roughly this:

Create queue

// 生产者创建工厂
producer := newMockedProducer()
// 消费者创建工厂
consumer := newMockedConsumer()
// 将生产者以及消费者的创建工厂函数传递给 NewQueue()
q := queue.NewQueue(func() (Producer, error) {
  return producer, nil
}, func() (Consumer, error) {
  return consumer, nil
})

Let's see what build conditions are needed for NewQueue

1. producer constructor
2. consumer constructor

Pass the factory functions of both parties to queue for execution and retry.

The purpose of these two needs is to encapsulate the construction of the producer/consumer and the production/consumption of the message in mq , and to hand over the entire logic of the producer/consumer to the developer for processing:

type (
    // 开发者需要实现此接口
    Producer interface {
        AddListener(listener ProduceListener)
        Produce() (string, bool)
    }
    ...
    // ProducerFactory定义了生成Producer的方法
    ProducerFactory func() (Producer, error)
)

1. In fact, the logic of the producer is handed over to the developer to complete it. mq only responsible for the message delivery and scheduling between producers/consumers.
2. The design of the factory method is to hand over the two tasks of the producer itself and the production message to queue for scheduling or retrying.

Production msg

The production news must of course go back to the producer itself:

type mockedProducer struct {
    total int32
    count int32
  // 使用waitgroup来模拟任务的完成
    wait  sync.WaitGroup
}
// 实现 Producer interface 的方法：Produce()
func (p *mockedProducer) Produce() (string, bool) {
    if atomic.AddInt32(&p.count, 1) <= p.total {
        p.wait.Done()
        return "item", true
    }
    time.Sleep(time.Second)
    return "", false
}

The producer programming in queue

• Produce() : The logic of the production message is written by the developer
• AddListener() : Producer

Consume msg

Similar to the producer:

type mockedConsumer struct {
    count  int32
}

func (c *mockedConsumer) Consume(string) error {
    atomic.AddInt32(&c.count, 1)
    return nil
}

Start queue

Start, and then verify whether the data transmission between the producer and consumer mentioned above is successful:

func TestQueue(t *testing.T) {
    producer := newMockedProducer(rounds)
    consumer := newMockedConsumer()
    // 创建 queue
    q := NewQueue(func() (Producer, error) {
        return producer, nil
    }, func() (Consumer, error) {
        return consumer, nil
    })
    // 当生产者生产完毕，执行 Stop() 关闭生产端生产
    go func() {
        producer.wait.Wait()
    // mq生产端停止生产，不是mq本身 Stop 运行
        q.Stop()
    }()
    // 启动
    q.Start()
    // 验证生产消费端是否消息消费完成
    assert.Equal(t, int32(rounds), atomic.LoadInt32(&consumer.count))
}

The above is the queue code for getting started with 0609b51924193a. Developers can freely define the production/consumption logic of the producer/consumer according to the actual situation of their business.

Overall design

The overall process is as shown above:

1. All communications are channel out by 0609b5192419ca
2. By adding the listener listener and the event trigger event , it is equivalent to separating the trigger logic
3. The producer has produceone , this is the logic of producing the message, but the Produce() is written by the developer [ interface above is exactly this function]
4. Similarly for consumers, Consume()

The basic message flow enters the above picture and the above description. We will leave the specific code analysis to the next article. Our analysis, especially how to control channel is the core of the entire design.

to sum up

This article briefly introduces the design queue from the use and analysis of the entire architecture. In the next article, we will go into the source code, analyze the internal message flow and channel control.

go-zero more design and implementation articles about 0609b519241af0, you can continue to follow us. Welcome everyone to pay attention and use.

project address

https://github.com/tal-tech/go-zero

Welcome to go-zero and star support us!

WeChat Exchange Group

Follow the "microservice practice" official account and reply to the group to get the QR code of the community group.

For the go-zero series of articles, please refer to the "Microservice Practice" public account