How to make the message queue reach the maximum throughput?

Have you paid attention to throughput when using message queues?

Have you considered the factors affecting throughput?

Have you considered how to improve?

Have you summarized best practices?

This article takes you to discuss the implementation of the Go Let's go!

Some thoughts on throughput

• The throughput of writing to the message queue depends on the following two aspects

  • Network bandwidth
  • Message queue (such as Kafka) write speed

  The best throughput is to make one of them full, and under normal circumstances, the intranet bandwidth will be very high, and it is unlikely to be full, so it is natural to say that the writing speed of the message queue is full. There are two points Need to balance

  • The size or number of bytes of messages written in batches
  • How long to delay writing

  go-zero's PeriodicalExecutor and ChunkExecutor are designed for this situation

• The throughput of consuming messages from the message queue depends on the following two aspects

  • The reading speed of the message queue, in general, the reading speed of the message queue itself is fast enough compared to the speed of processing messages
  • Processing speed, this depends on the business

  The core problem here is that you have to read too many messages into the memory without considering the business processing speed, otherwise it may cause two problems:

  • The memory usage is too high, even OOM appears, pod also has memory limit
  • When stopping pod , the accumulated messages are too late to be processed and the messages are lost

Solutions and implementation

Borrow a picture of Rob Pike , which is similar to queue consumption. gopher on the left are taken from the queue, and the four gopher on the right are taken over for processing. The ideal result is that the rates on the left and right are basically the same, no one is wasting, no one is waiting, and there is no accumulation in the middle exchange.

Let's take a look at how go-zero is implemented:

• Producer end

    for {
        select {
        case <-q.quit:
            logx.Info("Quitting producer")
            return
        default:
            if v, ok := q.produceOne(producer); ok {
                q.channel <- v
            }
        }
    }

No exit events will pass produceOne to read a message that is written after the success of channel . Using chan can solve the problem of connection between reading and consumption.

• Consumer end

    for {
        select {
        case message, ok := <-q.channel:
            if ok {
                q.consumeOne(consumer, message)
            } else {
                logx.Info("Task channel was closed, quitting consumer...")
                return
            }
        case event := <-eventChan:
            consumer.OnEvent(event)
        }
    }

Here, if you get the message, process it. When ok is false , it means channel has been closed and you can exit the entire processing loop. At the same time, we also support redis queue pause/resume . We used this queue a lot in social scenes, and we can notify consumer pause and continue.

• Start queue , with these we can achieve throughput tuning producer/consumer

func (q *Queue) Start() {
    q.startProducers(q.producerCount)
    q.startConsumers(q.consumerCount)

    q.producerRoutineGroup.Wait()
    close(q.channel)
    q.consumerRoutineGroup.Wait()
}

It should be noted here that you must stop producer first, and then wait for consumer to finish processing.

So far, the core control code is basically finished. In fact, it looks quite simple. You can also go to https://github.com/tal-tech/go-zero/tree/master/core/queue to see the complete implementation .

how to use

Basic usage process:

1. Create producer or consumer
2. Start queue
3. Production news / consumption news

Corresponding to queue , roughly as follows:

Create queue

// 生产者创建工厂
producer := newMockedProducer()
// 消费者创建工厂
consumer := newMockedConsumer()
// 将生产者以及消费者的创建工厂函数传递给 NewQueue()
q := queue.NewQueue(func() (Producer, error) {
  return producer, nil
}, func() (Consumer, error) {
  return consumer, nil
})

Let's see what parameters are needed for NewQueue

1. producer factory method
2. consumer Factory Method

producer & consumer the factory function of queue , which is responsible for the creation. The framework provides Producer and Consumer interfaces and factory method definitions, and then the entire process control queue implementation will be completed automatically.

Production message

We simulate by customizing a mockedProducer

type mockedProducer struct {
    total int32
    count int32
  // 使用waitgroup来模拟任务的完成
    wait  sync.WaitGroup
}
// 实现 Producer interface 的方法：Produce()
func (p *mockedProducer) Produce() (string, bool) {
    if atomic.AddInt32(&p.count, 1) <= p.total {
        p.wait.Done()
        return "item", true
    }
    time.Sleep(time.Second)
    return "", false
}

queue must be implemented:

• Produce() : The logic of the production message is written by the developer
• AddListener() : Add event listener

Consumption message

We simulate by customizing a mockedConsumer

type mockedConsumer struct {
    count  int32
}

func (c *mockedConsumer) Consume(string) error {
    atomic.AddInt32(&c.count, 1)
    return nil
}

Start queue

Start, and then verify whether the data transmission between the producer and consumer mentioned above is successful:

func main() {
    // 创建 queue
    q := NewQueue(func() (Producer, error) {
        return newMockedProducer(), nil
    }, func() (Consumer, error) {
        return newMockedConsumer(), nil
    })
  // 启动panic了也可以确保stop被执行以清理资源
  defer q.Stop()
    // 启动
    q.Start()
}

The above is the queue implementation example of 060a1dc3d6c302. Through this core/queue framework, redis and kafka etc., which have been fully tested in different business scenarios. You can also implement your own message queue service according to the actual situation of your business.

Overall design

The overall process is as shown above:

1. All communications are channel out by 060a1dc3d6c3ac
2. Producer and Consumer can be set to match different business needs
3. Produce specific implementation of 060a1dc3d6c42d and Consume is defined by the developer, and queue is responsible for the overall process

to sum up

This article explains how channel , and how to implement a general message queue processing framework, and simply shows how to implement a message queue processing service core/queue mock You can implement a message queue processing service rocketmq

For more design and implementation articles about go-zero

project address

https://github.com/tal-tech/go-zero

Welcome to use go-zero and star support us!

WeChat Exchange Group

Follow the " practice " public enter the group get the QR code of the community group.

For the go-zero series of articles, please refer to the "Microservice Practice" public account