Can Redis use List to implement message queue to ensure message reliability?

A necessary middleware in distributed systems is message queues. Through message queues, we can asynchronously decouple services, reduce traffic peaks, and achieve eventual consistency.

At present, there are already RabbitMQ、RochetMQ、ActiveMQ、Kafka and so on on the market. Some people will ask: "Is Redis suitable for message queues?"

Before answering this question, let's think about the essence:

• What features do message queues provide?
• How does Redis implement message queues? Does it meet the access requirements?

Today, Code Brother will take you step by step to analyze the implementation principle of using Redis List as a message queue based on the characteristics of message queues, and share how to integrate SpringBoot and Redission into the project.

What is a message queue

Message Queuing is an asynchronous inter-service communication method suitable for distributed and microservice architectures. Messages are stored on the queue until they are processed and deleted.

Each message can only be processed once by a user. Message queues can be used to separate heavy-duty processing, buffer or batch work, and relieve peak workloads.

• Producer: The message producer, responsible for generating and sending messages to the Broker;
• Broker: message processing center. Responsible for message storage, confirmation, retry, etc., generally including multiple queues;
• Consumer: message consumer, responsible for obtaining messages from the Broker and processing them accordingly;

What are the usage scenarios of message queues?

In practical applications, message queues include the following four scenarios:

• Application coupling: The sender and receiver systems do not need to know both sides, but only need to know the message. Multiple applications process the same message through the message queue to avoid the failure of the entire process due to the failure of calling the interface;
• Asynchronous processing: Multiple applications process the same message in the message queue, and the messages are processed concurrently between applications, which reduces processing time compared to serial processing;
• Current limiting and peak shaving: widely used in spike or panic buying activities to avoid the situation where the application system hangs due to excessive traffic;
• Message-driven system: The system is divided into message queues, message producers, and message consumers. The producer is responsible for generating messages, and the consumers (there may be multiple) are responsible for processing the messages;

What characteristics does a message queue satisfy

message ordering

Messages are processed asynchronously, but consumers need to consume in the order in which messages are sent by producers to avoid situations where messages sent later are processed first.

Duplicate message handling

The producer may retransmit messages due to network problems and consumers may receive multiple duplicate messages.

Repeating the same message multiple times may cause a business logic to be executed multiple times. It is necessary to ensure how to avoid the problem of repeated consumption.

Reliability

One-time guaranteed message delivery. If the receiver is unavailable when the message is sent, Message Queuing holds the message until it is successfully delivered.

When the consumer restarts, it can continue to read messages for processing to prevent message omission.

List implements message queue

Redis's list (List) is a linear ordered structure, which can store elements in the order in which they are pushed into the list, which can meet the requirements of "first-in, first-out". These elements can be either text data or binary data.

LPUSH

The producer uses LPUSH key element[element...] to insert the message into the head of the queue. If the key does not exist, it will create an empty queue and insert the message.

As follows, the producer inserts "Java", "Code Byte" and "Go" into the queue queue, and the return value indicates the number of messages inserted into the queue.

> LPUSH queue Java 码哥字节 Go
(integer) 3

RPOP

The consumer uses RPOP key read the messages in the queue in turn, first in first out, so "Java" will read the consumption first:

> RPOP queue
"Java"
> RPOP queue
"码哥字节"
> RPOP queue
"Go"

real-time consumption

65 Brother: Is it so simple to achieve?

Don't be too happy, LPUSH、RPOP has a performance risk. When the producer inserts data into the queue, the List will not actively notify the consumer to consume it in time.

We need to write a while(true) to call RPOP command continuously. When there is a new message, it will return a message, otherwise it will return empty.

The program needs to continuously poll and determine whether it is empty before executing the consumption logic, which will cause the consumer to keep calling RPOP command to occupy CPU resource even if no new messages are written to the queue.

65 Brother: How to avoid CPU performance loss caused by loop calls?

Redis provides the command BLPOP、BRPOP to block reading. consumer automatically blocks when there is no data in the read queue. Until a new message is written to the queue, it will continue to read new messages and execute business logic.

BRPOP queue 0

Parameter 0 means unlimited waiting time for blocking

Repeat consumption

• The message queue generates a "global ID" for each message;
• The producer creates a "global ID" for each message, and the consumer records the ID of a processed message to determine whether it is repeated.

In fact, this is idempotent. For the same message, the result of processing one time after the consumer receives it is consistent with the result of multiple times.

message reliability

65 Brother: The consumer reads a message from the List, and the message is down during the processing of the message, which will result in the message not being processed, but the data has not been saved in the List. What should I do?

The essence is that the consumer crashes while processing the message, and the message cannot be restored anymore, lacking a message confirmation mechanism.

Redis provides RPOPLPUSH, BRPOPLPUSH (blocking) two instructions, meaning that the message is read from List and copied to another List (backup), and it is an atomic operation.

We can implement the message confirmation mechanism by deleting the queue message after the business process is correctly processed. If the machine crashes while processing the message, read the message processing from the backup list after restarting.

LPUSH redisMQ 公众号 码哥字节
BRPOPLPUSH redisMQ redisMQBack

The producer uses LPUSH to insert the message into the redisMQ queue, the consumer uses BRPOPLPUSH read the message "Official Account", and the message will be inserted into the "redisMQBack" queue.

If the consumption is successful, the message of "redisMQBack" can be deleted. If there is an exception, you can continue to read the message from "redisMQBack" again for processing.

should be noted that if the producer message is sent quickly, and the consumer's processing speed is slow, it will lead to the accumulation of messages, which will put too much pressure on the memory of Redis.

Redission in action

In Java, we can use the API encapsulated by Redission to quickly implement queues. Next, Code Brother will show you how to integrate and practice based on SpringBoot 2.1.4 version.

For detailed API documentation, please refer to: https://github.com/redisson/redisson/wiki/7.-Distributed-collections

add dependencies

<dependency>
  <groupId>org.redisson</groupId>
  <artifactId>redisson-spring-boot-starter</artifactId>
  <version>3.16.7</version>
</dependency>

Add Redis configuration, code brother's Redis does not have a password configured, you can configure it according to the actual situation.

spring:
  application:
    name: redission
  redis:
    host: 127.0.0.1
    port: 6379
    ssl: false

Java code in action

RBlockingDeque inherits java.util.concurrent.BlockingDeque , and we can choose the appropriate API to implement the business logic according to the interface document.

The main method is as follows

Code brother uses double-ended queue as an example

@Slf4j
@Service
public class QueueService {

    @Autowired
    private RedissonClient redissonClient;

    private static final String REDIS_MQ = "redisMQ";

    /**
     * 发送消息到队列头部
     *
     * @param message
     */
    public void sendMessage(String message) {
        RBlockingDeque<String> blockingDeque = redissonClient.getBlockingDeque(REDIS_MQ);

        try {
            blockingDeque.putFirst(message);
            log.info("将消息: {} 插入到队列。", message);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    /**
     * 从队列尾部阻塞读取消息，若没有消息，线程就会阻塞等待新消息插入，防止 CPU 空转
     */
    public void onMessage() {
        RBlockingDeque<String> blockingDeque = redissonClient.getBlockingDeque(REDIS_MQ);
        while (true) {
            try {
                String message = blockingDeque.takeLast();
                log.info("从队列 {} 中读取到消息：{}.", REDIS_MQ, message);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }

        }
    }

unit test

@RunWith(SpringRunner.class)
@SpringBootTest(classes = RedissionApplication.class)
public class RedissionApplicationTests {

    @Autowired
    private QueueService queueService;

    @Test
    public void testQueue() throws InterruptedException {
        new Thread(() -> {
            for (int i = 0; i < 1000; i++) {
                queueService.sendMessage("消息" + i);
            }
        }).start();

        new Thread(() -> queueService.onMessage()).start();

        Thread.currentThread().join();
    }


}

Summarize

You can use the List data structure to implement message queues to satisfy first-in, first-out. In order to achieve message reliability, Redis provides the BRPOPLPUSH command as a solution.

Redis is a very lightweight key-value database. To deploy a Redis instance is to start a process and deploy a Redis cluster, that is, to deploy multiple Redis instances.

When Kafka and RabbitMQ are deployed, additional components are involved. For example, the operation of Kafka requires the deployment of ZooKeeper. Compared to Redis, Kafka and RabbitMQ are generally considered to be heavyweight message queues.

should be noted that we need to avoid the accumulation of messages caused by too fast producers and too slow consumers to occupy the memory of Redis.

Using Redis as a message queue when the amount of messages is small, it can bring us high-performance message reading and writing, which seems to be a good message queue solution.