Stop using Redis List to implement message queues, Stream is designed for queues

Last time I mentioned that uses Redis List to implement message queue has many limitations, such as:

• There is no good ACK mechanism;
• There is no concept of ConsumerGroup consumption group;
• Messages pile up.
• List is a linear structure. If you want to query the specified data, you need to traverse the entire list;

Stream is a data type specially designed for message queues introduced by Redis 5.0. Stream is an ordered queue containing 0 or more elements, which are arranged in order according to the size of the ID.

It implements most of the functions of the message queue:

• Serialized generation of message ID;
• message traversal;
• Blocking and non-blocking reads of messages;
• Consumer Groups Consumer Group;
• ACK acknowledgement mechanism.
• Multicast is supported.

Provides many message queue operation commands, and draws on the concept of Kafka's Consumer Groups to provide consumer group functions.

also provides message persistence and master-slave replication mechanisms. Clients can access data at any time, and can remember the access location of each client to ensure that messages are not lost.

Don't talk nonsense, let's see how to use it first, see the official website documentation: https://redis.io/topics/streams-intro

XADD: insert message

"The Yunlan Sect disciples obeyed and killed Xiao Yan!"

When the last word of Yunshan fell, the tense atmosphere that filled the air was suddenly broken, and the wings of many Yunlanzong elders floating in midair fluttered, and they streaked across the sky, chasing and killing Xiao Yan.

Yunshan used the following command to insert the command "Chase Xiao Yan" into the queue, and let the elder lead the children to execute it.

XADD 云岚宗 * task kill name 萧炎
"1645936602161-0"

Each element in the Stream consists of key-value , and different elements can contain different numbers of key-value .

The syntax of this command is as follows:

XADD streamName id field value [field value ...]

The "*" after the message queue name means that Redis can automatically generate a unique ID for the inserted message, of course, you can also define it yourself.

The message ID consists of two parts:

• Timestamp in the current millisecond;
• sequential numbering. Starting from 0, it is used to distinguish multiple commands generated at the same time.

By associating element IDs with times, and enforcing that new element IDs must be greater than old element IDs, Redis logically turns streams into an append-only data structure.

This feature is very important for users of message queues and event systems using streams:

Users can be sure that new messages and events will only appear after existing ones, just as new events always occur after existing ones in the real world, and everything is in order.

XREAD: read message

Yunling Laogou uses the following command to receive Yunshan's commands:

XREAD COUNT 1 BLOCK 0 STREAMS 云岚宗 0-0
1) 1) "\xe4\xba\x91\xe5\xb2\x9a\xe5\xae\x97"
   2) 1) 1) "1645936602161-0"
         2) 1) "task"
            2) "kill"
            3) "name"
            4) "萧炎" # 萧炎

XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] ID [ID ...]

This instruction can read multiple streams at the same time. The corresponding meaning of each method is as follows:

• COUNT: Indicates the maximum number of elements read in each stream;
• BLOCK: Block reading, when there is no message in the message queue, block waiting, 0 means infinite waiting, the unit is milliseconds.
• ID: message ID, can specify an ID when reading a message, and start reading from the next message of this ID, 0-0 means read from the first element.

If you want to use XREAD for sequential consumption, you need to remember the returned message ID after each reading. The next time you call XREAD, you can pass the last returned message ID as a parameter to the next call to continue consuming subsequent messages.

Sect Master Yunyun, I just arrived at Yunlan Sect today, and I will not receive any historical news. I just want to receive the news that was published through XADD from the moment I blocked and waited with XREAD. What should I do?
Just run the "$" mantra. The last "$" symbol of the mantra means to read the latest blocking message. If you can't read it, you will die.

During the waiting process, other elders add messages to the queue, and they will be read immediately.

XREAD COUNT 1 BLOCK 0 STREAMS 云岚宗 $

Is it so easy to implement a message queue? What about the ACK mechanism?
This is just an appetizer. The data read through XREAD is not actually deleted. It will be read again when the XREAD COUNT 2 BLOCK 0 STREAMS Yunlanzong 0-0 command is re-executed.

So we also need the ACK mechanism,

Next, we come to a real message queue.

ConsumerGroup

The ConsumerGroup (consumer group) of Redis Stream allows users to logically divide a stream into multiple different streams and let the consumers of the ConsumerGroup handle it.

It is a robust persistent message queue that supports multicast. Redis Stream borrows design from Kafka.

The high availability of Stream is based on master-slave replication. It is no different from the replication mechanism of other data structures. That is to say, Stream can support high availability in Sentinel and Cluster cluster environments.

• The structure of Redis Stream is shown in the figure above. There is a message linked list, each message has a unique ID and corresponding content;
• message persistence;
• The status of each consumer group is independent, it does not affect the same stream message will be consumed by all consumer groups;
• A consumer group can be composed of multiple consumers. The consumers are in a competitive relationship. Any consumer who reads the message will move the last_deliverd_id forward;
• Each consumer has a pending_ids variable, which is used to record the messages that the current consumer has read but not acked. It is used to ensure that the message is consumed by the client at least once.

The message queue implemented by the consumer group mainly involves the following three instructions:

• XGROUP is used to create, destroy and manage consumer groups.
• XREADGROUP is used to read from a stream via a consumer group.
• XACK is a command that allows consumers to mark pending messages as properly processed.

Create a consumer group

Stream creates a consumer group (Consumer Group) through the XGROUP CREATE instruction, and needs to pass the start message ID parameter to initialize last_delivered_id variable.

We use XADD to insert some messages into the bossStream queue:

XADD bossStream * name zhangsan age 26
XADD bossStream * name lisi age 2
XADD bossStream * name bigold age 40

The following command creates two consumer groups "Qinglongmen" and "six doors" for the message queue named bossStream.

# 语法如下
# XGROUP CREATE stream group start_id
XGROUP CREATE bossStream 青龙门 0-0 MKSTREAM
XGROUP CREATE bossStream 六扇门 0-0 MKSTREAM

• stream: specifies the name of the queue;
• group: specify the name of the consumer group;
• start_id: Specifies the starting ID of the consumer group in the Stream, which determines which ID the consumer group starts to read messages from. 0-0 starts reading from the first one, and $ means reading from the last one backward, only receiving new news.
• MKSTREAM: By default, the XGROUP CREATE command returns an error when the target stream does not exist. Streams can be created automatically using the optional MKSTREAM subcommand as the last argument after.

read message

Let bossStream consumer1

XREADGROUP GROUP 青龙门 consumer1 COUNT 1 BLOCK 0 STREAMS bossStream >
1) 1) "bossStream"
   2) 1) 1) "1645957821396-0"
         2) 1) "name"
            2) "zhangsan"
            3) "age"
            4) "26"

The syntax is as follows:

XREADGROUP GROUP groupName consumerName [COUNT n] [BLOCK ms] STREAMS streamName [stream ...] id [id ...]

[] indicates optional parameters. This command is similar to XREAD , but the difference is that the option GROUP groupName consumerName is added.

The two parameters of this option are used to specify the consumer group to be read and the consumer responsible for processing the message.

in:

• > : The last parameter of the command is > , which means to start reading from the message that has not been consumed;
• BLOCK: blocking read;

knocking on the blackboard

If a message in the message queue is consumed by a consumer in a consumer group, the message will not be read by other consumers in this consumer group.

For example consumer2 performs a read operation:

XREADGROUP GROUP 青龙门 consumer2 COUNT 1 BLOCK 0 STREAMS bossStream >
1) 1) "bossStream"
   2) 1) 1) "1645957838700-0"
         2) 1) "name"
            2) "lisi"
            3) "age"
            4) "2"

consumer2 can no longer read zhangsan , but read the next lisi because this message has been read by consumer1 .

Another purpose of using consumers is to allow multiple consumers in the group to share the read message, that is, each consumer reads part of the message, so as to achieve load balancing.

For example, a consumer group has three consumers C1, C2, C3 and a stream containing messages 1, 2, 3, 4, 5, 6, 7:

XPENDING View read and unacknowledged messages

In order to ensure that consumers can still read messages after a failure during consumption or a shutdown and restart, there is a queue (pending List) inside Stream to store the messages that each consumer reads but has not yet executed ACK.

If the consumer uses XREADGROUP GROUP groupName consumerName to read the message, but does not send the XACK command to the Stream, the message remains.

For example, check the unconfirmed message information that each consumer in the consumer group "Qinglongmen" in bossStream has read:

XPENDING bossStream 青龙门
1) (integer) 2
2) "1645957821396-0"
3) "1645957838700-0"
4) 1) 1) "consumer1"
      2) "1"
   2) 1) "consumer2"
      2) "1"

1. 1) The number of unconfirmed messages;
2. 2) ~ 3) The minimum and maximum IDs of messages read by all consumers in Qinglongmen;

To see what data consumer1 read, use the following command:

XPENDING bossStream 青龙门 - + 10 consumer1
1) 1) "1645957821396-0"
   2) "consumer1"
   3) (integer) 3758384
   4) (integer) 1

ACK acknowledgement

So when the message is received and the consumption is successful, we need to manually ACK to notify Streams, and the message will be deleted. The command is as follows:

XACK bossStream 青龙门 1645957821396-0 1645957838700-0
(integer) 2

The syntax is as follows:

XACK key group-key ID [ID ...]

Consumption confirmation increases the reliability of the message. Generally, after the business processing is completed, it is necessary to execute ack to confirm that the message has been consumed. The execution of the entire process is shown in the following figure:

Use Redisson in action

Add dependencies using maven

<dependency>
  <groupId>org.redisson</groupId>
  <artifactId>redisson-spring-boot-starter</artifactId>
  <version>3.16.7</version>
</dependency>

Add Redis configuration, code brother's Redis does not have a password configured, you can configure it according to the actual situation.

spring:
  application:
    name: redission
  redis:
    host: 127.0.0.1
    port: 6379
    ssl: false

@Slf4j
@Service
public class QueueService {
    @Autowired
    private RedissonClient redissonClient;
  
    /**
     * 发送消息到队列
     *
     * @param message
     */
    public void sendMessage(String message) {
        RStream<String, String> stream = redissonClient.getStream("sensor#4921");
        stream.add("speed", "19");
        stream.add("velocity", "39%");
        stream.add("temperature", "10C");
    }
  
    /**
     * 消费者消费消息
     *
     * @param message
     */
    public void consumerMessage(String message) {
        RStream<String, String> stream = redissonClient.getStream("sensor#4921");
        stream.createGroup("sensors_data", StreamMessageId.ALL);
        Map<StreamMessageId, Map<String, String>> messages = stream.readGroup("sensors_data", "consumer_1");
        for (Map.Entry<StreamMessageId, Map<String, String>> entry : messages.entrySet()) {
          Map<String, String> msg = entry.getValue();
          System.out.println(msg);
          stream.ack("sensors_data", entry.getKey());
        }
    }
}

If readers and friends have something to gain after reading, like, favorite and share , thank you for your support. altruistic self-interested dawn people.

Reference link:

https://blog.51cto.com/u_15239532/2835962

https://redis.io/topics/streams-intro

https://redisson.org/articles/redis-streams-for-java.html