Advanced Redis: Principle and Application of Publish-Subscribe Mode

"Brother 65, if you made a beautiful lady as your girlfriend, how would you spread the news to your WeChat friends?"

"Then don't take some beautiful photos of your girlfriend + intimate photos, get a Jiugongge graphic message and post it in the circle of friends to hype it up and attack the single dog."

Like this scenario where Brother 65 publishes news through the circle of friends, and friends who follow Brother 65 can receive notifications is called a "publish/subscribe mechanism".

Today, I won't talk about Miss Sister, let's have a deep understanding of "Redis publish/subscribe mechanism". principles and practical application.

Redis implements publish-subscribe messaging mode through SUBSCRIBE , UNSUBSCRIBE and PUBLISH , Redis provides two modes to achieve, namely "pub/sub to channel" and "Publish\Subscribe to Pattern".

[toc]

Introduction to Redis publish and subscribe

Redis publish-subscribe (Pus/Sub) is a message communication mode: the sender publishes the message through PUBLISH SUBSCRIBE , and the subscriber subscribes to receive the message through UNSUBSCRIBE unsubscribe.

It mainly consists of three parts: "Publisher", "Subscriber", "Channel".

Publishers and subscribers belong to the client, Channel is the Redis server, publishers publish messages to channels, and subscribers who subscribe to this channel receive messages.

As shown in the figure below, three "subscribers" subscribe to the "ChannelA" channel:

At this time, the team leader publishes a message to "ChannelA", and the subscribers of this message will receive the message "Follow Code Byte, Improve Technology":

Pub/Sub in action

Not much nonsense, after knowing the basic concepts, the first step in learning a technology is to run it, and then to explore the principle, so as to achieve the state of "knowing the truth and knowing the reason".

There are two modes to implement "publish/subscribe":

• Publish and subscribe using channels;
• Publish and subscribe using patterns.

It should be noted that the publish-subscribe mechanism has nothing to do with db space. For example, when publishing on db 10, the subscribers of db0 will also receive messages.

Implemented by channel (Channel)

Three steps:

1. Subscribers subscribe to channels;
2. The publisher publishes a message to the "Channel";
3. All subscribers who subscribe to the " channel " receive the message.

Subscribers subscribe to channels

Use SUBSCRIBE channel [channel ...] to subscribe to one or more channels, O(n) time complexity, n = number of channels subscribed.

 SUBSCRIBE develop
Reading messages... (press Ctrl-C to quit)
1) "subscribe" // 消息类型
2) "develop" // 频道
3) (integer) 1 // 消息内容

执行该指令后，客户端进入订阅状态，订阅者只能使用subscribe 、 unsubscribe 、 psubscribe punsubscribe四个Directives that are "publish/subscribe".

The client "Xiao Caiji" subscribes to the "develop" channel to receive messages from the group leader, and the message response body indicates:

• Message type: subscribe, message, unsubscribe
• channel
• Message content: It has different meanings depending on the message type.

After entering the subscription, the client can receive 3 types of message replies:

1. subscribe : The feedback message of the successful subscription, the second value is the channel name of the successful subscription, and the third value is the number of channels subscribed by the current client.
2. message : The client received the message, the second value is the name of the channel that generated the message, and the third value is the content of the message.
3. unsubscribe : Indicates that a channel has been successfully unsubscribed. The second value is the corresponding channel name, and the third value is the number of channels subscribed by the current client. When this value is 0, the client will exit the subscription state, and then other commands that are not in the "pub/sub" mode can be executed. .

Publisher publishes message

The team leader uses PUBLISH channel message to publish messages to the designated "develop" channel.

 PUBLISH develop 'do job'
(integer) 1

It should be noted that the published messages will not be persistent. If there is a new "development" pretty boy subscribed after the message is published, it can only receive subsequent messages published to the channel.

A good one "Don't ask about the past, just strive for the present".

Subscriber accepts messages

Subscribers who follow the "develop" channel will receive a "do job" message.

 // 订阅 develop 频道
SUBSCRIBE develop
Reading messages... (press Ctrl-C to quit)
1) "subscribe" // 订阅频道成功
2) "develop" // 频道
3) (integer) 1
// 当发布者发布消息，订阅者读取到的消息如下
1) "message" // 接受到消息
2) "develop" // 频道名称
3) "do job" // 消息内容

unsubscribe channel

The reverse operation of the subscription, "Brother 65" shows his affection in the circle of friends every day. He can't stand it anymore, so he unsubscribes from his circle of friends.

Using the UNSUBSCRIBE command to unsubscribe from the specified "mode" will not affect the channels subscribed to by the `subscribe command.

Likewise, unsubscribe command will not affect the rules subscribed to by the psubscribe command.

Implemented by pattern (Pattern)

Next, let's look at another way to implement publish and subscribe. The following figure shows that when the "matching pattern" matches this channel, when the message publishes a message to the channel, the message will also be published to the "pattern" that matches the channel. Subscribe to this Clients of the schema will also receive the message.

smile.girl.* The pattern means "You are beautiful when you smile" pattern, and the two channels that match this pattern are smile.girls.Tina , smile.girls.maggi , respectively expressing liking "Smile Tina" And fans who like "smiling maggi".

As shown below:

Now when Tina publishes news and sends messages to smile.girls.Tina channel, in addition to subscribed smile.girls.Tina this channel's fans receive the news, this news will also be sent to subscribe smile.girl.* of the pattern (because the channel matches the pattern).

These fans are more greedy, and all the "girls who are beautiful when they smile" are paying attention. LSP~~, Brother Code is not such a person.

Using the matching pattern, with PUBLISH publishing the message to the subscription smile.girls.Tina client, the "channel" will also be compared with the pattern in the "pub/sub pattern", if If the Channel matches a pattern, the message is also published to clients subscribed to the pattern.

subscription model

The command of the subscription mode is PSUBSCRIBE , which means that the LSP subscribes to the "smile.girl.*" mode:

 PSUBSCRIBE smile.girls.*
Reading messages... (press Ctrl-C to quit)
1) "psubscribe" // 消息类型
2) "smile.girls.*"// 模式
3) (integer) 1 //订阅数

The corresponding command for reverse cancel mode subscription is PUNSUBSCRIBE smile.girl.* .

Subscribe to the "smile.girls.Tina" channel :

 SUBSCRIBE smile.girls.Tina
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "smile.girls.Tina"
3) (integer) 1

Subscribe to the "smile.girls.maggi" channel :

 SUBSCRIBE smile.girls.maggi
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "smile.girls.maggi"
3) (integer) 1

Tina posted a message, and fans who followed "smile.girls.Tina" and those who subscribed to the "smile.girls.*" mode matching the channel received the message.

Fans who follow the "smile.girls.*" mode receive a message

 PSUBSCRIBE smile.girls.*
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "smile.girls.*"
3) (integer) 1
//进入订阅状态，接收到消息
1) "pmessage" 消息类型
2) "smile.girls.*"
3) "smile.girls.Tina"
4) "love u" // 消息内容

Fans who follow "smile.girls.Tina" receive a message

 127.0.0.1:6379> SUBSCRIBE smile.girls.Tina
Reading messages... (press Ctrl-C to quit)
// 订阅成功
1) "subscribe"
2) "smile.girls.Tina"
3) (integer) 1
// 接收消息
1) "message"
2) "smile.girls.Tina"
3) "love u"

It is important to note that if a client subscribes to a pattern and channel that matches the pattern, the client will receive the message multiple times.

For example, Brother 65 subscribes to the "smile.girls.Tina" channel and the mode of "smile.girls.*", then when Tina posts news to the channel, Brother 65 will receive two ticket messages, one message type is message , one type is pmessage .

Redisson and SpringBoot in action

Official documentation: https://github.com/redisson/redisson/wiki/6.-distributed-objects/#67-topic

producer code

 /**
 * 发布消息到 Topic
 * @param message 消息
 * @return 接收消息的客户端数量
 */
public long sendMessage(String message) {
    RTopic topic = redissonClient.getTopic(CHANNEL);
    long publish = topic.publish(message);
    log.info("生产者发送消息成功，msg = {}", message);
    return publish;
}

consumer code

 public void onMessage() {
  // in other thread or JVM
  RTopic topic = redissonClient.getTopic(CHANNEL);
  topic.addListener(String.class, (channel, msg) -> {
    log.info("channel: {} 收到消息 {}.",  channel, msg);
  });
}

It should be noted that publishing messages and monitoring messages must run in different JVMs. If the same redissonClient is used to publish, it will not monitor its own messages.

Principle analysis

We know the concept of publish and subscribe through the above, and there are two modes to realize publish and subscribe. And use native instructions and Redisson for actual combat.

Next, we need to deeply understand how Redis implements the publish and subscribe mechanism, so that we know what it is and why it is.

How is the publish/subscribe of channel (Channel) implemented?

65 Brother, what data structure would you use to locate the corresponding client based on the channel?

Brother code, I think it can be implemented with a dictionary. The key of the dictionary corresponds to the channel being subscribed to, and the value of the dictionary can be a linked list, which stores all the clients subscribed to this channel.

data structure

Smart, Redis uses redis.h there is a redisServer structure to maintain each server process to represent the server state, pubsub_channels attribute is a dictionary, used to save the subscription channel information.

 struct redisServer {
  ...
  /* Pubsub */
   dict *pubsub_channels;
  ...
}

As shown in the figure below, "Mage" and "Liangzai" subscribed to "redis-channel", and "Otaku" and "LSP" subscribed to "Zhi~Teng￥Y*xiang-ri":

send message to channel

The producer calls PUBLISH channel messsage to send a message. The program first locates the bucket where the key of the dictionary is located from pubsub_channels according to the channel, and then sends the message to all clients in the value list corresponding to this key.

unsubscribe channel

UNSUBSCRIBE The command can unsubscribe from the specified channel: drop and dictionary operation, find the follow list according to the key, traverse the list, delete the client, so that the message will not be sent to the client.

How is the publish/subscribe pattern implemented?

Next, let's continue to look at the principle of publish and subscribe based on pattern implementation...

When using PUBLISH to publish a message to a channel, not only all clients subscribed to the channel will receive the message, but clients matching this pattern will also receive the message.

The source code is defined in the redisServer.pubsub_patterns attribute in the server.h file.

 struct redisServer {
  ...
  /* A dict of pubsub_patterns */
    dict *pubsub_patterns;
  ...
}

It is also a dict dictionary type, the key corresponds to the "pattern" pattern, and the value is a linked list type structure: list *clients which contains a list of clients matching each pattern.

When the PSUBSCRIBE smile.girls.* command is executed, the pubsubSubscribePattern method will be executed.

Here I will share how to locate the key source code, publish and subscribe, we can search pubsub according to our experience, and then we can retrieve pubsub.c :

The Redis source code debugged by code brother using CLion comes from the same IDEA as our Java development, so the shortcut keys are the same, and then use Command + F12 pop-up method search, find pubsubSubscribePattern subscribe mode method.

The method parameters respectively indicate the client client c that pays attention to the pattern, and the pattern the client wants to pay attention to . The main logic of the method is as follows:

1. listSearchKey(c->pubsub_patterns,pattern) : According to the pattern, look up whether the key of this pattern already exists from the redisServer.pubsub_patterns dictionary. If it exists, call addReplyPubsubPatSubscribed to notify the client that it has subscribed, otherwise continue to execute the following logic.
2. dictFind(server.pubsub_patterns,pattern) ： pattern字典server.pubsub_patterns找到dictEntry 哈希桶，为空就调用---e9fd502b0dccb9f4544e1d727d498559--- listCreate() list *clients and put it in the dictionary, key = pattern, value = list *clients linked list.
3. If the hash bucket is not empty, then add the current client client *c to the tail node of list *clients linked list.

Therefore, the publish and subscribe mode implemented by the mode also uses the dictionary to save the relationship between the mode and the client, as shown in the following figure:

When using PUBLISH to publish a message, in addition to publishing to the client subscribed to channel , the channel will also be matched with the pubsub_patterns dictionary to find the matching pattern key The linked list of clients in the value of , and perform message sending.

Unsubscribe mode

Use the PUNSUBSCRIBE command to unsubscribe from the specified mode. This command performs the inverse operation of the subscription mode: find the client list from the pubsub_patterns dictionary according to the mode, and traverse the list to connect the current client delete.

Summarize

The Redis publish and subscribe function is mainly implemented through the following commands:

• subscribe channel [channel ...] : Subscribe to one or more channels;
• unsubscribe channel Unsubscribe from the specified channel;
• publish channel message Send a message to the specified channel;
• psubscribe pattern Subscription specified mode;
• punsubscribe pattern Unsubscribe from the specified mode.

Pub/Sub is database agnostic, for example, if you publish on DB0 DB1 , subscribers of ---3a4d2fc7e8f89ec9db20c8b797957819--- will also receive it.

The publish and subscribe information based on the channel implementation is stored by the server process redisServer.pubsub_channels dictionary, key = the subscribed channel, and value is the linked list of all clients subscribed to the channel.

When a message is published to the channel, iterate over the dictionary to get all clients and send the message to the channel's clients.

The information of the publish and subscribe based on the pattern is stored in the dictionary pubsub_patterns , key = pattern, value is the client list.

When a message is published to a channel, in addition to the client subscribed to the channel, all clients subscribed to the pattern matching the channel will also receive the message.

scenes to be used

Having said so much, what scenarios can Redis publish and subscribe in?

Sentinel communication

In the Sentinel cluster, each Sentinel node uses Pub/Sub to publish and subscribe to realize the mutual discovery of each other and the Slave between Sentinels. For details, click -> "Sentinel Cluster Principles and Those Things" .

After the sentinel establishes communication with the master, it uses the master to provide a publish/subscribe mechanism to publish its own information in __sentinel__:hello , such as height and weight, whether it is single, IP, port..., and subscribe to this channel to get the information of other sentinels. , thus realizing the communication between sentinels.

message queue

Previously, "Code Brother" shared with you how to use Redis List and Stream to implement message queues .

We can also use Redis to publish and subscribe to implement lightweight and simple MQ functions to achieve upstream and downstream decoupling. It should be noted that the messages of Redis publish and subscribe will not be persisted, so newly subscribed clients will not receive historical messages.

It also does not support the ACK mechanism, so the current business cannot tolerate these shortcomings, then use a professional message queue, and if you can tolerate it, you can enjoy the advantages of Redis fast.

Finally, can you call me "Pretty Boy" in the comment area? In order to write this article, Brother Code read a lot of girls who are so beautiful when they smile before writing it. It is not easy to be original.

Friends, like, share, favorite and support me.

References

1. Redis design and implementation

2. https://redisbook.readthedocs.io/en/latest/feature/pubsub.html