IM development dry goods sharing: Talking about the best practices of offline messages and historical messages in the IM system

This article was originally shared by the Rongyun technical team. The original title is "IM message data storage structure design", and the content has been revised.

1 Introduction

In today's mobile Internet era, IM products have become an indispensable part of our lives. Like WeChat, DingTalk, QQ, etc. are typical social products with IM as the core function. In addition, there are some applications, although the IM function is not the core, but the IM capability is also an extremely important part of the entire application, such as online games, e-commerce live broadcast and other applications.

Under the premise of more and more extensive application scenarios of IM technology, it is more and more necessary to learn and master instant messaging IM technology.

In the huge technical system of IM, the message system is undoubtedly the core, and in the message system, the most critical part is the distribution and storage of messages, and offline messages and historical messages are unavoidable technical points in this key link.

Based on the technical practice of the IM message system, this article will share the correct understanding of offline messages and historical messages, as well as specific technical cooperation and practice, hoping to bring best practice inspiration for your offline message and historical message technology design.

study Exchange:

• Introductory article on mobile IM development: "One entry is enough for beginners: developing mobile IM from scratch"
• Open source IM framework source code: https://github.com/JackJiang2011/MobileIMSDK

(This article is simultaneously published at: http://www.52im.net/thread-3887-1-1.html )

2. Related articles

Technical related articles:

"What is the reliability of IM systems? 》
"Optimization Practice of Online and Offline Chat Data Synchronization Mechanism of Xianyu IM"
"Reliable Delivery Optimization Practice of Xianyu Billion-level IM Message System"
"A set of IM architecture technology dry goods for hundreds of millions of users (Part 2): reliability, orderliness, weak network optimization, etc."
"Implementation of IM Message Delivery Guarantee Mechanism (2): Ensuring Reliable Delivery of Offline Messages"
"How do I solve a large number of offline messages that cause the client to freeze"

Other articles shared by the Rongyun technical team:

"Technical Practice of Network Link Keep Alive for Rongyun Android IM Products"
"Comprehensive Revealing the Reliable Delivery Mechanism of Billion-Level IM Messages"
"Decryption Rongyun IM Product Chat Message ID Generation Strategy"
"Thinking and Practice of Ten Thousand People Chat Message Delivery Scheme"
"Optimization practice of real-time audio and video first frame display time based on WebRTC"
"Rongyun IM Technology Sharing: Thinking and Practice of the Message Delivery Scheme for Thousands of People Chat"

3. General practice of IM message delivery

In a common IM message system, the following technical ideas are probably used for real-time messages, offline messages, and historical messages.

For online users: the message will be sent directly to the online receiver in real time. After the message is sent, the server will not store the message on the ground.

For offline users: the server will store the messages in the offline library. When the user logs in, the offline messages will be pulled from the offline library, and then the server will delete the offline messages.

The disadvantage of this implementation is that the message is not persistent, so that the message cannot support message roaming, which reduces the reliability of the message.

(PS: Actually, this is not actually a disadvantage, because it is not necessary to store historical messages in some scenarios, and the so-called message roaming capability is not necessary, such as WeChat.)

In the message system we designed, as long as the server receives the message from the sender, it will also store the message in the offline database and the historical message database while forwarding it to the receiver, and the landing of the historical message is also Can support message roaming and other related functions.

4. What are offline messages and historical messages?

Regarding offline messages and historical messages, technically, we define them this way.

1) Offline message:

Offline messages are messages received by users (that is, receivers) during the offline process. Most of these messages are messages that users are more concerned about and have certain timeliness.

According to our system experience, our offline messages only save the latest seven days of messages by default.

The user (that is, the receiver) will obtain the full amount of these offline messages after logging in next time, and then display the offline message UI on the client side according to the chat session (such as displaying an unread message bubble, etc.).

(PS: The possibility of a user being offline is actually technically composed of many situations, such as the other party being offline, the other party’s network being disconnected, the other party’s mobile phone crashing, the server sending an error, etc. Strictly speaking – as long as Messages that cannot be sent in real time are considered "offline messages".)

2) Historical news:

The history message stores all the chat messages of the user, including sent messages and received messages.

When the client obtains historical messages, it is usually paginated according to the session.

Based on our system experience, we design the storage time of historical messages to be half a year by default. Of course, this time can be determined according to the actual product operation rules, and there is no hard and fast rule.

5. IM message sending and storage process

The following is the overall message sending and storage process of our system:

As shown in the figure above: When a user sends a chat message to the server, it will first enter the message system, and the message system will distribute and store the message.

During this process: For online receivers, direct push messages will be selected. However, when the receiver is offline or the message push fails, there are other ways to obtain messages. For example, the receiver will actively pull unreceived messages from the server. But it is unknown when the receiver pulls the message from the server and where to pull it, so the meaning of storing the message in the offline library is here.

In the process of offline storage of the message system, in order not to affect the stability of the whole system, we use the MQ message queue for IO decoupling, so the chat messages are actually stored in the offline library asynchronously (the slow IO solution is performed through MQ). Even, this is actually the usual practice).

After the message is distributed: the message service will synchronize a piece of message data to the historical message service, and the historical message service will also store the message on the ground.

For new client devices: there will be a need for synchronous messages (so-called message roaming capability), which is the main role of historical messages. In the historical message database, the client can pull the full historical messages of any session.

6. Differences in storage logic between IM offline messages and historical messages

6.1 Overview It can be clearly seen from the above figure:

1) For offline messages, we use Redis as the storage medium;
2) We use HBase for historical news.
As for why we choose different storage media, we actually consider the different business scenarios and read-write modes of offline messages and historical messages.

Let's focus on the difference between offline message and historical message storage.

6.2 Offline message storage mode - "diffusion write"
For the storage mode of offline messages, we use diffusion writing.

As shown in the image above: each user has their own separate inbox and outbox:

1) The inbox stores all the messages that need to be synchronized to the receiver;
2) All messages sent by the sender are stored in the outbox.
Take a single chat as an example: in a two-person conversation in a chat, the message will be written twice, that is, the sender's outbox and the receiver's inbox.

In the group scenario, the writing will be amplified (diffused) even more. If there are N people in the group, a group message will be diffused and written N times.

To summarize:

1) The advantage of diffusion writing is that the logic of the receiving end will be very clear and simple, and it only needs to be read from the inbox once, which greatly reduces the reading pressure required for synchronizing messages;
2) The disadvantage of diffuse writing is that the writing will be exponentially amplified, especially for the group scenario.

6.3 Historical message storage mode - "diffusion reading"
For the storage mode of historical messages, we use diffusion reading.

Because in historical messages, each session saves the full amount of messages for the entire session. In diffuse read mode, each session's messages are stored only once.

Compared with the diffuse write mode, the advantages and disadvantages of diffuse read are as follows:

1) The advantage is that the number of writes is greatly reduced, especially for group messages, which only need to be stored once;
2) The disadvantage is that it is very complicated and inefficient for the receiving end to receive messages, because in this mode, the client can only synchronize once per session if it wants to pull all messages, the reading will be amplified, and many invalidations may occur. read, as some conversations may have no new messages at all.

6.4 Summary In the application scenario of IM, the message synchronization model of diffusion writing is usually used. One message is generated, but it may be read multiple times, which is a typical scenario of more reading and less writing.

An optimized IM system must balance reading and writing pressure in design, and avoid reading or writing from reaching the ceiling in any dimension.

Of course, this mode of diffusion writing also has its drawbacks. For example, if there are 10,000 people, a message will be written 10,000 times.

To sum up, we need to make corresponding design choices according to our own business scenarios. Taking our IM system as an example, we choose the combination mode of write diffusion and read diffusion according to different scenarios of offline and historical messages. The right one is the best, there is no need to stick to the theory.

7. Pull message logic of IM client

7.1 Offline message pulling logic For the IM client, the offline message acquisition is for its entire offline message, including all conversations (to put it bluntly, it is to pull all the unpaid messages during the offline process when going online). offline messages).

The acquisition of offline messages is a top-down method (in time series). Our experience is to acquire 200 messages at a time (PS: If there are too many offline messages, they will be pulled multiple times by page. Pulling 1 "time" can be understood as pull 1 "page").

In the signaling for the client to pull the offline message, the maximum timestamp of the message currently cached by the client needs to be included.

From the figure in the previous section, we should know that the offline message is stored in a linear structure (referring to chronological order), and the server will look down the offline message according to this timestamp. When the app is reinstalled or newly installed, the client's "Maximum timestamp of messages cached by the client" can be uploaded with 0.

The server will also cache the timestamp of the last message pulled by the client, and then decide where to start pulling according to the business scenario, client type and other factors. The corresponding flag bit tells the client to continue pulling, and the client pulls in a loop until all offline messages are pulled.

7.2 History message pull logic The history message is usually obtained for a single session.

During the pull process, two parameters need to be submitted to the server:

1) The ID of the other party (if it is a single chat, it is the UserID of the other party, if it is a group, it is the group ID);
2) The timestamp of the foremost message of the current session (that is, the timestamp of the oldest message of the current session).
According to these two parameters, the server can locate the session of the client, and then obtain 20 historical messages at a time.

The pull sequence of messages is bottom-up (that is, the time sequence is reversed), that is, flipping from the back to the front. As long as there is a message, the client can always scroll forward and manually trigger the acquisition of historical messages of the session.

The above pull logic usually corresponds to pull-down or click "load more" in the IM interface function, such as this:

8. Summary of this article

This article mainly shares the correctness of offline messages and historical messages in IM, including the difference between offline messages and historical messages, as well as their best practices in terms of storage, distribution, and pull logic. If you have any objection to the content of the article, please leave a message for discussion.

9. References

[1] A set of practice sharing of mobile IM architecture design for massive online users (including detailed pictures and texts)
[2] A set of original distributed instant messaging (IM) system theoretical architecture scheme
[3] From zero to excellence: the evolution of the technical architecture of the JD.com customer service instant messaging system
[4] A set of IM architecture technology dry goods for hundreds of millions of users (Part 1): overall architecture, service splitting, etc.
[5] Architecture Evolution of Xianyu Billion-level IM Messaging System
[6] Reliable delivery optimization practice of Xianyu billion-level IM message system
[7] Timeliness optimization practice of Xianyu billion-level IM message system
[8] Based on practice: Summary of technical points of a small-scale IM system with millions of messages
[9] Implementation of IM message delivery guarantee mechanism (1): ensure reliable delivery of online real-time messages
[10] Understand the "reliability" and "consistency" of IM messages, and discuss solutions
[11] Introduction to zero-based IM development (1): What is an IM system?

(This article is simultaneously published at: http://www.52im.net/thread-3887-1-1.html )