Based on practice: a summary of the technical points of a small-scale IM system with a million messages

This article is shared by the public account "Background Technology Collection". The original title is "Based on practice, design a million-level high-availability & high-reliability IM messaging system", the original link is at the end of the article. Due to the fact that there are many errors and inaccurate content in the original text, there are a lot of revisions and changes.

1 Introduction

Hello, everyone, I am the blogger "YiMeiJian" of the official account "Background Technology Exchange".

I have been engaged in back-end development for more than 3 years, and one of the projects that impressed me the most was the IM messaging system that I took over from the architect two years ago.

The content of this article will start from the developer’s perspective (mainly my own development experience), focusing on the project background, business requirements, technical principles, development plans and other topics, and analyze it step by step with everyone: design a set of millions of messages Technical points that need to be paid attention to in the design of small-scale IM system architecture.

study Exchange:

• 5 groups for instant messaging/push technology development and communication: 215477170 [recommended]
• Introduction to Mobile IM Development: "One entry is enough for novices: Develop mobile IM from scratch"
• Open source IM framework source code: https://github.com/JackJiang2011/MobileIMSDK

(This article was published synchronously at: http://www.52im.net/thread-3752-1-1.html)

2. Project background

When we observe carefully, we can find that any type of Internet service in our lives has an IM system.

for example:

1) Basic service category-Tencent News (comment news);
2) Business applications-Dingding (approval workflow notification);
3) Communication and entertainment-QQ/WeChat (private chat group chat & discussion group & circle of friends);
4) Internet self-media-Douyin Kuaishou (Like and reward notification).

In these numerous Internet ecological products, the instant messaging system, as the underlying capability, has always played a vital role in ensuring normal business and optimizing user experience.

Therefore, in today's Internet products, instant messaging technology is no longer limited to the traditional IM chat tool itself, it has already been embedded in various forms of Internet applications in tangible or intangible ways. IM technology (or instant messaging technology) is indeed indispensable and indispensable for many developers.

3. System capabilities
A typical IM system usually needs to meet four capabilities: high reliability, high availability, real-time and orderly.

I will not expand on these concepts in detail. If you are a beginner in IM development, you can read the following articles in detail:

"Introduction to zero-based IM development (1): What is an IM system? 》
"Introduction to zero-based IM development (2): What is the real-time nature of the IM system? 》
"Introduction to zero-based IM development (3): What is the reliability of the IM system? 》
"Introduction to zero-based IM development (4): What is the message timing consistency of the IM system? 》

4. Architecture design

For my project, the main points of architecture design are:

1) Microservices: split into user microservices & message connection services & message business services;
2) Storage architecture: compatible with performance and resource overhead, select reids&mysql;
3) High availability: can support high concurrency scenarios, choose websocket provided by Spring;
4) Support multi-terminal message synchronization: app terminal, web terminal, WeChat official account, small program message;
5) Support online and offline messaging scenarios.

The business architecture diagram is mainly like this:

The hierarchical architecture of technical modules is roughly like this:

5. Key points of message storage technology

5.1 Understand read diffusion and write diffusion
5.1.1) Basic concepts:

Let us give an example to illustrate what is read diffusion and what is write diffusion:

A group chat "love each other in a family", members: father, mother, brother, sister and me (5 people in total).

Because you recently had a girlfriend, you sent a message "I'm out of order" to the group, so naturally I hope that all four relatives, including parents, brothers and sisters, can receive it.

Under normal logic, the process of sending group chat messages should look like this:

1) Traverse the members of the group chat and send messages;
2) Query the online status of each member;
3) Store offline when the member is not online;
4) Real-time push of members online.

The data distribution model is as follows:

The problem is: if there is an abnormality in step 4, the group friends will lose the message, which will cause some family members to not know that "you are out of the order", resulting in serious consequences of urging marriage.

Therefore, the optimized solution is to store the message first regardless of whether the group member is online or not.

According to the above ideas, the optimized group message process is as follows:

1) Traverse the members of the group chat and send messages;
2) Everyone in the group chat will save a copy;
3) Query the online status of each member;
4) Online real-time push.

The optimized solution above is the so-called "write diffusion".

The problem is that everyone saves a copy of the same "You are out of order" message, which causes a lot of waste of disk and bandwidth (this is the biggest drawback of write proliferation).

Therefore, the optimized solution is to store one copy of the group message entity, and users only store the message ID index.

So the optimized sending group message process is as follows:

1) Traverse the members of the group chat and send messages;
2) Save a copy of the message entity first;
3) Then everyone in the group chat stores a copy of the ID reference of the message entity;
4) Query the online status of each member;
5) Online real-time push.

The solution after the second optimization is the so-called "read diffusion".

5.1.2) Summary:

1) Read diffusion: read operations are heavy, write operations are light, and resource consumption is relatively small;
2) Write proliferation: read operations are light, write operations are heavy, and resource consumption is relatively large.
Judging from the public technical information, the group chat messages of WeChat and Dingding should use the writing diffusion method. For details, please refer to these two articles: "WeChat background team: WeChat background asynchronous message queue optimization and upgrade practice sharing", "Alibaba" IM technology sharing (4): Reliable delivery optimization practice of Xianyu billion-level IM messaging system" (note the section "5.5 Server-side storage model optimization").

5.2 Objects associated with "message"
5.2.1) Message entity model:

Common messaging services can be abstracted into several entity model concepts: user/user relationship/user equipment/user connection status/message/message queue.

The entity model relationship in the IM system is roughly as follows:

5.2.2) Entity model concept explanation:

User entity:

1) User -> User terminal equipment: each user can log in at multiple terminals and send and receive messages;
2) User -> Message: Considering the read proliferation, the relationship between each user and the message is 1:n;
3) User -> Message Queue: Considering the read proliferation, each user will maintain his own "message list" (1:1). If capacity expansion is taken into account, you can even open up a message overflow list to receive more than the "message list" "The message data of the capacity (in this case, it is 1:n);
4) User -> User Connection Status: Considering that the user can log in from multiple terminals, the app/web will have corresponding online status information (1:n);
5) User -> Contact relationship: Considering that users are ultimately connected by a certain business to form multiple contact relationships, and finally form a private chat or group chat (1:n);

Contact relationship (mainly business determines the relationship between users and users), for example:

1) How many people are in a family, and how many people are in the family group chat;
2) In the ToB scene, in the DingTalk Enterprise Edition, we often have the existence of corporate group chat.

Message entity:

Message -> Message Queue: Considering the read proliferation, the message will eventually belong to one or more message queues, so it will be distributed in different message queues in the group chat scene.

Message queue entity:

Message queue: To be precise, it is a message reference queue, and the index element in it ultimately points to a specific message entity object.

User connection status:

1) For the app side: disconnection caused by network reasons, or the user manually kills the application process, it is offline;
2) For the web side: the browser is disconnected due to network reasons, or the user manually closes the tab, it is offline;
3) For official accounts: cannot be offline and online respectively;
4) For small programs: cannot be offline and online respectively.

User terminal equipment:

The client is generally Android&IOS, the web is generally a browser, and there are other flexible WebViews (official accounts/small programs).

5.3 Message storage scheme
For the message storage scheme, there are essentially only three schemes: either in the memory, or in the disk, or a combination of the two storage (it is said that in order to optimize the performance of large companies, the active message data is stored in the memory, after all, there is money. ~).

The advantages and disadvantages of the main solutions are analyzed below:

1) Option 1: Considering performance, all data is stored in redis;
2) Option 2: Considering resources, use redis + mysql to store data.

5.3.1) For scheme one: redis

Prerequisite: The user & contact relationship is business data, so it is stored in a relational database by default.

flow chart:

The explanation is as follows:

1) The user sends a message;
2) Redis creates an entity data & an entity data timer;
3) Redis adds entity data references in the user queue of user B;
4) B user pulls the message (the pull mode will be mentioned in 5.2).

Implementation plan:

1) User queue, zset (score to ensure orderliness);
2) List of message entities, hash (msg_id ensures uniqueness);
3) Message entity counter, hash (supports the number of references to group chat messages, and deletes the corresponding message in the entity list when the countdown reaches zero to save resources).

The advantages are: memory operation, good response performance

The disadvantages are:

1) The memory consumption is huge, eg: Unless the large factory, the precious memory resources of the server of the small company can't consume the business, as the business grows, if you don't want to expand the resources, you need to manually clean up the data;
2) Affected by the redis disaster tolerance strategy, if redis goes down, it will directly cause data loss (you can use redis cluster deployment/sentinel mechanism/master-slave replication and other means to solve it).

5.3.2) Option 2: redis+mysql

Prerequisite: The user & contact relationship, because it is business data, is unified and defaulted to use relational database storage.

flow chart:

The explanation is as follows:

1) The user sends a message;
2) mysql creates an entity data;
3) Redis adds entity data references in the user queue of user B;
4) B user pulls messages (the pull mode will be mentioned below).

Implementation plan:

1) User queue, zset (score to ensure orderliness);
2) Message entity list, transferred to mysql (table primary key id to ensure uniqueness);
3) Message entity counter, hash (delete this concept, because the total available disk resources are much higher than the total memory resources, even if the mysql database has been stored, there will be no big problem when the business volume is millions, if it is a huge volume Business needs to consider the performance of sub-table and sub-database processing to retrieve data).

The advantage is:

1) The message entity with the largest amount of data is removed, which greatly saves memory resources;
2) Disk resources are easy to expand, cheap and practical.

The disadvantage is: disk read operation, poor response performance (from the perspective of product design, is the IM system you maintain a strong IM or a weak IM).

6. News consumption pattern

6.1 Pull mode
Reasons for choosing the message pull mode:

1) Due to the large number of users (observers), the server cannot monitor the status of the client one by one, so the data interaction of the message module uses the pull mode, which can save server resources;
2) When the user has unread messages, the client actively initiates the request, which can refresh the client status in time.
6.2 ack mechanism
Technical principle:

1) The data pull request based on the pull mode (the first fetch interface) and the data pull confirmation request (the second fetch interface) are paired;
2) The client calls the fetch interface a second time, and needs to tell the server the anchor point of the last message consumption, and the server then deletes the read messages.

The schematic diagram of the request model is as follows:

Implementation scheme 1: Based on each message number ACK:

1) Implementation: After receiving the message, the client sends an ACK message number to the server to inform that it has received the message. When the server receives the ACK message number, it marks that the message has been sent successfully;
2) Disadvantages: In this scheme, because the client is numbered one by one ACK message, it will cause too many interactions between the client and the server. Of course, the client can batch ACK multiple messages asynchronously, thereby reducing the number of times.

Implementation scheme 2: Based on sliding window ACK:

1) After the client receives the message number, it compares it with the local message number:

• If it is smaller than the local one, it means that the message has been received, so ignore it without processing;
• If it is larger than the local one, the local message number is used to pull the message list larger than the local message number from the server, that is, the incremental message list.
• After the pull is complete, update the largest message number in the message list to the new local message number;

2) When the server receives the ack message, it will mark read or delete in batches.

This method is called a push-pull combination solution in business, and it is often used to realize real-time data synchronization in distributed message queues, configuration centers, and registration centers.

6.3 Benefits of the ack mechanism
After getting the message for the first time, if there is no ack mechanism, the process is:

1) The server deletes the read message data;
2) The server responds to the client with the data packet.

If the client cannot obtain data for a long time due to network delays, the client will disconnect the HTTP request and ignore the processing of the response data. Eventually, the message data will be deleted and cannot be restored later.

With the ack mechanism, even if it fails to get the message for the first time, the client can continue to request the message data, because the message data will not be deleted until the ack is confirmed.

7. Microservice design

Generally speaking, IM microservices can be divided into three basic microservices:

1) User service;
2) Business services;
3) Connection management service.

Reference architecture diagram:

Their division of labor is as follows.

User microservices (login & logout of user equipment):

1) Equipment number storage;
2) Connection status update;
3) Other login users are kicked out, etc.

Connection management microservices:

1) State preservation: save the long-connected objects of the user equipment;
2) Eliminate invalid connections: Retraining has long-connected object status, and delete objects over time;
3) Accept the client's heartbeat packet: refresh the status of the persistent connection object.

Messaging business microservices:

1) Message storage: the message storage strategy for private chat/group chat (please refer to the section "Message Storage Model");
2) Message consumption: message acquisition response and ack confirmation deletion (please refer to the section "Message Consumption Mode");
3) Message routing: When the user is online, the message notification package is routed to the "message connection management microservice" to notify the user client to fetch the message.

Finally, let me mention the routing of messages:
There are also communication methods between microservices, such as business services to connection management services, and real-time message routing notifications can be implemented between the two through RPC.

8. Offline news push

In the offline push scheme, everyone generally considers two schemes:

1) Enterprise self-developed backstage offline PUSH system;
2) The enterprise connects to the PUSH system of a third-party mobile phone manufacturer by itself.

8.1 Enterprise self-developed backstage offline PUSH system
Technical principle:

At the application level, the client maintains a long connection with the background offline PUSH system. When the user's status is detected as offline, the client is notified through this long connection that there is a new message, and then the phone pop-up title is awakened.

The disadvantages are:

As the restrictions on Android and Apple systems become more and more stringent, the activity cycle of the general client is strictly limited. Once the client process is moved to the background, it will be killed immediately, which makes it particularly difficult to keep the client alive ( This is also a headache for many small and medium enterprises. After all, there are only primary market apps like WeChat or QQ, and the mobile phone system is willing to leave a back door for them to keep alive). For details, you can read the "Android P official version is coming: the real nightmare of background application keep-alive and message push".

8.2 The enterprise connects to the PUSH system of a third-party manufacturer by itself
Technical principle:

At the system level, each hardware system will maintain a long connection with the corresponding mobile phone manufacturer. When the user status is detected as offline, the background will push messages through HTTP requests to notify the third-party mobile phone manufacturer’s server, and then wake up the app’s bombs through the system. Window title.

The disadvantages are:

1) As the application side, whether the message is delivered to the user side is unknown; the stability of the push also depends on the service stability of the third-party mobile phone manufacturer;
2) Doing additional sdk docking work, which increases the workload;
3) Third-party vendors may upgrade the sdk version at any time, causing the server that has not upgraded the sdk to fail to push, which brings difficulties to the deployment of the Sass system;
4) The configuration of the push certificate should also consider the maintenance cost.

In short, offline message push in IM is a very headache (Of course, it is mainly Android, and Apple's official APNs in iOS are more comfortable). If you are interested, read the following articles:

"Comprehensive inventory of the real operating effects of the current Android background keep-alive solution (as of 2019)"
"Rongyun Technology Sharing: Practice of Network Link Keep Alive Technology of Rongyun Android IM Products"
"In 2020, is there still a scene for Android background keep-alive? See how I can achieve it elegantly! 》
"The strongest Android keep-alive idea in history: an in-depth analysis of Tencent TIM's process immortality technology"
"The ultimate reveal of Android process immortality technology: the underlying principle of process killing, APP response to killing skills"
"Android keep-alive from entry to giving up: obediently guide users to add whitelists (with 7 examples of whitening models)"
"Alibaba IM Technology Sharing (6): Optimization of the offline push arrival rate of Xianyu billion-level IM messaging system"

9. Other technical points to be considered

9.1 Security
Regarding IM security, my personal experience is this:

1) The security of business data transmission is accessed using https;
2) Real-time messages use SSL/TLS to encrypt long connections;
3) Using private protocols, it is not easy to parse;
4) End-to-end encryption of content security, no decryption in any intermediate links (that is, the sending and receiving ends exchange mutual keys for decryption, and the server cannot decrypt it);
5) The server does not store messages.

Among the above points: the long-term connection security in IM is more important and not easy to deal with, because it requires a balance and trade-off between security and performance (cannot patronize security and lose the high throughput performance of IM long-term connection). For details, please refer to the "WeChat New Generation Communication Security Solution: Detailed Explanation of MMTLS Based on TLS1.3" shared by the WeChat team.

In addition: For higher security scenarios, a combined encryption scheme can be considered. For details, please refer to "Exploring the Application of Combined Encryption Algorithm in IM".

9.2 Consistency
The problem of IM message consistency is mainly to ensure that messages are not out of order. For this topic, beginners can read this "Introduction to Zero-based IM Development (4): What is the message timing consistency of the IM system? ", I won't repeat it.

There are many entry points to solve the consistency problem. The most common one is to use the orderly message unique ID. Regarding the orderly and unique ID generation problem, the WeChat team’s idea is very good. You can learn from the "WeChat Technology Sharing: WeChat’s Massive IM Chat Message Serial Number Generation Practice (Principles of Algorithms)".

In addition, the following articles on the order of news are also very good, you can refer to:

"How to ensure the "sequence" and "consistency" of IM real-time messages? 》
"A low-cost method to ensure the timing of IM messages"
"A set of IM architecture technical dry goods for hundreds of millions of users (Part 2): reliability, orderliness, weak network optimization, etc."

9.3 Reliability
The so-called reliability in IM, to put it bluntly, is to ensure that the message is not lost. This seems to be a natural and common technical point, and it is another big topic in the IM system. Given my limited level, I don’t do anything about it. IM beginners may be able to pass "Zero-based IM Development Introduction (3): What is the reliability of the IM system? "This article is to understand the concept of reliability.

Then read the "Implementation of IM Message Delivery Guarantee Mechanism (1): Guarantee the Reliable Delivery of Online Real-time Messages" and "Implementation of the IM Message Delivery Guarantee Mechanism (2): Guarantee the Reliable Delivery of Offline Messages". Basically You can have a deeper understanding of the technical key points of IM reliability.

The following practical summaries are suitable for peers who have some IM experience to learn. You can learn from them:

"Rongyun Technology Sharing: Fully Revealing the Reliable Delivery Mechanism of 100 Million-level IM Messages"
"IM development dry goods sharing: how to elegantly realize the reliable delivery of a large number of offline messages"
"From the perspective of the client to talk about the message reliability and delivery mechanism of the mobile terminal IM"
"Alibaba IM Technology Sharing (4): Reliable Delivery Optimization Practice of Xianyu's 100-million-level IM Message System"

9.4 Real-time
The technical point of IM real-time has returned to the foundation of the "instant messaging" technology. It can be said that without real-time, there will be no technical category of "instant messaging", and we can see its importance. Regarding the concept of real-time, beginners can go through "Introduction to Zero-based IM Development (2): What is the real-time of IM system? "Let’s study this article, I won’t be too long-winded, people are better than I said.

The real-time communication solutions in the author’s project all use WebSocket (if you don’t understand WebSocket, you can read "WebSocket from entry to proficiency, half an hour!" and "Understanding modern web-side instant messaging technology is enough. : WebSocket, socket.io, SSE"), but some low-version browsers may not support WebSocket, so in actual development, it is necessary to be compatible with the capabilities provided by the front-end for program design.

The following two peer-to-peer practical summaries on real-time are also good:

"How to ensure the efficiency and real-time performance of large-scale group message push in mobile IM? 》
"Alibaba IM Technology Sharing (5): Timeliness Optimization Practice of Xianyu's Billion-level IM Message System"

10. My experience in project practice

As a developer, I have been maintaining the iterative company's IM messaging system for more than two years. The following is my own little experience.

The key and difficult points I have experienced are as follows:

1) Business closed loop: how messages are written to storage, how messages are consumed, how online messages are realized, how offline messages are realized, what is the difference between group chats/private chats, and how multi-terminal messages are realized;
2) Resolve bugs and fill holes: how to configure the third-party push certificate if online messages cannot be received;
3) Code optimization: Monolithic architecture splits microservices;
4) Storage optimization: Redis version 1.0 is stored to redis+mysql version 2.0;
5) Performance optimization: Interface performance optimization such as unread reminders.

There are still areas for optimization of the project:

1) One of the high-availability solutions: is to deploy multiple connection management servers to support more user connections;
2) The second high-availability solution: For a single connection management service, Netty is used to optimize the framework layer, so that one server can support more user connections;
3) When the volume of messages increases sharply: further optimization of message storage can be considered;
4) Hot and cold news deployment: There will be differences in business volume in different regions. For example, in some economically developed provinces, the IM system will face greater pressure, and some underdeveloped provinces will have lower service pressure, so this can be considered Hot and cold deployment of data.

11. Write at the end

The IM message system module that I took over from the architect two years ago has allowed me to gradually cultivate architectural thinking. I see the best and thank you for my teacher.

IM technology is an enduring field, but at the same time, the technical assets that can be directly used are also very scarce. The products of traditional IM giants are usually privatization agreements and privatization plans. It is difficult to have a common solution in the industry that can be used directly. (Including materials or open source code), it is this kind of non-universal and inaccuracy that indirectly leads to an increase in the threshold of IM technology. Therefore, if a company wants to engage in IM, if there is no technology accumulation, it can only build wheels from scratch.

In order to change this situation, I also hope that the students engaged in IM development will not be boring to build cars, they should learn from the ideas of their peers, and at the same time can actively share their own experience, so that IM development will no longer be painful.

The above is a good start, welcome to leave a message to discuss, and make progress together.

12. Reference materials

[1] One entry is enough for beginners: develop mobile IM from scratch
[2] Why does the mobile terminal IM based on the TCP protocol still need a heartbeat keep-alive mechanism?
[3] The official version of Android P is coming: the real nightmare of background application keep alive and message push
[4] WebSocket from entry to proficiency, half an hour is enough!
[5] One article is enough to understand modern web-side instant messaging technology: WebSocket, socket.io, SSE
[6] A set of mobile IM architecture design practice sharing for massive online users (including detailed pictures and texts)
[7] A set of original distributed instant messaging (IM) system theoretical architecture scheme
[8] A set of high-availability, easy-scalable, high-concurrency IM group chat and single chat architecture design practices
[9] WeChat Technology Sharing: Practice of Generating Massive IM Chat Message Sequence Numbers in WeChat (Principles of Algorithms)
[10] Alibaba IM technology sharing (4): Reliable delivery optimization practice of Xianyu billion-level IM messaging system
[11] Ali IM technology sharing (5): Timeliness optimization practice of Xianyu billion-level IM messaging system
[12] A set of IM architecture technical dry goods for hundreds of millions of users (Part 2): reliability, orderliness, weak network optimization, etc.
[13] From novice to expert: How to design a distributed IM system with billions of messages
[14] Revealing the IM architecture design of enterprise WeChat: message model, 10,000 people, read receipts, message withdrawal, etc.
[15] Rongyun Technology Sharing: Fully Reveal the Reliable Delivery Mechanism of 100-Million-Class IM Messages
[16] Instant Messaging Security (6): Principles and Application Practice of Asymmetric Encryption Technology
[17] Easy to understand: an article to master the security principles of instant messaging message transmission
[18] WeChat's next-generation communication security solution: Detailed explanation of MMTLS based on TLS1.3
[19] Introduction to zero-based IM development (2): What is the real-time nature of the IM system?
[20] Introduction to zero-based IM development (3): What is the reliability of the IM system?
[21] Introduction to zero-based IM development (4): What is the message timing consistency of the IM system?

This article has been simultaneously published on the official account of "Instant Messaging Technology Circle".
The synchronous publishing link is: http://www.52im.net/thread-3752-1-1.html