In-depth analysis of the "circle group" relationship system design | "circle group" technical series articles

Introduction: NetEase Yunxin's new IM top-tier product "Circle Group" has attracted great attention after its debut. Many Yunxin customers are also very concerned about the underlying technical details and principles of "Circle Group" while accessing. To this end, we decided to launch a series of technical articles related to Yunxin's "Circle Group" to share some of NetEase Yunxin's thoughts on the technical design of "Circle Group".

Business Features

There is a popular saying in the Internet industry that technology serves business. In practice, an important aspect is to design technical solutions for business characteristics. Therefore, if you want to understand the relationship system design of "Circle Group", you must first understand the relationship business characteristics of "Circle Group".

What are the characteristics of the relationship business of "Circle Group"? First, the relationship is complex, that is, there are many relationship subjects, complex management mechanisms, and heavy linkage and coupling; the second is huge scale, that is, the number of members can reach one million, and the batch of changes can reach one million.

The so-called complex relationship, specifically: First, there are many relationship subjects. In the "circle group" business, the relationship subjects include servers, channels, identity groups, channel groups, etc. The server carries the community relationship and is responsible for maintaining community member relationships; the channel belongs to the server, carries the content relationship, and is responsible for the maintenance of the content interaction relationship; An identity group can be subordinate to a server or channel, bear the identity authority relationship, and be responsible for identity setting and authority configuration; a channel group is subordinate to a server, and is associated with a group of channels, carrying the channel template relationship, and is responsible for classifying channels and sharing configuration. The second is the management mechanism. In the "circle group" business, only as far as the member management mechanism is concerned, the server member adopts the invitation/application mechanism, the channel member adopts the public/private mode + black/whitelist mechanism, the identity group member adopts the join/removal mechanism, and the channel group member adopts the join/removal mechanism. Use a synchronization mechanism with channel members. The last is linkage coupling. In the "circle group" business, taking the maintenance of channel members as an example, channel members are not only affected by public/private mode + black/whitelist configuration changes, but are also affected by changes in server members, identity groups, and identity group members. Linkage changes.

The so-called huge scale, specifically: on the one hand, the number of members can reach millions. In the "circle group" business, the number of server members can reach millions. Further, the number of channels and identity groups under a million-member server can also reach millions. On the other hand, the batch of changes can be in the order of millions. In the "circle group" business, not only the number of members is huge, but the number of batches of change operations can also reach millions. Including: deleting a server/channel/identity group with one million members, adding and deleting a million-membership group in the blacklist and whitelist of channel/channel grouping, etc.

Starting from the two characteristics of the "circle group" relationship business, it can be found that the "circle group" relationship is a new business scenario that is different from the group relationship, and will face new technical difficulties.

Technical Difficulties

There are two main technical difficulties in the "circle group" relationship system. One is the complexity of business logic caused by the association and coupling of multiple relationship entities and multiple management mechanisms in a hierarchical structure; the other is the complexity of business processing in terms of time, space, resources and other overheads caused by the number of members and the huge scale of change batches .

business logic complexity

First of all, the "circle group" has a multi-level structure. Including server/channel secondary structure, server/channel grouping/channel tertiary structure, etc. The change of a single relationship subject involves not only the change of itself, but also the change of the subject of the relationship between superiors and subordinates, which can be said to affect the whole body. In contrast, groups have no hierarchy, and group changes can only be done on their own.

Secondly, the "circle group" has an identity group. An identity group is a collection of server members with common permissions. Members of different identity groups can cross each other, and the identity group will participate in member management as a whole. That is to say, the change of members is no longer just the entry and exit of individual members (1-100 people), but there will be a large in and out of the whole group of members (1-1000000 people). In contrast, groups do not have identity groups, and special group members, including group owners and administrators, are small in number and do not repeat each other.

Finally, there are a variety of member management mechanisms for the "circle group". The management mechanism of server members and identity group members is similar to that of groups, but the management mechanism of channel members and channel group members is a new model. Channels are divided into two types: public and private. Public mode allows all server members to be visible by default, but excludes blacklisted identity groups and blacklisted members; private mode does not allow all server members to be visible by default, but allows whitelisted identity groups and whitelists to be released. member. In addition to being affected by public/private mode + black/whitelist configuration changes, channel members are also affected by changes to the relationship principals (server members, identity groups, identity group members) on which they depend. Further, channel members are also affected by synchronized channel grouping changes. In comparison, the invitation/application mechanism for group members can be said to be insignificant.

business processing complexity

The first is the huge number of members. Since the number of members can reach one million, the storage space overhead and network transmission overhead of the entire member list become very huge. Regardless of the server cache of the full member list data or the synchronization of the full member list data from the server to the client, it will become hard to accomplish.

The second is the huge change batch size. The relationship change of a single interface call may be accompanied by a million-scale linkage relationship change, which will lead to huge processing time overhead and computing resource overhead. No matter if all changes are processed synchronously, or all changes are processed on a single machine, it will become difficult. accomplish.

The last is the huge size of notification messages. The relational system not only needs to do data processing of relational changes, but also needs to notify the client of the result of the change. Due to the huge number of members of each relationship subject in the "circle group", a single change needs to be spread into millions of notifications and issued at the same time, which requires huge computing resource overhead and network transmission overhead.

In contrast, the group plan does not involve these technical difficulties due to the limited number of members and the batch size of changes.

Starting from the two technical difficulties of the "circle group" relationship system, it can be found that the "circle group" relationship system faces new technical difficulties that are different from groups. To solve these technical difficulties, innovative technical solutions are required.

Technical Analysis

The "circle group" relationship system is an important part of the entire "circle group" solution. Before introducing the technical solution of the "circle group" relationship system in detail, it is necessary to first understand the overall structure of the "circle group" solution.

The overall structure of the "circle group"

The above shows the overall structure of the "Circle Group" scheme. It can be seen that the "Circle Group" as a whole is a layered structure. From top to bottom:

• Client layer: including mobile, desktop, and cross-platform SDKs for client integration, and OpenAPI for server calls.
• Access layer: including LBS service, persistent connection service and API gateway, corresponding to client SDK and user server respectively.
• Network layer: including the self-developed global real-time transmission network WE-CAN.
• Business layer: including App service for SDK business processing and WebServer service for OpenAPI business processing.
• Service layer: It is divided into service modules such as login, message, relationship, identity group, and support. Each service module includes multiple microservices or consumers.
• Infrastructure layer: including the database and middleware used by the system.

Relational System Architecture

Relational System Technical Details

In the "Circle Group" relationship system, the technical details of the three main points of the scheme are discussed below, including channel membership management, online broadcast of change notifications, and cloud retrieval of relationship data.

Channel Membership Management

Channel membership management is a very challenging problem in the "circle group". Channel members involve multiple relationship entities, multiple management mechanisms, serious linkage changes, and huge number of members and change batches. It can be said to be a typical representative of the "circle group" relationship business. The complexity of channel membership management in terms of business logic and business processing can be imagined. In response to the channel membership management problem, "Circle Group" has designed two mechanisms to solve it. Including: the combination mechanism of final state maintenance and transition calculation, and the mechanism of sequential asynchronous parallel processing of events.

The final state maintenance and transition calculation mechanism are combined. Specifically, the channel membership data is finally maintained in the persistent database, and the channel member data query requirements are directly supported in the final state stage when the channel members are not changed. When a channel member changes, due to the complexity of the change logic and change processing, it takes a period of time to complete the relationship change, which is called the transition phase. During the transition phase, the data of the channel member table persisted in the database is not completely accurate and cannot directly support the query requirements of the channel member data. At this time, the channel member configuration metadata directly calculates the channel member to support query requirements. Because the change of channel member configuration metadata is processed synchronously, direct calculation of channel members from channel member configuration metadata during the transition phase can ensure query accuracy. By dividing the management of channel membership into two stages: final state and transition, and adopting different channel member query schemes in different stages, it not only solves the problem of high cost of obtaining channel member resources simply by calculation, but also solves the problem of channel member change delay caused by The problem of inaccurate results of channel members obtained from the database.

In addition to the acquisition query problem of channel members, the change processing of channel members is also very important. The sequential asynchronous parallel processing mechanism of events is used to solve the change processing problem of channel members. The first is to significantly reduce the complexity of the business logic of channel membership management by hierarchically and systematically defining change operations that affect channel membership as change events. Second, through ID hash, distributed lock, event version number control, etc. to ensure the sequential processing of change events, effectively avoiding persistent data errors caused by out-of-order event processing. Third, the event is transferred through the message queue and processed asynchronously on the consumer, which effectively solves the problem of blocking the interface call caused by the large batch of linkage changes. Fourth, the time delay of channel membership changes is significantly shortened through multi-threaded parallel acceleration and local cache reuse acceleration in single event processing.

Change Notification Online Broadcast

The relational system not only needs to do data processing of relational changes, but also needs to notify the client of the result of the change. In the millions of "circle group" relationships, each relationship change notification will face a massive proliferation of recipients. In addition to the surge in notification distribution, the difference in the priority of notification reception by different recipients is also worthy of attention. In response to the problem of online broadcasting of change notices, “Circle Group” has designed two mechanisms to solve it. Including: change classification notification mechanism, data notification pull mechanism.

In the change classification notification mechanism, on the one hand, according to the role of the relevant personnel in the change, it is divided into participants and observers for notification, that is, the participants must notify, and the observers notify according to the subscription requirements. Among them, participants are generally a few key personnel in the change, and observers are other people who can see the results of the change except the participants. By classifying notifications, different recipients can pay reasonable attention to the difference in the priority of receiving notifications, and the diffusion scale of change notifications is also accurately reduced. On the other hand, observers can give full play to the advantages of the online broadcast subscription model of "Circle Group" by notifying them according to their subscription requirements. The so-called online broadcast subscription mode means that after the user logs in, they need to subscribe to the notifications of the servers/channels they are interested in. The “Circle Group” system will record these subscription information. Relationships instead of member lists + online status Get a list of users who need to broadcast online, eliminating the need to traverse all members of a server/channel and their online status. By adopting the online broadcast subscription model, not only the computational overhead and bandwidth overhead of the online broadcast of change notifications are significantly reduced, but also the parallel acceleration and horizontal expansion of the online broadcast of change notifications in long-connection service clusters can be realized.

The ultimate purpose of the change notification is to send the changed data to the client. Different from groups, "Circle Group" does not directly bring the changed data to the client through notification, but adopts the mechanism of notifying the client that there is a change and then triggering the client to pull the result data. The reason is that unlike the group that synchronizes the full amount of relational data to the client, the "circle group" client no longer stores the full mirror of relational data, so it is no longer necessary to maintain the full amount of history + incremental changes on the client. Full mirroring of relational data. At the same time, the observers who subscribe to the change notification do not need to care about the result data of the change all the time. The number of observers who care about the result data of a certain change will be much less than the number of observers who subscribe to the change notification. Therefore, , the data notification pull mechanism will significantly reduce the resource overhead of change notifications. In addition, compared to notifications with changed data, only notifications have changes, which is convenient for merging notifications of the same type directly, without worrying about the timing and concurrency of merging changed data. In this way, the data notification pull mechanism can notify the merge in a short time. Significantly reduces server online broadcast overhead and client notification reception overhead.

Relational data cloud retrieval

In the "circle group", with the substantial increase in the scale of the relationship, the group based on the application server to fully query the relational data or the client to fully synchronize the relational data to achieve accurate query and flexible sorting is no longer applicable. In this regard, "Circle Group" adopts the solution of relational data cloud retrieval.

The "Circle Group" relational data cloud retrieval solution can support the retrieval capabilities of servers, channels, members, etc. In terms of retrieval scenarios, it includes square retrieval and internal retrieval.

Square Retrieval: Used to retrieve servers of interest. It can be searched according to various dimensions such as name and category. Search results can be sorted by predefined fields (number of members, etc.) or custom values (data popularity, etc.).

Internal Retrieval: Used to retrieve the servers, channels, members, etc. visible to the user. It can be searched according to various dimensions such as name and nickname. Search results can be sorted by predefined fields (creation time, etc.) or custom values (data popularity, etc.).

Summarize

Having said so much, as a newly designed product, Yunxin "Circle Group" does not have any limitations of historical baggage (but it can fully absorb historical advantages), you can use it to build a Discord-like product, or anything you want. Social/entertainment/game products, you are welcome to choose.