News Push Technology Dry Goods: The Technological Evolution of Meituan's Real-time News Push Service

This article was shared by the Meituan technical team. The author "Jianwu, Jiameng, Lu Kai, Feng Jiang", the original title "The Evolution of Meituan's Terminal Message Delivery Service Pike", has been revised.

1 Introduction

In the traditional sense, real-time message push is usually the technical category of applications such as IM instant messaging. With the popularization of the mobile Internet, it is normal for everyone to have a mobile phone and be “online” at any time, so the real-time news is reached. Capabilities have been widely demanded and are no longer limited to applications such as IM instant messaging.

For mobile "entry" applications like Meituan, the ability to push real-time messages has penetrated into all aspects of the entire APP. The push technology currently used in the Meituan application is a set of easy-to-access, highly reliable, and high-performance two-way message real-time delivery services named Pike.

This article will first introduce the technical evolution of version 2.0 from Pike's system architecture upgrade, work mode upgrade, and long-term stability keep-alive mechanism upgrade, etc., and then introduce its technical feature support in new business scenarios such as live broadcast and games, and provide support for the entire system The technical practice during the upgrade process is summarized. I hope this article can help and inspire readers who are interested in real-time message push services or are engaged in related work.

(This article was published simultaneously at: http://www.52im.net/thread-3662-1-1.html)

2. Related articles

Real-time news push technical article reference:

1. "Meizu 25 million long-connected real-time message push architecture technical practice sharing"
2. "Interview with Meizu Architects: Experiences and Experiences of Real-time Message Push System with Massive Long Connections"
3. "Millions Online's Real-time Push Technology Practice Road of Meipai Live Barrage System"
4. "The Evolution of the Message Push Architecture of the Jingtokyo Mai Merchant Open Platform"
5. "Decrypting "Dada-JD Daojia" Orders Instant Dispatch Technical Principles and Practices"
6. "Special Topic on Long Connection Gateway Technology (4): Practice of iQiyi WebSocket Real-time Push Gateway Technology"
7. "Himalaya's offline message push system architecture design practice with hundreds of millions of users"
8. "The Evolution of 15 Million Online Message Architecture in a Single Room of WeChat Live Chat Room"
9. "Baidu Live's Mass User Real-time Messaging System Architecture Evolution Practice"
10. "Technical dry goods: from scratch, teach you to design a million-level message push system"

Other articles shared by the Meituan technical team:

1. "Meituan Dianping's mobile terminal network optimization practice: greatly improving connection success rate, speed, etc."
2. "Deep Decryption of Meituan's Distributed ID Generation Algorithm"

3. The past and present of v1.0

3.1 The birth background of v1.0
In 2015, Meituan launched the Shark terminal network channel to provide long-link agency acceleration services for the company's mobile terminal. Shark improves the end-to-end success rate of network requests through the global deployment of network access points and maintains long-term connections, reduces end-to-end delay, and improves user experience.

Pike 1.0 is an in-app push service based on the Shark connection channel. Because the underlying transmission is based on the Shark long-connection channel, Pike 1.0 is born with excellent genes such as low latency, high reliability, and anti-DNS hijacking. At present, Pike 1.0 is widely used in business scenarios such as instant messaging chat, marketing push, status delivery, and configuration synchronization within Meituan.

3.2 v1.0 workflow
The Pike 1.0 mobile SDK will after each successful creation of a persistent connection:

• 1) Use APPID, the unique device identifier UnionID (Meituan unique identifier, comment unique identifier, etc.) to initiate registration with the server;
• 2) After successful registration, the business server can actively push messages to the device's App through the interface provided by the Pike 1.0 server SDK;
• 3) The message pushed by the server reaches the client through the long connection channel, and is finally delivered to the business party through the registered callback interface.

The overall workflow is shown in the figure below:

3.3 Advantages of v1.0
The underlying transmission of Pike 1.0 is based on the Shark long link channel.

So Pike 1.0 has a good performance in the following aspects:

• 1) Anti-hijacking: The underlying channel directly uses IP direct connection, which saves time-consuming DNS resolution and avoids the risk of being hijacked by DNS;
• 2) Low latency: Shark long connection uses the nearby access point long connection method, eliminating the need for multiple connection establishments and handshake consumption in traditional HTTP, and the end-to-end data transmission delay is greatly shortened compared with HTTP;
• 3) High security: Shark adopts a custom binary protocol for data transmission, and carries out channel-level TLS encryption, which is tamper-proof and safer;
• 4) Good experience: Pike 1.0 shares service clusters with Shark. Shark long-distance connection channels have deployed access points in many overseas locations. The proxy speeds up access, and the network delay and success rate are better than regular requests.

PS: For the optimization articles of HTTP and DNS under the mobile network, you can look at the following articles:

• "Comprehensive understanding of miscellaneous problems such as mobile DNS domain name hijacking: principles, root causes, HttpDNS solutions, etc."
• "Meitu App's mobile DNS optimization practice: HTTPS request time is reduced by nearly half"
• "Baidu APP mobile terminal network deep optimization practice sharing (1): DNS optimization"

3.4 Technical Pain Points of v1.0
As a derivative product of Shark, Pike 1.0 has its shining points, but the pain points caused by the strong reliance on Shark make developers complain. The main pain points are as follows.

3.4.1) Code structure coupling:

In the client SDK, the Pike 1.0 code is coupled with the Shark code structure, sharing the logic of the underlying channel connection, data encryption and decryption, and binary protocol.

Pike 1.0 and Shark code structure diagram:

Disadvantage one of coupling: Difficulty in code optimization and upgrade. Changes to one SDK often require more consideration of whether it has a negative impact on another SDK and whether the impact is controllable, which unreasonably increases development costs.

The second disadvantage of coupling: Shark and Pike 1.0 share the network configuration environment. As shown in the figure above, the network environment configuration of SharkTunnel through DebugPanel will take effect for both Shark and Pike 1.0 at the same time, but the business side often only pays attention to it when using it. One of the SDKs, the interaction between different SDKs has introduced many customer service issues, and also brought more interference factors to the troubleshooting of customer service issues.

3.4.2) The account system is chaotic:

Pike 1.0 supports only one device unique identifier UnionID on the same App. The UnionID registered on different apps will be different. For example, Meituan uses the Meituan unique identifier, and reviews use the comment unique identifier.

If a business is only used on one app, Pike 1.0 can naturally work well, but the same business may need to be used on multiple apps at the same time (as shown in the figure below). If the business side is not compatible with the account system, Services that use the unique ID of Meituan as a push identifier on the Meituan App will not work, and businesses that use the unique ID of Meituan as a push identifier on the Meituan App will also not work.

As a result, the push ID logic of the same service on different apps will be very complicated, and the back-end must maintain the mapping between multiple account systems at the same time to solve the problem of account system confusion.

Pike 1.0 account system incompatibility diagram:

3.4.3) Push connection is unstable:

Pike 1.0 lacks special optimization for push scenarios due to the shared channel logic of Shark, and it is not good enough in detecting channel abnormalities and disconnected recovery. In terms of channel availability, the SLAs of Shark and Pike 1.0 are also very different.

For example, when the long connection channel is unavailable, Shark can downgrade the short connection to avoid the problem of lowering the success rate caused by the continuous failure of business network requests. But for Pike 1.0 at this time, if the channel cannot be recovered quickly, it will cause business message delivery failure, which will directly affect the message delivery success rate. Therefore, the public logic of the Shark channel for connection keep-alive cannot be perfectly applied to the Pike 1.0 business scenario.

Although Pike 1.0 further strengthens the heartbeat detection mechanism at the protocol layer on the basis of the Shark channel to improve channel availability, the channel cannot detect abnormalities in time.

In addition: The reliability of the event distribution technology used inside Pike 1.0 has not yet reached 100%, and some customer service issues that are abnormally disconnected and cause unsuccessful push will be reported sporadically.

To sum up: The demand for special optimization for unstable push connection has been continuously put on the agenda.

3.5 Birth of v2.0
The existing technical pain points of Pike 1.0 have encountered many challenges in the current situation of increasingly rich business scenarios.

In order to solve the existing problems encountered in the operation of Pike 1.0 on the Android and iOS platforms:

• 1) We re-organize the product architecture and code implementation;
• 2) Product integration with Pike Web, another message delivery service serving H5 by the Basic Technology Department.

And then launched a new upgraded product-Pike 2.0.

The picture below shows the product panorama of Pike 2.0. In view of the current situation of Pike 1.0, Pike 2.0 has made many optimizations on the front and back ends, including technical architecture upgrades, cluster independence, and protocol expansion.

On the client side, Pike 2.0 provides an SDK based on multi-language implementation to serve multiple platforms. On the server side, Pike uses a distributed cluster that deploys Java applications to provide services.

Pike 2.0 product panorama:

The following content will mainly explain the technical design of the Pike 2.0 client SDK from the perspective of the client, and explain the technical advantages brought by Pike 2.0 in principle.

4. v2.0 architecture design

In response to the technical pain points of Pike 1.0 code structure coupling mentioned above, Pike 2.0 has carried out a new architecture upgrade, maintaining product isolation from Shark in terms of code structure, environment configuration, and service clusters.

4.1 Design Idea
After nearly a year of technology accumulation and precipitation, the TunnelKit long-link kernel components and TNTunnel general channel components extracted from Shark have stabilized, so Pike 2.0 chose to build a two-way message channel service based on TunnelKit and TNTunnel.

Specific advantages are:

• 1) Pike 2.0 is built based on the TunnelKit long-link kernel, which can effectively reuse the existing long-link control related functions and reduce unnecessary development work;
• 2) Pike 2.0 can share the relevant general features of TNTunnel, such as Shark protocol encapsulation, data encryption and decryption, etc., and the subsequent maintenance cost is small;
• 3) The Pike 2.0 protocol is used as the payload transmission of the Shark protocol, which can flexibly customize the protocol related to its own characteristics.

4.2 Overall architecture
Client architecture evolution diagram:

The overall structure is shown in the figure above, including:

• 1) Pike interface layer;
• 2) Pike channel layer;
• 3) TNTunnel channel layer;
• 4) TunnelKit is connected to the kernel layer for a long time.

4.2.1) Interface layer:

The Pike interface layer aims to provide a simple and reliable interface for all businesses that require in-app messaging services in the mainstream front-end technology stack.

mainly:

• 1) Pike 2.0 provides SDK access to mainstream technology stacks of companies such as Android, iOS, MRN, etc., and services can be flexibly selected according to their needs;
• 2) Pike 2.0 has designed two different clients for different message QPS (see below for details);
• 3) Pike 2.0 provides a business-insensitive migration solution for the online Pike 1.0 system. The business side can migrate from the previous Pike 1.0 system to the Pike 2.0 system to send and receive messages without any human input.

For point 2), we designed it like this:

• 1) For services with a volume of more than 50 messages per second, such as live barrage push, we recommend accessing the aggregate message Client;
• 2) For other services with small message volume, ordinary message Client can meet the demand.

4.2.2) Channel layer:

The Pike channel layer is the implementation layer of the feature. All API calls of the Pike interface layer will be converted into encapsulated Tasks through thread scheduling to complete specific operations at the Pike channel layer. The Pike channel layer is a single-threaded model, which avoids thread safety issues to the greatest extent. .

Pike features are as follows:

• 1) Reconnection after disconnection: In view of the unstable characteristics of long connections, the Pike 2.0 channel uses the disconnection reconnection mechanism to allow the business party to consider that it is continuously available without any network failures;
• 2) Service authentication: The back end of the service can monitor connection changes through the Pike 2.0 channel, and at the same time, it can judge the availability of client devices that access the network;
• 3) Alias mechanism: The business identity is isolated for different business parties, and each business can customize the identity ID, which solves the pain point of Pike 1.0 that different businesses on the same App platform must force the use of the same identity ID;
• 4) Uplink/downlink message: Pike 2.0 is a two-way channel service. It not only supports the original message push capability of Pike 1.0, that is, the server sends downlink messages to the client; it also supports the client to actively send messages, that is, the client to the server. Send an upstream message. As long as the business passes through the Pike 2.0 system, a closed loop of messages can be formed;
• 5) Grouping/aggregating messages: Pike 2.0 supports message grouping and message aggregation to meet the use of high QPS business scenarios. Among them, message grouping means that the service can broadcast messages to a group of users through custom labels; message aggregation means that the blowout messages will be aggregated and delivered in a short time to improve the throughput of the system;
• 6) Message order protection: Pike 2.0 supports orderly delivery of uplink messages sent by the same client to a fixed service server;
• 7) Independent channel: Pike 2.0 defaults that all services use a shared channel. For services with large business volume or throughput requirements, the exclusive channel can be automatically switched to ensure the success rate and delay of message delivery;
• 8) Channel keep-alive: Pike 2.0 adds channel inspection based on the connection keep-alive, which can automatically restart the channel when an abnormality is detected, ensuring that the channel availability is further improved in an environment that requires long-term stability.

4.2.3) TNTunnel channel layer:

The TNTunnel channel layer is a functional layer that encapsulates general channel logic. It mainly involves general core modules such as channel state management, protocol encapsulation, and data encryption and decryption. It is an independent layer that extracts general channel logic from the original Shark channel.

Although the Pike protocol is an application layer protocol built on top of the existing Shark protocol, the Pike channel has been completely decoupled logically from the original Shark channel.

• On the one hand, Pike 2.0 will reuse the mature technology of the Shark protocol to the greatest extent, but does not rely on the original Shark logic;
• On the other hand, subsequent upgrades and optimizations involving binary protocols, security protocols and other protocols can all serve Pike 2.0 at the same time.

4.2.4) TunnelKit connects to the core layer:

The main function of the core layer of the TunnelKit long connection is to connect to the Socket to process the sending and receiving of TCP or UDP data, and to manage the availability of each connection.

Each Pike 2.0 channel maintains a connection in TunnelKit. The heartbeat keep-alive mechanism and connection management ensure that there is always a connection to carry Pike data under the normal network environment.

As the foundation of all channel layers, TunnelKit is the most important layer that determines the stability of the upper long-connected channel.

5. v2.0 working mechanism

On the basis of the new push architecture upgrade, Pike redesigned and improved the working mechanism for the pain points of the Pike 1.0 account system chaos and unstable push connection mentioned above.

Among them, PikeClient, as the portal for the Pike system to connect to the business side, plays a vital role in the entire Pike 2.0 system. This section will introduce its working mechanism with PikeClient as the starting point.

5.1 PikeClient life cycle
In order to better maintain the internal state of Pike 2.0, PikeClient uses a state machine to be responsible for life cycle management.

PikeClient life cycle diagram:

As shown in the figure above, the life cycle of PikeClient mainly includes the following parts:

• 1) onStart: This state is the state that the business party enters after calling StartClient or RestartClient. At this time, PikeClient has been started normally, and then Pike 2.0 will initiate business authentication and flow to other states according to the authentication result, as shown in the figure. If the service authentication fails, it enters the onStop state, if the service authentication succeeds, it enters the running state;
• 2) onStop: This state is the state entered after the business party calls StopClient or the business authentication fails. At this time, PikeClient has stopped working. After the client enters this state, it needs to be Restarted before it can be reused;
• 3) Running: This state is the state where PikeClient is working steadily. At this time, Pike 2.0 is waiting to respond to the downlink message pushed by the service or is ready to send uplink messages at any time. As a two-way message channel, the ability of Pike 2.0 to process upstream and downstream messages is completely parallel;
• 4) onReceive: This state is the state that PikeClient enters after successfully receiving the downlink message. Pike 2.0 will re-enter the running state after delivering the received message to the business party and wait for the next operation;
• 5) onSendSuccess/onSendFailure: This state is the state that PikeClient enters after sending an uplink message. The business party can obtain the result of this message transmission by monitoring this state.

Through the life cycle management based on the state machine, it not only strictly defines the workflow of PikeClient, but also accurately monitors its internal state, which improves the maintainability of PikeClient.

5.2 PikeClient working mode
In response to the chaotic account system pain points of Pike 1.0, Pike 2.0 has designed a new working mode.

As shown in the figure below, Pike provides two modes of shared channel and independent channel through the channel proxy module to meet the needs of different business scenarios.

5.2.1) Shared channel mode:

The shared channel mode is the basic working mode of Pike 2.0, and new business parties will use this mode to access Pike 2.0 by default.

In Pike 2.0, the PikeClient and the Pike channel service have a many-to-one sharing relationship. Each business party will have its own PikeClient. Each PikeClient can customize its message push ID and avoid using the global ID. The business backend can streamline the push identification logic and avoid maintaining multiple account systems at the same time.

PikeClient for different services is only isolated at the access level, and unified management will be completed by the Pike channel service in the Pike 2.0 channel. This many-to-one sharing relationship allows all Pike businesses to share the Pike 2.0 channel characteristics, and at the same time, it can set its specific message processing capabilities for each business use scenario. Each business party that accesses Pike 2.0 only needs to pay attention to it. Just your own PikeClient.

5.2.2) Independent channel mode:

Independent channel mode is the expansion capability of shared channel mode. Pike 2.0 decides whether to switch to this mode through configuration control.

By default in Pike 2.0, all business parties share the same Pike channel service. However, in view of the different business scenarios, each business has different requirements for message throughput, message delay and other SLA indicators. For example, game business has different message delays. Tolerance for too long is worse. For special services, Pike 2.0 provides independent channel switching capability support.

All PikeClients connect to Pike channel services through the Pike channel proxy module. The Pike channel proxy module can control PikeClient to work with specific Pike channel services through switch configuration. Through the use of proxy mode, the integrity of the original structure is guaranteed, and independent channel capability support can be completed without adjusting the Pike channel code logic; it can also expand the channel switching capability, effectively manage the channel switching process, and let The Pike 2.0 channel maximizes service capabilities while avoiding resource waste.

5.3 PikeClient keep-alive mechanism
The keep-alive of PikeClient completely relies on the keep-alive of the Pike 2.0 channel. Aiming at the pain point of the unstable push connection of Pike 1.0, the Pike 2.0 channel continues to optimize on the basis of absorbing the technology precipitated by Pike 1.0 in the keep-alive mechanism, and finally designed based on the heartbeat detection. , Reconnection mechanism and triple keep-alive mechanism of channel inspection.

The keep-alive mechanism is as follows:

5.3.1) Heartbeat detection:

Heartbeat detection is a common method to check the status of the network connection. Pike long connection is a TCP connection, while TCP is a virtual connection: if factors such as abnormal network nodes in the actual physical link cause the connection to be abnormal, the client and server will not be connected. If the connection abnormality cannot be detected in time, the connection status will be in the ESTABLISHED state, but the connection may be dead. The heartbeat detection is a technical solution to solve this network abnormality.

PS: Regarding why the tcp protocol still needs a heartbeat keep-alive, you can read this article "Why does a TCP-based mobile terminal IM still need a heartbeat keep-alive mechanism?" ".

When the heartbeat period set by the heartbeat patrol timer arrives, the client judges whether there is an abnormality of the last heartbeat timeout. If the heartbeat timeout, the connection is considered to be unavailable, and the connection is removed from the connection pool and triggers the following retry. Even mechanism.

In order to find channel abnormalities faster, Pike 2.0 is configurable for the heartbeat period and heartbeat timeout, and can be flexibly set for different App usage scenarios.

Moreover, every time uplink data is sent, whether the last heartbeat has timed out is detected in time, so that the heartbeat detection result does not have to wait until the next heartbeat cycle arrives.

Pike 2.0 does not use a fixed heartbeat frequency to send heartbeat packets. Pike 2.0 uses the uplink and downlink data packets of the channel to dynamically reduce the number of heartbeat packets sent.

In addition, smart heartbeat is also a topic that Pike 2.0 continues to pay attention to. If you are interested, read the following:

1. "WeChat team original sharing: Android version of WeChat background keep-alive actual sharing (process keep-alive)"
2. "WeChat team original sharing: Android version of WeChat background keep-alive actual sharing (network keep-alive)"
3. "Mobile IM Practice: Realizing the Smart Heartbeat Mechanism of Android WeChat"
4. "Mobile IM Practice: Analysis of the Heartbeat Strategy of WhatsApp, Line and WeChat"
5. "An article to understand the network heartbeat packet mechanism in instant messaging applications: functions, principles, realization ideas, etc."
6. "Rongyun Technology Sharing: Practice of Network Link Keep Alive Technology of Rongyun Android IM Products"
7. "Correctly understand the heartbeat and reconnection mechanism of the IM long connection, and implement it by hand (there is a complete IM source code)"
8. "A Discussion on the Design and Implementation of an IM Intelligent Heartbeat Algorithm on Android (with sample code)"

5.3.2) Reconnection mechanism:

The reconnection mechanism is the core feature of Pike 2.0 as a long connection channel, and it is also the most important part of the construction of Pike 2.0 connection stability.

The client will decide whether to trigger a reconnection in three links: sending a message, receiving a message, and heartbeat detection:

• 1) On the one hand, if it is actively found that the available connections in the connection pool are insufficient, the reconnection mechanism is automatically started;
• 2) On the other hand, when the existing available connection is closed, the reconnection mechanism will also be triggered automatically.

Pike 2.0 will use the Fibonacci sequence backoff algorithm to initiate a connection request during the reconnection process until the connection is successfully established:

• 1) On the one hand, Pike 2.0 guarantees that as long as the network is available, it can always maintain an available long connection to serve business messages;
• 2) On the other hand, Pike 2.0 avoids continuous connection to make the system fully loaded when the network is continuously unavailable.

For articles about disconnection and reconnection, you can read the following systematically:

• "Web-side instant messaging practice dry goods: How to make your WebSocket disconnect and reconnect faster? 》
• "Correctly understand the heartbeat and reconnection mechanism of the IM long connection, and implement it by hand (there is a complete IM source code)"
• "Teach you how to use Netty to realize the heartbeat mechanism and disconnection reconnection mechanism of network communication programs"

PS: If you need practical code, you can also read the open source project MobileIMSDK. It has a complete logical implementation of the heartbeat and reconnection mechanism of im, which can be used for reference.

5.3.3) Channel inspection:

Channel inspection is an effective mechanism to further improve the stability of Pike 2.0 based on the heartbeat detection and reconnection mechanism.

The client will set a global inspection timer according to the heartbeat cycle. When the time set by the timer arrives, the client will trigger the channel anomaly detection logic, and will try to restart the channel once an abnormality is found.

Pike 2.0 will first obtain the current channel status when the channel anomaly detection is triggered. If the channel is not actively closed but the channel is in an unavailable state, Pike 2.0 will force a self-startup.

In addition, during the channel inspection process, the inspection manager will continuously collect the timeout exceptions that occurred during the message sending and receiving process. When the number of timeout exceptions continuously exceeds the configured maximum threshold, Pike 2.0 will consider the current channel availability to be low and need Force shutdown and perform a self-startup.

6. New technical features of v2.0

As an upgraded version of Pike 1.0, Pike 2.0 is not only to solve the technical pain points of Pike 1.0, but also to open up new application scenarios through new features is also the focus of Pike 2.0.

6.1 Aggregate messages
With the rise of live broadcast services in the company, many business parties within the company also use Pike 1.0 as a transmission channel for downlink real-time messages such as barrage, commentary, and live room control signaling.

However, based on the previous design architecture, Pike 1.0's ability to provide reliable services for scenes such as bullet screens and comments that need to process a large amount of messages in a short period of time is gradually inadequate.

It is mainly manifested in aspects such as a decrease in the success rate of message delivery, an increase in delay, and an increase in system performance overhead when the QPS increases substantially. Pike proposes a more general solution for the delivery of messages in live broadcast scenes by introducing aggregated messages.

6.1.1) Design ideas:

The messages involved in the live broadcast scenario mainly have the following characteristics:

• 1) Barrage, as a real-time interactive carrier, needs to process a large amount of pictures, texts and other information in a short time. If aggregation is not done, a lot of bandwidth will be wasted;
• 2) Compared with the normal push scene in the live room, since the user has entered the live room, the user behavior is relatively uniform and controllable, so a group message is needed for unified processing;
• 3) Different types of message processing logic can be prioritized in the live broadcast room. For example, lottery and control signaling require reliability and cannot be discarded, while barrage can be appropriately discarded according to the heat of the live broadcast room and service affordability.

Aggregate messages mainly adopt the following ideas in design:

• 1) Aggregate messages from the time dimension to reduce unnecessary bandwidth consumption;
• 2) Adopt the message classification strategy, set different priorities according to the type of messages, to ensure the reliability of important messages;
• 3) Abstract the aggregation unit similar to the live broadcast room, and uniformly manage the user behaviors that join the aggregation unit;
• 4) Adopt the client-side active pull strategy;
• 5) Provide upstream message capability and provide a more complete message circulation path.

For point 4): Compared with the traditional server-side push strategy, active pull is to use the client's natural distributed characteristics to save the user state on the client. The server can reserve more resources by reducing state maintenance. For business processing.

6.1.2) Scheme process:

Pike 2.0 uses a circular queue for each aggregation unit to maintain a message list. After priority filtering, messages sent to the aggregation unit will be inserted into the position indicated by the tail pointer of the queue. As the messages in the aggregation unit continue to increase, they finally reach the maximum. When the queue is long, the head pointer will continue to move to make room for the tail pointer. The aggregation unit avoids service performance problems caused by the short-term blowout growth of messages by controlling the maximum length of the circular queue.

When the client actively pulls, it will carry the offset of the last obtained message in the circular queue, so that the service will aggregate the messages between the position indicated by the offset and the position indicated by the tail pointer as The result of this pull is returned to the client at one time. Different clients maintain their own offsets to avoid the server's state maintenance of the client.

The specific interaction between the client and the server is shown in the following figure: the client takes the initiative to pull after joining the aggregation unit. If the offset carried in this pull can get the aggregated message from the service circular queue, then the message will be called back. Perform the next pull operation immediately after giving the business. If the offset carried this time is already at the position of the tail pointer of the circular queue, the server will not make any response. The client will wait for the pull timeout to start the next pull operation, and repeat the process until the client leaves the Polymerization unit. At the same time, if the business server has a message that needs to be pushed, it will be sent to the Pike server by RPC, and the message processing module will insert the effective message filtered by the message classification strategy into the ring queue.

Flow chart of aggregated message interaction:

6.2 Message Preservation
Pike 1.0 was only suitable for message push scenarios at the beginning of its design, and Pike 2.0 evolved into a two-way message delivery service based on it, which not only supports downstream message push, but also supports upstream message delivery. Pike 2.0 further expands the function of message order protection in the aspect of upstream message delivery.

The message order here mainly includes two levels of meaning:

• 1) First, the message sent by each business client reaches the same business server to the greatest extent;
• 2) Secondly, these messages arrive at the service server consistently in accordance with the sequence sent by the client.

6.2.1) Sticky sessions:

In order to make the messages sent by each business client reach the same business server to the greatest extent, Pike 2.0 introduces the concept of sticky sessions.

A sticky session refers to the fact that messages on the same client connection are forwarded to a specific business machine for processing. After the client is disconnected and reconnected, the messages on the new connection are still forwarded to the business machine.

Sticky sessions can be summarized into the following process:

• 1) At the first business login, the Pike 2.0 server will select a business server using the load balancing algorithm, and notify the client of the route identification of the business server through the business login result and save it. After that, if the channel status is stable, all upstream messages will be saved. Will be delivered to the service server;
• 2) If the channel status fluctuates and disconnection occurs during the period, Pike 2.0 will re-login the business after initiating the reconnection. This time the business login will re-report the previously saved routing identification to the Pike 2.0 server, so that the Pike 2.0 server will Re-bind the service server through the routing identifier;
• 3) If the business server indicated by the routing identifier has stopped providing services, the Pike 2.0 server will re-select a new business server through the load balancing algorithm, and the client will obtain the new routing identifier, and the subsequent logic will repeat the process until The Pike 2.0 client exits.

6.2.2) Timing consistency:

We all know that TCP is in order, so under the premise of the same TCP connection, under what circumstances will the messages sent by the client arrive at the business server out of order?

The reason is: the Pike 2.0 server reads the message from the TCP and delivers it to the business server through RPC asynchronous call.

In order to solve this problem: the simplest solution is of course that the client limits the sending window of the message queue to 1. Each sent message can only receive the ACK after the Pike 2.0 server delivers it to the business server, and then send the next message . However, considering that the delay of network transmission on the link is much greater than the delay of processing on the end, the QPS of this scheme is set by the network transmission as a bottleneck. Assuming an RTT is 200ms, then the theory of this scheme can only reach 5 QPS.

In order to improve the QPS of upstream message order delivery, Pike 2.0 adopts the solution of setting up a message queue buffer on the server side.

As shown in the figure below: the client can send multiple messages at one time within the range allowed by the sending window, and the server buffers the received messages in the message queue in order, and then serially buffers these messages through RPC calls The messages of are delivered to the business server in order.

This order-preserving scheme shifts the bottleneck point of QPS performance from the previous network transmission delay on the link to the delay of RPC calls. In actual scenarios, an RPC call is often between a few milliseconds, which is much smaller than that of the network. The transmission delay on the link then significantly improves the QPS.

The problem of message timing consistency is a hot technical point in the real-time communication field:

• "Introduction to zero-based IM development (4): What is the message timing consistency of the IM system? 》
• "How to ensure the "sequence" and "consistency" of IM real-time messages? 》
• "A low-cost method to ensure the timing of IM messages"
• "A set of IM architecture technical dry goods for hundreds of millions of users (Part 2): reliability, orderliness, weak network optimization, etc."

7. Stability guarantee of v2.0

7.1 Monitoring system
Pike 2.0 relies on the Meituan monitoring platform Raptor to complete the monitoring system construction. Both the server and the client have built their own comprehensive indicator monitoring.

The Pike 2.0 client uses Raptor's end-to-end indicator capabilities and custom indicator capabilities to output more than 10 monitoring indicators to monitor the Pike system in real time. These indicators cover multiple dimensions such as channel establishment, message delivery, business login, and system exceptions.

On the basis of real-time indicator monitoring, Pike 2.0 has configured alarm thresholds for different indicators. Taking push messages as an example, if the market data of a particular App fluctuates by more than 10% per minute, the Raptor system will report to the members of the Pike project team. Push alarm information.

Based on all Raptor monitoring indicators, Pike 2.0 refines the core SLA indicators as follows:

Pike 2.0 will regularly output large-scale data reports based on core SLA indicators. At the same time, data can be filtered based on multiple dimensions such as App, business type, and network type to meet the needs of different users for indicator data.

7.2 Individual User Tracking
The monitoring system can reflect the stability of the push system from a global perspective. For individual users, the Pike management platform provides complete link tracking information.

Each Pike 2.0 connection is distinguished by a unique identification Token. Through the active detection of the "connection sniffing" module of the Pike management platform through the unique identification Token, the interaction flow of all signaling on the corresponding connection can be obtained.

As shown in the figure below: The process clearly marked the client's signaling such as connection establishment, initiation of authentication, and alias binding. Click on the corresponding signaling to jump to the signaling details to further view the information carried by the signaling, and then combine it with the SDK to embed it. Click on the offline log of Meituan log service Logan to quickly find and locate the problem.

8. Construction results

As of June 2021, Pike has access to 200+ services, with an average daily message volume of about 5 billion+, Pike 2.0 message arrival rate>99.5% (up 0.4% compared to Pike 1.0), and Pike 2.0 has an average end-to-end delay Time <220ms (about 37% reduction compared with Pike 1.0).

Some application cases:

• 1) Live broadcast scene message service plan: Support live broadcast interactive function of live broadcast business, and have the ability to support millions of large-scale live broadcasts online at the same time;
• 2) Real-time reach solutions such as message push and feed stream pre-loading: support real-time push of marketing, control and other business messages, increase the arrival rate of business messages by up to 10%, and reduce the time required to establish a long connection channel by 5%;
• 3) IoT device access solution: support the IoT access capability of the meal cupboard business, and help the business message arrival rate increase from 98.4% to 99.6%;
• 4) Small game scene message delivery scheme: Supports Meituan small game scene communication capability, the message arrival rate reaches more than 99.8%, and the uplink delay is as low as 195ms.

9. Future prospects

Pike's real-time message push service is widely used in Meituan. Currently, it mainly focuses on business scenarios such as real-time contact, interactive live broadcast, and mobile synchronization. With the rapid development of the company's business, Pike has put forward higher requirements for usability, ease of use, and scalability, and hopes to improve the network experience in various business scenarios.

Therefore, the main focus of Pike's future planning is to provide multi-terminal and multi-scenario network communication solutions, continuously improve the protocol ecology, and combat complex networks in various application scenarios.

Specifically:

• 1) Expand general basic capabilities: improve channel performance. By optimizing the order-preserving scheme, providing dedicated channels, optimizing the transmission protocol and other methods, the throughput and stability are further improved, and the push delay is reduced;
• 2) Building Internet of Things access: Provide IoT access layer solutions. Provide a unified IoT access layer solution for the company's Internet of Things application scenarios (bicycles, power banks, dining cabinets, smart helmets, warehouses, store equipment, etc.), and support multiple access protocols (HTTP, MQTT, CoAP, etc.) , Provide safe and reliable equipment connection and communication capabilities for business;
• 3) Optimize the weak network communication experience: Based on Meituan’s self-developed MQUIC network protocol library, explore Pike over QUIC on the mobile side and IoT side, explore WebTransport technology on the desktop side, and fully support the QUIC protocol to improve the situation in the weak network large package scenario Network performance, reducing the time-consuming of long-tail distributed requests.

Appendix: More real-time news push technical articles

"Detailed Explanation of iOS Push Service APNs: Design Ideas, Technical Principles and Defects, etc."
"Carrier Pigeon Team Original: Let's Walk Through the Pit of APNS on iOS10"
"Android-side message push summary: implementation principle, heartbeat keep-alive, problems encountered, etc."
"Literacy Sticker: Understanding the MQTT Communication Protocol"
"A complete Android push Demo based on MQTT communication protocol"
"Interview with IBM Technical Manager: The development process and development status of the MQTT protocol, etc."
"Seeking advice on android message push: the pros and cons of GCM, XMPP, and MQTT"
"Analysis of Real-time Message Push Technology on Mobile"
"Literacy Post: A Brief Talk on the Principles and Differences of Real-time Message Push in the Background of iOS and Android"
"Absolute Dry Goods: Technical Essentials of Push Service for Massive Access Based on Netty"
"Mobile IM Practice: Google Message Push Service (GCM) Research (from WeChat)"
"Why IM tools like WeChat and QQ don't use GCM to push messages? 》
"Technical Practice Sharing of Large-scale and High-concurrency Architecture of Aurora Push System"
"From HTTP to MQTT: An Overview of Application Data Communication Practice Based on Location Services"
"Meizu 25 million long-connected real-time message push architecture technical practice sharing"
"Interview with Meizu Architects: Experiences and Experiences of Real-time Message Push System with Massive Long Connections"
"In-depth talk about the small matter of Android message push"
"Practice of Message Push for Hybrid Mobile Applications Based on WebSocket (including code examples)"
"Implementation Ideas for a Secure and Scalable Subscription/Push Service Based on Long Connections"
"Practice Sharing: How to build a highly available mobile messaging system? 》
"The Practice of Go Language to Build a Ten Million-Level Online Highly Concurrent Message Push System (From 360 Company)"
"Tencent pigeon technology sharing: practical experience of tens of billions of real-time message push"
"Millions Online's Real-time Push Technology Practice Road of Meipai Live Barrage System"
"The Evolution of the Message Push Architecture of the Jingtokyo Mai Merchant Open Platform"
"Understanding iOS Push is enough: The most comprehensive iOS Push technology in history"
"Achieving iOS's High-Performance Message Push Based on the Latest HTTP/2 Interface of APNs (Server Side)"
"Decrypting "Dada-JD Daojia" Orders Instant Dispatch Technical Principles and Practices"
"Technical dry goods: from scratch, teach you to design a million-level message push system"
"Special Topic on Long Connection Gateway Technology (4): Practice of iQiyi WebSocket Real-time Push Gateway Technology"
"Himalaya's offline message push system architecture design practice with hundreds of millions of users"
"Live Broadcast System Chat Technology (3): The Evolution of 15 Million Online Message Architecture in a Single Room of WeChat Live Chat Room"
"Live broadcast system chat technology (4): Baidu live broadcast massive user real-time messaging system architecture evolution practice"
"Message Push Technology Dry Goods: The Technological Evolution of Meituan's Real-time Message Push Service"

This article has been simultaneously published on the official account of "Instant Messaging Technology Circle".

▲ The link of this article on the official account is: click here to enter. The synchronous publishing link is: http://www.52im.net/thread-3662-1-1.html