SIGCOMM's first Multi-path QUIC paper: Alibaba's self-developed multi-path transmission technology XLINK

Author: Yun Fei, Miao Ji, Zhi You, Hong Qiang

The XLINK multiplex transmission technology jointly developed by the Alibaba Amoy Technology Department and the Dharma Academy XG Laboratory. The related paper "XLINK: QoE-driven multi-path QUIC transport in large-scale video services" has been formally accepted by the top academic conference SIGCOMM 2021 Accepted, this is also the first paper on multipath QUIC in the history of the SIGCOMM conference.

Summary

Have you ever experienced:

(1) When you watch the video, you’re watching the show, and suddenly you find that the video has become very stuck. Why is it useless to reconnect?

(2) When you make a voice call from the mall to the parking lot, the call is suddenly disconnected. Must you dial to reconnect?

(3) When you want to work against the clock on a high-speed office, but you find that the email can't be sent out?

The above-mentioned problems can all be attributed to one problem, that is, the "weak network." Due to the inherent spectrum limitations of wireless networks, insufficient wireless signal coverage, resource competition among multiple users, frequent base station switching in high-mobility scenarios, etc., all may lead to frequent "weak networks". Overcoming a weak network is crucial to the user experience. To this end, the Tao Department Architecture Team of the Technical Department of Alibaba's Tao Department and the XG Laboratory of Dharma Academy jointly developed the XLINK multiplex transmission technology. XLINK enables Taobao users to use multi-path (5G/4G, WiFi) to transmit data at the same time, which fundamentally solves the user experience problem caused by a single-path weak network. XLINK is based on the IETF Multipath QUIC draft proposed by Alibaba[1], which is currently the only Multipath QUIC standard draft that has been tested in large-scale practice.

The QUIC technology was proposed by Google. Google published QUIC-related papers at the SIGCOMM conference in 2017 and caused a huge response from the industry. This year, the IETF QUIC 1.0 standard work will be officially completed. The next-generation HTTP protocol HTTP3 is based on QUIC. It can be said that QUIC is the core and key transmission technology in the current mobile Internet. Nowadays, more than 50% of Chrome browser traffic and 75% of Facebook traffic are transmitted using QUIC. After unremitting efforts in the past few years, Alibaba has rapidly grown from a follower of QUIC technology to an innovator of QUIC technology, and has made breakthroughs in multi-path QUIC technology. XLINK related papers have been officially accepted by the top academic conference SIGCOMM 2021. It is also the first article on multipath QUIC in the history of the SIGCOMM conference.

XLINK Related Articles Paper Reference:

Zhilong Zheng†, Yunfei Ma†, Yanmei Liu†, Furong Yang, Zhenyu Li, Yuanbo Zhang, Jiuhai Zhang, Wei Shi, Wentao Chen, Ding Li, Qing An, Hai Hong, Hongqiang Harry Liu, and Ming Zhang, XLINK: QoE -driven multi-path QUIC transport in large-scale video services, to appear in SIGCOMM 2021. († means a joint work)

XLINK is implemented based on the IETF multipath (Multipath) QUIC draft framework proposed by Alibaba. The draft is led by the Taoist Architecture and XG Lab, and conducted in-depth discussions with the Group Standardization Department, the Computing Institute of the Chinese Academy of Sciences, and the former chairman of the Internet Architecture Board, Christian Huitema. Demonstration and cooperation are currently the only Multipath QUIC standard draft that has been tested in large-scale practice.

XLINK is based on Alibaba's IETF Multipath (Multipath) QUIC draft reference:

Yanmei Liu, Yunfei Ma, Christian Huitema, Qing An, and Zhenyu Li. Multipath Extension for QUIC, Internet Engineering Task Force, December 2020. Work in Progress.

https://tools.ietf.org/html/draft-liu-multipath-quic-03

Multi-path technology: a decade-long exploration in academia and industry

The idea of using two paths for simultaneous transmission sounds simple, but it is difficult to do. At present, the only successful deployment of multi-path transmission in the industry is Apple’s Siri, Apple Music and other scenarios (Apple uses MPTCP RFC6824, which was standardized by IETF in 2013). These applications are all audio (Audio) types. They mainly use multipath to increase the robustness of transmission. However, in the past, large-scale applications of multiplex transmission technology were rarely seen in the industry's video applications (Video) or real-time audio and video applications (Real-time Video), because this type of scenario has a significant impact on the data bandwidth rate and Time delay has very demanding requirements. In a wireless network, the path quality changes very quickly, and the past multi-path protocols and scheduling algorithms will cause obvious stalls in high-speed changing scenarios. Before XLINK, multi-path transmission could not be used in audio and video. Academia has been struggling for many years. Many optimization solutions based on MPTCP have been proposed. However, there is no solution that can completely solve the following problems. Essential defect:

• kernel implementation, unable to provide customized optimization for application scenarios: audio and video applications have a feature, that is, the experience objectives (QoS) between different applications are very different, and the required transmission protocols and algorithms are also very different. For example, short video applications (smart short video) pay attention to the second opening rate, high-definition long video applications (Youku) have high bandwidth requirements, and video conferences and live broadcasts (Dingding, Hand Tao live) pay more attention to delay and smoothness. These hugely different application scenarios require transmission algorithms and protocols to be specially optimized for the application’s own QoS requirements. However, MPTCP is located in the network protocol stack of the operating system kernel, and all applications use the same set of algorithms, which makes it difficult for everyone to adjust and transmit the protocol at the same time. And the optimization iteration of the scheduling algorithm is also very difficult.

• Heterogeneous Network: MPTCP's multi-channel aggregation effect is not ideal. Because the multi-path transmission on the public network is heterogeneous, the delay difference between 5G/LTE and Wi-Fi is large, and multi-path will occur at this time. Head-of-line blocking problem (MP-HOL) [2]. The MP-HOL congestion problem refers to that when part of the data packets go on a slow path and some data packets go on a fast path, the fast path packets will arrive first, but they must wait for the slow path packets to arrive before they can be transmitted to the application, resulting in increased delay. In some cases, it is even worse than the better quality single path of the two paths. The bigger challenge is that the wireless link changes rapidly, so the current bandwidth prediction is difficult to be very accurate (in the wireless scenario, it is difficult for us to predict the bandwidth of the next second). Therefore, the scheduling algorithm based on bandwidth prediction frequently breaks down and settles on the long tail problem.
• traffic cost: In order to overcome the problem of heterogeneous networks, some multi-path transmission schemes choose to send redundant packets (sending the same data packet repeatedly on two paths) to avoid the problem of multi-path head-of-line blocking. Two new problems: 1. Repeatedly sending data packets will greatly increase the cost of additional data traffic. This brings high bandwidth costs for video applications, which is why Apple only uses MPTCP in audio scenarios such as Siri that require little traffic. 2. Redundant data packets will also occupy bandwidth resources, which in turn reduces the overall bandwidth utilization efficiency.

Other multi-path schemes such as MPUDP and MPRTP currently have their own difficulties and limitations. First, UDP does not guarantee the reliability of data packet delivery. Therefore, the different delays of multiple channels will bring a large number of out-of-sequence packets to upper-layer applications, and UDP does not recover lost packets, so it is rarely used at present. Since MPRTP was proposed, it has not been implemented on a large scale [6]. The reason is that when MPRTP distributes data packets to each path, it relies on the accurate estimation of the bandwidth and delay of each path, but unless it has a large amount of physical layer information, LTE signal The prediction itself is a very difficult problem to solve.

XLINK: QoE-driven multi-path solution

called XLINK?

XLINK (= X+LINK), originally meant to link through multiple paths (LINK), where X represents an unknown number, and also represents the continuous exploration of unknown areas in Alibaba. XLINK technology realizes WiFi/LTE/5G multi-path parallel transmission in the user mode based on the QUIC protocol, effectively increasing the transmission bandwidth, greatly reducing the transmission delay and stall rate, and exhibiting excellent transmission stability in high-mobility scenarios. At the same time, XLINK is also the world's first multi-path QUIC communication protocol deployed and verified in large-scale video scenarios. XLINK is very convenient to use. XLINK is implemented in the user mode protocol stack XQUIC[3] and supports platform deployment such as Android/iOS/Linux. For mobile app developers only need to integrate the XQUIC protocol library to use XLINK technology. Users only need to upgrade the app to enjoy the experience benefits brought by multi-path transmission [4].

In order to break through the fundamental limitation of single-path transmission and solve various landing problems encountered in past practices in multi-path protocols such as MPTCP, we developed XLINK. The biggest difference between XLINK and all previous multipath technologies is that it directly uses the QoE information of the application to implement path selection, switching and scheduling strategies. From a technical point of view, XLINK breaks through the design framework of the traditional multi-path protocol. Based on the user mode characteristics of QUIC, it proposes a Client-Server QoE feedback-driven multi-transmission scheduling scheme, which overcomes the multi-path protocol that has plagued the past ten years. (For example, MPTCP, MPUDP & MPRTP[5]) Two major problems in actual public network deployment:

• Transmission stalls and reduced aggregation efficiency caused by multi-way line head blocking problems
• The high extra bandwidth cost and traffic overhead introduced by redundant data packet sending

The overall structure of XLINK is shown in Figure 1, which has the following characteristics:

1. user mode deployment: XLINK works in user mode, integrated in the App, and realizes reliable data transmission and congestion control on top of UDP. It accompanies the rapid deployment and iteration of applications, plug and play. XLINK mobile phone applications can be updated on a weekly basis, and XLINK server applications and algorithm updates can be updated on a day or hour basis. Since XLINK is implemented in the user mode protocol stack and integrated with the app, XLINK can be customized and optimized directly for different applications.
2. High performance: XLINK uses the QoE feedback of video applications as the control signal of the scheduler. The QoE signal can contain a variety of parameters related to user experience. Through this QoE feedback control scheduler, XLINK successfully overcomes what MP-HoL brings The high performance and cost issues make the use of multipath technology in large-scale video applications feasible. When XLINK performs multi-path scheduling, it does not need to make an accurate estimation of the path bandwidth and delay, but adopts an adaptive compensation method of packet reinjection (Reinjection) to make the data packets adaptive on multiple paths Achieve balance. In addition, XLINK can further optimize user experience through its understanding of user QoE. For example, XLINK can make special optimizations for the first frame of short videos to reduce the user's first frame download time, thereby increasing the user's second opening rate.
3. Low cost: XLINK's scheduling algorithm can not only overcome the performance problems caused by MP-HoL, but also hardly increase the amount of additional data. QoE feedback helps XLINK adjust the intensity of reinjection to achieve the best performance and cost. Therefore, video applications with high bit rates can safely use multi-path transmission on a large scale without worrying about the cost of traffic.
4. Lightweight XLINK is developed completely based on the C language (implemented in the XQUIC user mode protocol stack). The overall package size of XQUIC is only 300+KB, which is very suitable for the use of various mobile terminals.

Hand wash landing effect

XLINK has already completed large-scale grayscale verification in hand-tao. The test results show that XLINK can reduce the average download time of short video fragments by 15.03% when used under weak networks, and reduce the time consumption of video fragment downloads by 25.28% on weak networks. [5]. In addition, during the journey, XLINK users can use WiFi hotspots and mobile LTE at the same time to maintain a smooth video viewing experience in high-mobility scenarios.

XLINK Hand Tao Demo display:

The following video shows the use effect of XLINK integrated in the mobile Tao. The application on the left uses XLINK with WiFi and LTE turned on, and the application on the right uses QUIC with single-path WiFi turned on; it can be seen that XLINK starts playing faster and the whole process The playback is smooth, and the single-path case has obvious freezes due to the jitter of the WiFi network during playback.

The video can be seen: SIGCOMM's first Multi-path QUIC paper:

Look to the future

The multi-path QUIC technology XLINK developed by Dharma Academy’s XG Laboratory and Tao Technology Co., Ltd. not only has a good experience optimization effect in scenes such as hand Tao short videos-short video download time can be reduced by 25 under weak network conditions % Or more; this year, we have gradually started to buy short videos and shop around by hand - and the multi-path QUIC draft has also received extensive attention from the IETF QUIC working group. As XLINK continues to be recognized by its peers on the international stage, it can be said that XLINK has achieved exciting results in terms of technology leadership, international standard formulation, and internal business empowerment.

We are at a key node leading to the era of 5G and edge computing, and the network architecture and technology that follow will produce revolutionary changes. 5G will further accelerate the explosive growth of diversified, terminal computing and edge computing services and application technologies such as big data, artificial intelligence, and mass storage. It can be inferred from this that, as the entrance to these new services, wireless networks are bound to usher in technological change once again, moving towards a new generation of high-performance, high-stability, and high-agile networks to respond to these new services.

At the same time, with the continuous enrichment of video applications and the heterogeneity of video QoS requirements, the previous method of adapting all applications to the transport layer is difficult to achieve better QoE. The user-mode protocol stack represented by QUIC passes through and video applications. Joint optimization can achieve the ultimate effect that was previously unattainable by the kernel protocol stack. XLINK has proven the powerful power of this synergy by combining QoE with multi-path. In the future, we will further promote the standardization of XLINK related technologies in the IETF, and we also expect that XLINK technology can better serve Alibaba users.

Appendix

[1] Multipath QUIC draft: https://tools.ietf.org/html/draft-liu-multipath-quic-03

[2] MP-HOL blocking problem: Refers to Multi-path Head-of-line blocking problem

[3] XQUIC is the IETF QUIC standardized protocol library developed by Ali

[4] The user decides whether to turn on and off

[5] The weak network statistics here refer to the p99 quantile long tail download time consumption of video fragments

[6] Singh, Varun, Saba Ahsan, and Jörg Ott. "MPRTP: multipath considerations for real-time media."Proceedings of the 4th ACM Multimedia Systems Conference. 2013.

, 3 mobile technology practices & dry goods for you to think about every week!