Demystifying how Alibaba Cloud RTS SDK realizes live broadcast to reduce latency and lag

Author: Yu Tao Tu Tan

This summer, nothing can be more enjoyable than a hearty Olympic game. However, watching the game live on the video platform also has pain points: "staggering" and "delay". Due to different geographical areas, complex network conditions, transmission speed and other reasons, when you are still anxious about the match point, cheers have been heard from the next door, and there is no interest in watching the match. In order to create a more smooth, stable and low-latency live broadcast experience, more and more companies pay attention to the application of low-latency live broadcast technology.

Alibaba Cloud's low-latency live broadcast RTS (Real-time Streaming) is based on the video live broadcast, and optimizes the underlying technology such as full-link delay monitoring, CDN transmission protocol transformation, and UDP. Through the integration of the live broadcast terminal SDK, it supports millions of The millisecond-level delay live broadcast capability between nodes in the high-level concurrency scenario makes up for the 3~6 second delay problem of traditional live broadcast, and guarantees the ultimate live broadcast viewing experience with low latency, low lag, and smooth seconds.

In order to reduce the end-to-end delay of live broadcasts, in 2020, Alibaba Cloud officially launched GRTN (Global Real Time Network), an ultra-low-latency, fully-distributed, sinking communication network deployed on edge cloud nodes. Class-level streaming media transmission network to support services such as video conferencing, live video broadcasting, and real-time audio and video. To use the short-delay/real-time transmission service provided by GRTN, the client can access it through the webrtc capability of the browser, or it can develop its own client protocol stack to access it.

What is RTS NetSDK?

The low-latency live streaming SDK is composed of RTS NetSDK, push streaming SDK and player SDK. Among them, the push stream SDK sends the audio and video frames output by the encoder to RTS NetSDK, and the player SDK reads audio PCM data and video H264/H265 data in the form of frames from RTS NetSDK.

Example: ffmpeg integrates RTS NetSDK to pull ARTC stream

RTS NetSDK is a protocol stack developed and running on the client based on GRTN. RTS NetSDK communicates with the SFU on GRTN through the UDP transport layer protocol to collect audio and video data. Considering the unknown network conditions, RTS NetSDK integrates WebRTC's QoS algorithm to adapt to network changes, thereby providing high-quality audio and video transmission services. The lightweight design of RTS NetSDK makes it very easy to be integrated with open source frameworks such as ffmpeg and obs.

RTS client configuration

RTS NetSDK is responsible for connecting Alibaba Cloud RTS services with push and pull streams. Its main features are small package size, simple integration, and no intrusion into existing services. It helps live broadcast customers to easily and quickly upgrade traditional live broadcasts to Alibaba Cloud RTS live broadcasts.

RTS NetSDK supports multiple platforms, Android/iOS/MacOS/linux/Windows/RTOS, supports multiple open source project integration access codes, ijkPlayer, obs, FFmpeg, and Alibaba Cloud push streaming SDK and player SDK have built-in RTS live broadcast Push-pull flow capability.

RTMP is based on TCP transmission, and RTS is based on UDP transmission. Under the condition of sufficient network bandwidth, RTS does not have many advantages over rtmp, but this ideal situation is more difficult to guarantee. The difference in the transmission protocol makes RTMP do not have too many QoS means for changing networks, and it completely relies on TCP congestion control. This congestion control does not aim at low latency. Alibaba Cloud's RTS introduces WebRTC's QoS technology, which has a better tolerance for network packet loss. Excellent bandwidth estimation and congestion control provide a delicate network usage strategy.

In addition, after the Adobe Flash Player stopped updating, RTMP is no longer available on the web. Customers either choose HLS technology with a very large delay, or choose WebRTC technology. The RTS H5 version is essentially WebRTC and can be run on web pages.

Product advantages

Low latency

Distribution plan	Features	delay
HLS	Wide range of support, acceleration of class files, mature distribution technology	Generally more than 10s
RTMP	It is mostly used to push streaming, and it is blooming all over the country. Used for client distribution is not mainstream	Generally more than 5s
FLV	Mostly used for client-side distribution, which is the current domestic mainstream solution	Generally more than 5s
Alibaba Cloud RTS Live	Use udp transmission technology	About 500ms (based on laboratory data)

Because of its TS file slice-the whole slice reading mechanism, HLS live broadcast has a delay of at least one gop.

RTMP, FLV live broadcast, in order to ensure the smooth playback of the client, the client is often kept cached for more than 5 seconds.

Alibaba Cloud RTS live broadcast is based on udp transmission technology, and the entire link is optimized for delay. The latest v1.8.0 version can already reduce the delay to about 500 milliseconds.

Anti-weak net

Alibaba Cloud RTS live broadcast is based on UDP transmission, through the application of WebRTC advanced technology (such as BWE, jitter buffer, NetEQ, FEC, NACK and other QoS mechanisms) and self-developed anti-weak network technology, in the complex and changeable user network environment , It has better fluency and delay than the live broadcast based on TCP transmission.

Small package size

RTS NetSDK has a compact package size. Take the latest versions of Android and iOS as examples:

platform	Increment size
iOS	755 KB（arm64）
Android	0.9MB（arm64）

Easy to integrate

Most of the live broadcast customers already have the formed live streaming and live streaming capabilities. RTS NetSDK is integrated into the customer’s existing SDK in the form of a library to achieve low-cost upgrades and is compatible with previous business logic. Alibaba Cloud RTS live broadcast already supports all the basic features of traditional live broadcast.

Ability type	RTS live broadcast support
Video encoding	h264、h265
Audio coding	aac、opus、g722
Audio sampling rate	44100、48k、16000 (g722)
Video SEI	support
Private NAL expansion	Even if it conflicts with the rtp definition type, it supports
rtmp push stream artc pull stream	support
artc push rtmp/hls pull	support
artc push flow artc pull flow	support
encryption	Alibaba Cloud KMS encryption system

Product application architecture

RTS live broadcast will support all the basic capabilities of traditional live broadcast, such as live recording, anti-theft chain, and broadcast director.

RTS NetSDK can be integrated into various products on the streaming side. In addition, the source code integrated into ffplay and ijkPlayer is provided, which customers can use directly or modify by reference. Alibaba Cloud Player SDK has built-in streaming capability of RTS NetSDK.

RTS NetSDK can also be integrated into end-side products. RTS NetSDK gives push products the ability to broadcast live RTS, and also provides the source code integrated into ffmpeg and obs. Alibaba Cloud Streaming SDK has built-in RTS NetSDK streaming capabilities.

RTS NetSDK is the cornerstone of the future integrated SDK for live broadcasting and communication. On top of RTS NetSDK, add a Multimedia Framework and QoS message processing to form an integrated SDK. This is good news for customers who already have their own Framework. There is no need to develop separate software for live broadcast and communication, and it also simplifies the implementation of live broadcast and microphone scenarios.

"Video Cloud Technology" Your most noteworthy audio and video technology public account, pushes practical technical articles from the front line of Alibaba Cloud every week, and exchanges and exchanges with first-class engineers in the audio and video field. The official account backstage reply [Technology] You can join the Alibaba Cloud Video Cloud product technology exchange group, discuss audio and video technologies with industry leaders, and get more industry latest information.