Play the live broadcast series from 0 to 1 to build a simple live broadcast system (1)

I. Introduction

With the advent of the 5G era, the audio and video industry may also usher in a spring of the industry. Live broadcasting has always been an important product form of the new video industry. From the initial show live broadcast, game live broadcast, to this year due to the epidemic, the current More popular online education live broadcasts, live broadcasts with goods, etc., various new live broadcast formats are increasingly shown in front of the public.

As a technology developer, let’s take a brief look today, how to quickly build a set of the simplest live broadcasting system, and briefly understand the architecture model of mainstream live broadcasting.

Two, push-pull flow model

First of all, let's look at a complete model diagram of the live push and pull stream. We can clearly see the macro architecture model diagram of the live broadcast.

2.1 Three main modules of live broadcast

Push module

The push streaming module is mainly divided into the collection of audio and video data. If it is a show-like live broadcast, it can be used as a beauty filter related function to improve the picture quality and user experience of the live broadcast. Finally, the volume of the audio and video data is reduced by encoding and compression. , Finally, the data is delivered to the RTMP server in a fixed format through the streaming media transmission protocol, so that the entire push-streaming terminal is completed.

RTMP server module

In the traditional sense, the RTMP server may actually only have the function of transcoding, which converts the data passed by the push end into data files in network formats such as flv, which is convenient for viewing at the playback end. However, cloud providers currently provide a complete set of solutions. , Such as definition transcoding, content health check, live cover generation, data statistics, recording and playback functions, which are also business encapsulation on the basis of the RTMP server, so as to provide a complete set of solutions.

Player module

The logic of the player is relatively simple. In short, it is to obtain the streaming address and play audio and video. However, in the actual development process, the business workload and technical optimization points of the player are the most, as shown in the figure above. The first screen opens in seconds, decoding optimization, switching live room and other functions, all require a lot of energy, and continue to evolve and optimize according to the business.

Three, build steps

This simple tutorial for introductory live broadcast is mainly divided into the following modules:

Build a live server;

Use OBS for streaming;

How to watch the live stream;

The realization of the news in the live broadcast room.

3.1 Set up live broadcast server

The live broadcast server parses and encodes and decodes the video stream uploaded by the push streaming terminal in real time for viewing at the viewing terminal that supports live broadcast protocols such as rtmp, hls, or httpflv.

There are many open source live server solutions on the market, such as livego, srs and nginx-rtmp, or the current mainstream cloud solutions. Currently, Alibaba Cloud, Qiniu Cloud, Tencent Cloud, etc. all provide standard mature Solution, this article aims to quickly build a simple live broadcast, so we can use the open source method of livego to build a push-pull streaming server, livego is written in pure go language, high performance and cross-platform, installation and use are very It is simple, supports commonly used transmission protocols, file formats and encoding formats, or installs as shown above, and directly broadcasts live broadcast services in cloud providers.

There are three main ways to install livego: 1) directly download the binary executable file; 2) start from Docker; 3) compile from source code.

docker run -p 1935:1935 -p 7001:7001 -p 7002:7002 -p 8090:8090 -d gwuhaolin/livego

Among them, the meaning of each port is as follows:

8090: HTTP management access monitoring address

1935: RTMP service listening address

7001: HTTP-FLV service listening address

7002: HLS service listening address

3.2 Use OBS to push streaming

OBS (Open Broadcaster Software) is an open source and free software that provides video recording and live broadcast functions. Go to OBS official website download the software for the corresponding platform and install it.

To push the stream, the first thing to solve is the "what to push" problem, that is, to clarify the source of the stream. Open OBS and click the New "Source" button. As shown in step 1 in the figure below, you can see that OBS supports a variety of sources, including media sources, monitor capture, browser and window capture, and so on. Here, the existing mp4 file is used for circular push, so the source is selected as "media source", and the name is the default. After clicking "OK", set the video file to be played, and then click "OK".

Then, what needs to be solved is the "where to push" problem, that is, an available push address is needed.

We have set up the livego live server before, and it provides a default push URL: rtmp://localhost:1935/live. The push URL of a standard RTMP server is similar to this format: rtmp://domain/AppName /StreamName, but if you want to use the streaming address, you need an authorized channelkey.

By visiting http://localhost:8090/control/get?room=movie you can get the channelkey used for streaming, as shown below, where the data field is the channelkey obtained this time.

{
    "status": 200,
    "data": "rfBd56ti2SMtYvSgD5xAV0YU99zampta7Z7S575KLkIZ9PYk"
}

Up to now, both the streaming address and channelkey are available. You only need to configure the relevant settings in OBS to perform the streaming. First click the "Settings" button of "Controls" to enter the settings panel.

Then, select the "Push Stream" option. Select "Custom" for the service, set the server to: rtmp://localhost:1935/live, and set the streaming key to the channelkey: rfBd56ti2SMtYvSgD5xAV0YU99zampta7Z7S575KLkIZ9PYk obtained earlier. After setting, click the "Start Streaming" button of "Controls" to start streaming.

Under normal circumstances, the default output configuration is sufficient for most scenarios, but if you want to get a live broadcast effect that is more suitable for you, you can set the "Advanced" output mode in the "Output" option. If you don't need this, You can skip this part directly. As shown in the figure below, in the advanced output settings interface, you can configure streaming, recording, audio, and playback buffers. Among them, the most important is the streaming settings. Encoder software can choose x264 and QuickSync H.264, just use the powerful x264. "Rescale output" can set the output resolution, and the original video resolution is used by default.

Bit rate (bit rate) means the amount of data per second after the video is compressed and encoded. The unit is Kbps, where K=1000. The larger the value, the larger the video data stream pushed per second, and the higher the video quality, but it also occupies more bandwidth, which can be adjusted according to needs. Generally, 2000~2500Kbps is usually used for live show and game live broadcast is possible. The bit rate requirements are higher, and you can make corresponding adjustments.

When live streaming, you can use a variety of rate control methods, mainly CBR, ABR, VBR and CRF.

CBR (Constant Bitrate) constant bit rate , the bit rate remains basically constant within a certain time range. When using this mode, the image quality will deteriorate in scenes with more dynamic video images, and the image quality will become better in scenes with more static images.

VBR (Variable Bitrate) variable bit rate , its bit rate can vary with the complexity of the image. When using this mode, in scenes with relatively simple image content, less bit rate is allocated, and in scenes with complex image content, more bit rate is allocated. This not only guarantees the quality, but also takes into account the bandwidth limitation, giving priority to the image quality.

ABR (Average Bitrate) The average bit rate is , which is an interpolation parameter of VBR. Simple scenes are allocated a lower bit rate, and complex scenes are allocated a sufficient bit rate, which is similar to VBR. At the same time, the average code rate within a certain period of time is close to the set target code rate, which is similar to CBR. It can be considered that ABR is a compromise between CBR and VBR.

CRF (Constant Rate Factor) constant rate coefficient . The CRF value can be understood as a fixed output value expected for the clarity and fluency of the video, that is, whether it is in a complex scene or a simple scene, a stable subjective video quality is desired.

The key frame interval (Group of Pictures, GOP) refers to a sequence of frames consisting of an I frame, multiple P frames, and B frames. A frame is a picture in the video, where:

I frame (intra coded picture) : The most complete picture, with all the information, it can be decoded without referring to other frames. Each GOP starts with an I frame;

P frame (predictive coded picture) : inter-frame predictive coded picture, it needs to refer to the previous I frame or P frame to decode, and the compression rate is high;

B frame (bipredictive coded picture) : Bi-directional predictive coded picture, the previous frame and the subsequent frame are used as the reference frame, and the compression rate is the highest.

For ordinary video, increasing the GOP length is beneficial to reduce the video volume, but in a live broadcast scenario, the GOP is too large, which causes the client's first screen playback time to become longer. The smaller the GOP, the higher the picture quality. It is recommended to set it to 2 seconds, and the longest should not exceed 4 seconds.

3.3 Live streaming viewing

We have just set up an RTMP server, and use the current mature and feature-rich streaming tool OBS for streaming. Next, we will solve the problem of how to watch it on the user terminal.

FLV (Flash Video) is a network video format and a streaming media format. At present, most of the streaming media formats used by some mainstream live broadcast networks are flv, which can be played without installing any plug-ins.

3.3.1 Small test: use VLC tool to watch

VLC is an audio and video player that can play local media or media on the Internet. to the official website 1609b519159b0a https://www.videolan.org/index.zh.html download the corresponding installation package and install it.

Click the "Open Network Streaming" option under the "Media" tab, and set the network address to: rtmp://localhost:1935/live/movie, click "OK" and you can see the OBS streaming video.

The use of VLC is mainly to facilitate the development of students to watch and test, such as viewing the problem of freezing, viewing the resolution, and locating the delay problem. VLC is a relatively professional tool that can facilitate us to locate and solve the problem.

3.3.2 Use flv.js for browser-side viewing

flv.js is the most popular html5 pure javascript, and it is also the solution for the mainstream domestic browser terminals to play flv format. In this section, we will use flv.js for simple playback. Open the following website: http ://bilibili.github.io/flv.js/demo/ .

Can be seen as shown, the input box below the streamURL http://127.0.0.1:7001/live/movie.flv later, after the switch to click to play MediaDataSource Load following screen.

3.3.3 Brief introduction of live broadcast protocol

So far, we have successfully built a small RTMP framework and understood the complete process of the entire push-pull stream. Next, we need to have an introductory understanding of several live network transmission protocols that are strongly related to the RTMP protocol.

There are several common domestic live broadcast protocols:

RTMP

HLS

HTTP-FLV

HLS stands for HTTP Live Streaming. This is a live streaming protocol proposed by Apple. Currently, both IOS and higher versions of Android support HLS. The two main contents of HLS are .m3u8 files and .ts playback files. The receiving server will buffer the received video streams, and then after buffering to a certain extent, these video streams will be encoded and formatted, and a .m3u8 file and many other .ts files will be generated at the same time. The advantage of HLS is that The platform is relatively good. HTML5 can be directly opened for playback. The mobile terminal has good compatibility. The disadvantage is also obvious, that is, the delay is relatively high. If some live broadcasts, such as live broadcasts with low interactivity, you can use this protocol. The HLS network transmission format is Very suitable for on-demand scenes.

RTMP stands for Real Time Messaging Protocol, that is, real-time messaging protocol. For developers, we first make clear that RTMP is an application layer protocol, and the bottom layer is the TCP transmission protocol used. Here we know that RTMP is a protocol for audio and video related fields Therefore, the use of TCP as the main transport layer protocol also leaves a lot of room for the various evolutions of the subsequent RTMP network. In the live broadcast industry, especially on the streaming end, the RTMP protocol is a veritable overlord. , Basically all mainstream live broadcast websites support the rtmp protocol for streaming. For the specific protocol details of RTMP, follow-up articles have specific analysis.

FLV (Flash Video) is another video format launched by Adobe, which is a storage container format for streaming media data transmitted on the network. Its format is relatively simple and lightweight, and does not require large media header information. The entire FLV is composed of The FLV Header, The FLV Body and other tags. Therefore, the loading speed is extremely fast. The suffix of the file encapsulated in FLV format is .flv.

Streaming media protocol RTMP, HTTP-FLV, HLS simple comparison:

3.3.4 Messages in the live broadcast

In the live broadcast system of the show, if the realization of audio and video functions is to dress up the live broadcast with a gorgeous new appearance, then the realization of the message system in the live broadcast system is the soul of the entire live broadcast under the gorgeous new decoration. How to build high availability The live broadcast room message system is also a problem that every live broadcast system must solve.

Before designing the message system for the show live broadcast, we need to briefly sort out the message types in the live broadcast room.

Notification messages such as gifts, barrage, entry, list changes, level changes, and so on. Their feature is to notify users of events in the live broadcast room, create a live broadcast room atmosphere, and enhance the user's experience of watching the live broadcast.

Functional messages such as kicking people, anti-spam review, red envelopes, PK messages, and so on. The feature of this type of message is to assist in the development of live broadcast services, and to connect the three roles of the starter, the viewer, and the server in the process.

From a business perspective, we can analyze all kinds of news in the live broadcast room. Although because of the various business forms, the final presentation is also colorful and gorgeous, but we can analyze all kinds of news display forms. The news is from the development From a perspective, there are the following characteristics. We divide according to whether the message can be discarded and real-time. We can classify all business messages into the following categories:

In the live broadcast system, there are more message signaling communications in the show live broadcast and live broadcast with goods, mainly due to the nature of the business. The interactive nature of the two types of live broadcasts is relatively compared. Strong, and the gameplay is more diverse. According to the classification in the above picture, the discardability and real-time requirements of each business message are different. Therefore, when developing the message system, it is also necessary to prioritize the messages and sort the messages. The real-time nature of the distribution also needs business performance considerations.

I just made a business-related explanation on the two attributes of live broadcast messages, real-time and non-discardability. However, for live broadcast messages, the first element is stability. How to accurately and stably distribute messages to designated live broadcast rooms is also One of the issues we need to consider is the realization of the distribution of live messages. Generally speaking, it can be divided into two ways. The first is to rely on the instant messaging between the live broadcast rooms (Instant Messaging), which is what we often call the IM messaging system. The second is to rely on http short polling. For example, the client requests the server every 1 second, and the server returns the incremental message information that occurred within this second. The client obtains the incremental information, and then according to the specific message service Type, and then perform relative business page UI rendering, so that’s it. Technically speaking, one is a "push" model and the other is a "pull" model. Today, because we are building a simple live room message system, we use it first A simple "pull" model for simple implementation.

Basic realization idea: The client calls the server interface according to the id of the live broadcast room every very short time, such as 1 second or less, and polls the messages that occur in the live broadcast room. We use the server here Redis's SortedSet data structure to store the message, where key is the room id of the live room, score is the timestamp generated by the server receiving the message event, and value can simply directly store the serialized string of the message, so that you can follow To store messages in chronological order, and configure the deletion logic of expired messages, the entire message storage can be simply set up.

The message storage is shown in java pseudo-code:

 long time = new Date().getTime();
 
try {
     // redis中插入消息数据
     jedisTemplate.zadd(V_UNIQUE_ROOM_ID, time, JSON.toJSONString(roomMessage));
 
     // 按照概率性的去删除redis中过期的消息数据
     if (probability()) {
           deleteOverTimeCache(V_UNIQUE_ROOM_ID);
        }
     } catch (Exception e) {
            log.error("message save error", e);
 }

You can see the message storage. If you use redis sortedSet for storage, it is more convenient. The next thing we need to deal with is the deletion of expired messages in redis, because invalid expired messages are of no value (all messages can be stored persistently) , If there are too many messages stored in a single key in redis, it will also lead to slow search of messages and continuous increase in memory usage. This is what we don't want to see. Because it is sample code, we simply deal with it. Delete logic.

    private void deleteOverTimeCache(String roomId) {
 
        Long totalCount = jedisTemplate.zcard(roomId);
 
        log.info("deleteOldTimeCache size is {}", totalCount);
 
        if (totalCount < 600) {
            return;
        }
 
        // 倒序删除过期数据
        Set<Tuple> tuples = jedisTemplate.zrangeWithScores(roomId, -601, -1);
 
        if (CollectionUtils.isNotEmpty(tuples)) {
            for (Tuple tuple : tuples) {
                // 这是第一个-600条的那个score
                double score = tuple.getScore();
                jedisTemplate.zremrangeByScore(roomId, 0d, score);
                break;
            }
        }
    }

The above pseudo code probability() first makes a probabilistic judgment. For example, we make a random judgment of one percent to judge whether the request is to delete the message (please note that the logic of our deletion is placed in the logic of insertion Among them. If you need to determine whether to delete expired data every time you insert, it will affect the performance of the insert). If the probabilistic judgment is passed, we will give priority to judging the number of messages in a certain live broadcast room. If the number of messages is still relatively small, then exit the deletion logic. If the message threshold is exceeded, the expired messages will be deleted in reverse chronological order.

After talking about the storage of HTTP short polling messages, let's finally briefly talk about the client-side message query implementation logic. The client requests the server to query the message in the live broadcast room through the two fields of the live broadcast room id and timestamp. The "timestamp" is returned by the server every time. This timestamp is progressive. The next time the client requests the service When the data on the end, it will bring the timestamp returned by the last server. The pseudo code is as follows:

   @Override
 public RoomMessage queryRoomMessages(MessageMessageReq messageMessageReq) {
 
        RoomMessage result = new RoomMessage();
 
        long timestamp = messageMessageReq.getTimestamp();
 
        Set<Tuple> tuples = null;
        if (timestamp == 0) {
            // 如果传递是0，说明这个客户端终端是第一次来轮询，我们只要返回一个最近最新的消息返回即可
            tuples = jedisTemplate.zrevrangeWithScores(UNIQUE_ROOM_ID, 0, 0);
        } else
            // 加上一毫秒，返回后续的消息，每次返回5个，防止客户端因为低端手机原因，过多的消息渲染不出来
            tuples = jedisTemplate.zrangeByScoreWithScores(UNIQUE_ROOM_ID, timestamp + 1, System.currentTimeMillis(), 0, 5);
        }
 
        List<EachRoomMessage> eachRoomMessages = new ArrayList<>();
        long lastTimestamp = 0L;
 
        if (!CollectionUtils.isEmpty(tuples)) {
            for (Tuple tuple : tuples) {
                //最后一次循环后，会把最后一条消息产生的时间戳，返回给客户端，这样下次客户端就可以拿着这个时间戳来进行查询
                lastTimestamp = new Double(tuple.getScore()).longValue();
                eachRoomMessages.add(JSON.parseObject(tuple.getElement(), EachRoomMessage.class));
            }
        }
 
        result.setTimestamp(lastTimestamp);
        result.setEachRoomMessages(eachRoomMessages);
        return result;
    }

The above three relatively complete codes mainly state the ability of a live broadcast room that relies on http short polling to be implemented quickly. This method is relatively rough, but it is a good idea for implementation. At present, our online part The business also implements some modules based on this polling idea.

There is also a pitfall in this way of thinking. If you use this idea to achieve it, you can try to circumvent it. If the Android client is disconnected from the network, the polling thread will not stop, for example, the network is completely disconnected at 8 o'clock in the evening, and the network is restored at 8:01. When the network is restored, the first polling will be It will cause the server to return a large number of messages, which need to be processed here, otherwise too many messages will be returned, the server will also have a slow check, and the client will also have a flash jump in some message display intervals due to the rendering of expired messages. . For example, various messages may appear "crazy" in the public screen area. These can be circumvented by the agreement between the client and the server. For example, when a network problem occurs on the client, the timestamp can be set for more than 5 seconds. If the value is 0, the server is required to return the latest live broadcast message, and the messages lost in the middle can be discarded in the service return.

Four, summary

This article mainly wants everyone to have a preliminary understanding of live broadcast, understand the basic conceptual model of live broadcast, and some basic concepts. In the follow-up, we will learn more about the specific modules of live broadcast to further understand the principle of live broadcast, which can also help us better. Do a good job of live broadcast business.

Author: vivo Internet server team-Li Guolin