Starting from the live broadcast hardware, analyze the intelligent trend from live broadcast technology to IoT devices

Preface

There are personal live broadcasts (pan-entertainment live broadcasts) and corporate live broadcasts (commercial & business live broadcasts); personal live broadcasts have low investment costs, and can be broadcast anytime and anywhere with a mobile phone, and each live broadcast App has enough live broadcast auxiliary tools to assist The start of the broadcast, the demand for value-added services is not high; the enterprise live broadcasts are standard, there is a dedicated live broadcast room, the investment of manpower and material resources is large, the live broadcast has a wide range of influences, and the professional live broadcast hardware, PaaS, SaaS and other software services, operation guidance, and after-sales service Various value-added services have clear demands, so this analysis is based on TO B's enterprise live broadcast business.

Author: Qin Mi

1. Live broadcast market research

background

The world economy is slowing down due to the epidemic, fierce competition among companies, and the core demands of companies to increase revenue and reduce expenditures, and reduce costs and increase efficiency are getting stronger. The digital economy is also developing rapidly and gradually becoming a new economic growth driver; corporate live broadcasts can promote companies The value enhancement of multiple links such as production and operation, control and cost saving, help enterprises to spread, reach more users, and accelerate the digital transformation of enterprises.

Source: iResearch, "Research Report on the Development of China's Enterprise Live Broadcast Service Industry in 2021"

Live stream participants

1. C-end users: information acquisition, leisure and entertainment, shopping, etc.;
2. B-end enterprise: training, conference, operation, marketing, etc.;
3. Live broadcast service providers: live broadcast hardware, software, cloud services, data services, marketing services, office collaboration, etc.

The live broadcast scene of the B-end enterprise

From education, finance, automotive, medical, Internet, media, home furnishing, real estate and other vertical industries, from small businesses with dozens of people to large enterprises with tens of thousands of people, corporate live broadcasts are rapidly applied; corporate live broadcasts are internal, And external dual demand; internal corporate training, conferences and exhibitions, corporate publicity; external product introductions, live events, e-commerce delivery, etc.

B-end companies' demands for live broadcast

Compared with public cloud, enterprises hope to integrate the hybrid cloud model of public cloud and private cloud to obtain low-cost functional iteration and stable performance of public cloud while maintaining the ability to customize the business; both PaaS and SaaS features The aPaaS service can better meet the needs of companies to build live broadcast systems (even companies build their own video middle stations). The low-code, scalable, and flexible capabilities provided by aPaaS allow companies to better customize their live broadcast services. ; At the same time, low latency, ultra high definition, high concurrency, and high security are important indicators to measure user experience.

The architecture and ecology of live broadcast service providers

Live broadcast service providers provide one-stop solutions for upstream, midstream, and downstream. At the same time, they continue to integrate enterprise-level services such as CRM, HRM, collaborative office, and data services, so that live broadcast services and other enterprise-level services have a synergistic effect. The business is the core enterprise service ecosystem.

Source: iResearch, "Research Report on the Development of China's Enterprise Live Broadcast Service Industry in 2021"

Where are the IoT positioning and opportunities?

Based on its own positioning and technical advantages, IoT is more likely to provide basic capabilities such as CDN distribution, intelligent review, transcoding, and on-demand broadcasts in the upstream. At the same time, it focuses on intelligent live broadcast hardware, live PaaS (iPaaS, aPaaS), and data in the midstream. Research and development in analysis services, etc.; integrate DingTalk office services and Umeng marketing services in the ecological association to provide live broadcast service providers with competitive core capabilities, first TO B and then TO C, and output competitive core software and hardware Ability to empower live broadcast service providers.

2. Live broadcast related technology

The technology related to live broadcast involves basic push-pull streaming and corresponding protocols on the device side, cloud transcoding, auditing, watermarking and other functions; it also includes some AI algorithm capabilities running on the device side and the cloud.

Live streaming service process

1. Host: The collected and packaged audio and video live streams are pushed to the live service center by the push SDK;
2. Server: The pushed video stream is accelerated by the CDN distribution node to ensure the stability of uplink transmission;
3. Server: After the video stream is pushed to the multimedia live broadcast center, the video stream can be transcoded, recorded, screenshot, and security audited as needed;
4. Audience side: The processed video stream is sent to the audience's equipment for playback through the CDN distribution node;
5. Audience: Live video can also be recorded in the cloud and transferred to the on-demand system to provide on-demand services.

Multimedia framework

FFmpeg：

• A complete open source solution that integrates recording, conversion, audio/video encoding and decoding functions;
• Support video capture, video format conversion, video capture, video watermarking, audio/video conversion into stream;
• It can push the video of the camera and the audio of the microphone in the specified format (such as FLV) to push the RTMP protocol;
• Support protocols: HTTP, RTP, RTSP, RTMP, etc.

core module:

• Read/write device data module: libavdevice;
• Audio/video codec module: libavcodec (codec = enCode (encoding) + deCode (decoding));
• Audio/video format packaging and unpacking module: libavformat;
• Player (use ffmpeg to parse and decode, you can use pull streaming to play): ffplay.

encoding (compression processing, convenient for storage and transmission) format:

• Video: H.262, H.264, H.265 (with copyright); VP8, VP9, AV1 (without copyright);
• Audio: MP3, AAC, etc.;
• Common encoders:
  Video: libx264 (H.264 format), libx265, libvpx (VP8, VP9 format), libaom (AV1 format);
  Audio: aac (AAC format)

push-pull flow process:

hardware acceleration codec:

• FFmpeg supports hardware-accelerated encoding and decoding through hwaccel cuda, h264_cuvid, hevc_cuvid and h264_nvenc, hevc_nvenc module (in the case of NVIDIA GPU). The hardware codec can produce the same output as the software codec, but may use less power and CPU;
• Hardware codec uses GPU processing power, saves power, has fast performance, shares CPU pressure and reduces CPU heat. Because the chip performance and interface parameters of different hardware platforms are different, hardware manufacturers need to adapt at the hardware and software levels; software editing The decoding does not use GPU, and the encoding and decoding is mainly carried out by CPU, which consumes power and has poor performance, but has strong compatibility with various hardware platforms; general live broadcast platforms will adapt to the specific configuration of the device to ensure the live broadcast effect.

Commonly used live broadcast protocol

RTMP（Real Time Streaming Protocol）

• Adobe real-time message transmission protocol, based on TCP long connection, supports continuous video stream in FLV format (received data is immediately forwarded);
• Low latency (1~3S), but no player that natively supports RTMP is provided on the iOS platform;
• The 1935 non-public port is used by default, which is easy to be blocked by the firewall, and a third-party decoder is required on the iOS side;
• It is suitable for use on both the push and pull ends.

HTTP-FLV（HTTP Flash Video）

• RTMP based on HTTP long connection, low latency (1~3S), support HTML5;
• Encapsulate audio/video data into FLV format, and then transmit it via HTTP protocol;
• Combines the low latency of RTMP and a streaming protocol that reuses existing HTTP distribution resources;
• Real-time performance is equal to RTMP. Compared with RTMP, it saves part of the protocol interaction time, the first screen time is shorter, and there are more expandable functions;
  HTTP-FLV is generally used for streaming playback.

HLS（HTTP Live Streaming）

• Apple video live broadcast technology supports sliced video streams in m3u8 format;
• Collect and cut the streaming media for a period of time on the server, generate continuous ts slice files, and update the m3u8 index;
• High latency (5~20S), based on HTTP short connection, need to continuously establish connection with the server, support HTML5;
• Because it is based on HTTP/80 transmission, it can effectively avoid firewall interception;
• HLS is generally used for streaming playback or review.

Live related algorithms

live content review (cloud side)

• Due to the huge amount of content and the high cost of manual review, AI review can be filtered based on pornography, violence, terror and other content.

personalized recommendation (cloud side)

• For active search or personalized recommendation based on scene recognition, relevant content recommendations are based on real-time interests.

advertising marketing (cloud side)

• Advertisement promotion based on live video content, scenes, etc., and commercialization of live content.

Live beauty (device side)

• Live beauty (whitening, dermabrasion, stickers, filters), beauty (key point detection, facial features adjustment), etc.;
• The equipment-side algorithmic reasoning needs to take into account the real-time requirements while meeting the effect, and at the same time adapt to different hardware configurations.

3. Appeals for all-in-one live streaming

Traditional mobile phone-based live broadcasts have various problems. There are always demands on the market for professional smart devices such as live broadcast all-in-ones, which are used to control and improve the efficiency of the whole process of live broadcast activities.

Problems with traditional live broadcast props

1. Mobile phone live broadcast is hot and freezes, the live broadcast screen is small, it is inconvenient to interact with fans, and it cannot be extended to other high-end professional live broadcast external devices;
2. Equipped with mobile phones, computers, large-screen same-screen devices, broadcast directors, encoders, audio consoles and other equipment for live broadcast, coordination is troublesome, and the operation is too complicated;
3. Complex equipment coordination requires multiple people to participate in a live broadcast, which is expensive in labor costs;
4. The decoration of the live room requires additional costs such as design, materials, and labor, and cannot be switched at will to change the background of the live room;
5. It cannot support multiple cameras at the same time, and multiple cameras cannot work together to achieve far/near shots, product close-ups, picture-in-picture and other effects.

Live broadcast all-in-one can integrate functions

1. Multi-function in one: integrated director, encoder, capture card, monitor, audio console, same screen device, teleprompter in one;
2. Real-time matting function: One-click to switch the live broadcast background, eliminating the need for decoration in the live broadcast room; support pictures, PPT, and video as the live broadcast background;
3. Multi-camera switchable: panoramic, close-up, close-up camera positions, which can present the selling point of live broadcast from multiple angles;

4. Support multi-camera: It can support HDMI, USB and other multi-camera access;
5. Support one-click start broadcast: Built-in mainstream live broadcast platforms such as Taobao Live, Douyin, Kuaishou, Xiaohongshu, etc., can start broadcast with one click;
6. Multimedia interface expansion: SLR, camera, mirrorless single, microphone, sound card, keyboard and mouse and other multimedia equipment can be expanded;
7. Remote OTA capability: System, function (including built-in App upgrade, newly installed App, etc.), performance and other upgrades can be upgraded remotely by OTA.

The software and hardware configuration of the live broadcast all-in-one

hardware device:

• All-in-one broadcasting machine (Android system, with CPU and GPU capabilities);
• Camera (supports multiple cameras).

auxiliary equipment:

• Lighting, wireless microphone, green screen keying cloth.

software suite:

• Integrate green screen cutout, live patch, multi-channel director, picture-in-picture, watermark, camera switch, audio/picture quality adjustment, teleprompter and other capabilities;
• The software can be split into a basic software package and an auxiliary package for advanced live broadcast functions.

business model:

• The hardware is purchased at one time, and the software package can be charged annually for service.

From the demand of all-in-one live broadcasting to the intelligent popularization of IoT devices

Due to the shortage of mobile phone live broadcast, the demand for integrated live broadcast machines has promoted the demand for a complete live broadcast integrated solution based on live broadcast hardware, and this is only the tip of the iceberg in the current thousands of industries. At present, IoT (physical network) has developed rapidly in the two directions of AIoT (Artificial Intelligence Internet of Things) and IIoT (Industrial Internet of Things); especially in the field of AIoT, it has penetrated into communities, parks, cities, agriculture, commerce, etc. And gradually form specialized solutions for each subdivision of end, edge, pipe, and cloud integration.

Hardware intelligence is the cornerstone of these solutions. The intelligent hardware is based on the principle of nearby consumption of data and nearby computing power, and collaborative interconnection with edge computing and cloud computing, which promotes the accelerated popularization of AIoT; in the future, it will be on the end side. , There will be 12 billion IoT devices and 150 billion embedded devices on the edge, and these will be fertile ground for intelligence.

Starting from the familiar live broadcast business, this research disassembles the requirements and composition of the intelligent hardware equipment of the live broadcast all-in-one. There are already many such intelligent hardware, such as cloud POS machines, cloud projection machines, electronic smart scales, AI cameras, Vlog interactive screens, security edge integrated machines, commercial passenger flow analysis integrated machines, industrial defect detection integrated machines, etc. Make intelligent transformations on various previously existing equipment one by one.

The era of IoT will never come, because it is constantly coming every day from now on. The future will include chip hardware, operating systems, algorithm inference, data analysis, engineering integration, cloud monitoring & operation and maintenance, etc. A huge number of opportunities are waiting for everyone to explore together.

"Video Cloud Technology" Your most noteworthy audio and video technology official account, pushes practical technical articles from the front line of Alibaba Cloud every week, and exchanges and exchanges with first-class engineers in the audio and video field. The official account backstage reply [Technology] You can join the Alibaba Cloud Video Cloud product technology exchange group, discuss audio and video technologies with industry leaders, and get more industry latest information.