Analysis of key features of OpenHarmony 3.1 Release version - OpenHarmony's new audio and video engine - HiStreamer

OpenAtom OpenHarmony (hereinafter referred to as "OpenHarmony") is an open source project incubated and operated by the OpenAtom Foundation (OpenAtom Foundation). The framework and platform of the operating system promote the prosperity and development of the Internet of Everything industry. In the era of the Internet of Everything, we have created a new audio and video engine, HiStreamer, for OpenHarmony.

1. HiStreamer generates background

Digital multimedia technology has developed rapidly in the past decades, music, telephone, television, movies, video conferencing, etc., accompany us every day. In order to provide users with rich multimedia processing capabilities, the industry already has relatively mature audio and video engines, such as the open source audio and video engine GStreamer. Why are we building a new audio and video engine?
With the advent of the Internet of Everything era, more and more intelligent devices appear in our lives. For example: smart refrigerators can tell people that the stored dishes are about to expire through the screen and sound; smart gates can automatically complete ticket checking through face recognition; smart door locks can improve the efficiency and security of unlocking through voice and video. .....
Different from standard (Standard) devices such as PCs and mobile phones, many intelligent devices have relatively weak CPU processing capabilities and relatively small memory, and traditional audio and video engines cannot support such devices . HiStreamer came into being, which supports both lightweight Mini/Small devices and Standard devices (currently supports some functions) . HiStreamer is constantly developing and improving, and will support more functions of Standard devices in the future.

2. "Pipeline + Plug-in" to achieve flexible deployment

In order to support Mini/Small/Standard devices, HiStreamer adopts the software architecture of pipeline (Pipeline) and plug-in (plugin), so that it can be deployed flexibly according to the hardware and requirements of the device. HiStreamer abstracts each process of audio and video processing into nodes, the output of the previous node is used as the input of the next node, and multiple nodes are connected to form a pipeline as a whole to complete the reading and decoding of audio and video data. The complete process of encapsulation, decoding and output. At the same time, plug-ins can provide rich extension functions for Pipeline nodes, making HiSteamer's audio and video processing capabilities more powerful.

1. Introduction to the Pipeline Framework In order to let everyone understand the Pipeline framework of HiStreamer, the following is an example of MP3 audio playback:
   The input is an MP3 file, and the output is the music played, and there are many steps in between.
   Let's take a look at the MP3 file structure first:
   

    图1 MP3文件结构

   MP3 files are composed of ID3 Metadata container header and several MP3 Frames (MP3 data frames). Each MP3 Frame is composed of MP3 Header (MP3 header information) and MP3 Data. This series of MP3 Frames is called ES Data (Element Stream Data).
   ● ID3 Metadata: Container header, mainly including title, artist, album, number of tracks, etc.
   ● MP3 Header: Contains MP3 Sync word (indicating the starting position of MP3 data frame) and MPEG version information.
   ● MP3 Data: Contains compressed audio information.
   To play MP3 files, firstly, you need to read in the MP3 file data, then remove the ID3 Metadata container header (ie decapsulation), then decompress a series of MP3 Frames into PCM (Pulse-Code Modulation) data, and finally drive the speaker to sound. This process can be abstracted into the following four nodes in sequence:

 图2 MP3音频播放的Pipeline

1. Input node (MediaSourceFilter): Read MP3 raw data and pass it to the next node.
2. Decapsulation node (DemuxerFilter): Parse the ID3 Metadata container header information, input it as a parameter of the subsequent node, and pass a frame of MP3 Frame (ie ES Data) to the subsequent decoding node.
3. Decoding node (AudioDecoderFilter): Decode ES Data into PCM data and pass it to the output node.
4. Output node (AudioSinkFilter): Output PCM data to drive the speaker to sound.
   As can be seen from the above examples, HiStreamer abstracts each process of audio and video processing into nodes through the Pipeline framework. These nodes are decoupled and can be assembled flexibly, so that different pipelines can be assembled according to business needs. At the same time, in order to make multiple nodes work better together, HiStreamer also supports automatic parameter negotiation between nodes.
   **
   2. Introduction to HiStreamer plugin <br>After understanding the pipeline framework of HiStreamer, let's take a look at the HiStreamer plugin.
   Many nodes of HiStreamer's Pipeline framework (such as input nodes, decapsulation nodes, decoding nodes, output nodes, etc.) support plug-in extensions. Through plug-ins, the functions of nodes become richer and more powerful.
   The application scenarios of plugins are very wide, such as:
   ● There are many media formats, and new formats will be produced in the future, and new media formats can be supported through plug-ins.
   ● Different OS platforms or devices have different processing methods, which can be supported by plug-ins.
   ● Different types of devices have different requirements and different CPU/ROM/RAM resources, which can also be supported by plug-ins.

3. Flexible deployment
HiStreamer is based on the software architecture of Pipeline and Plugin, which can be deployed elastically according to the hardware and requirements of devices.

 图3 HiStreamer弹性部署

As shown in Figure 3, Mini devices (such as speakers) have very weak CPU processing power, few ROM/RAM resources, and require fewer functions, only audio playback. HiStreamer can be configured to only support audio playback, select lightweight plug-ins, configure synchronous decoding mode, and reduce resource consumption. The Small device has stronger CPU processing capability and larger ROM/RAM space, and needs audio playback and video playback functions. HiStreamer can be configured to support audio and video playback, and select more powerful plug-ins.

3. HiStreamer logical architecture

After the above introduction, we understand the software architecture of HiStreamer's "pipeline + plug-in". Let's take a look at the detailed logical architecture of HiStreamer.

 图4 HiStreamer逻辑架构图

HiStreamer is mainly composed of HiStreamer engine and HiStreamer plug-ins.
Among them, the HiStreamer engine is divided into the following four layers:
● Service encapsulation layer: Implement player and recorder functions based on Pipeline encapsulation, simplifying the use of upper-layer applications.
● Pipeline framework layer: Provides implementation of Pipeline and several nodes (input, decapsulation, decoding and output), and supports connecting multiple nodes together to form Pipeline.
● Plug-in management layer: used for plug-in life cycle management, supports dynamic loading or static linking to use plug-ins.
● Tool library layer: Provides framework-dependent tools, isolates operating system differences, and provides debugging functions.
HiStreamer plug-ins are divided into platform software plug-ins and vendor hardware plug-ins:
● Platform software plug-ins: provided by the OpenHarmony platform, software algorithm plug-ins that can be reused across products.
● Manufacturer's hardware plug-in: a hardware-accelerated plug-in provided by the manufacturer, such as a hardware-accelerated codec plug-in.
Application developers can directly use ready-made plug-ins to implement multimedia functions, saving a lot of development time. The richer the plug-ins, the more powerful the audio and video processing capabilities of HiStreamer. Developers are welcome to participate in the development of HiStreamer plug-ins and enrich HiStreamer plug-ins together!

Fourth, HiStreamer plug-in development and examples

The following is an introduction to the development process and example explanation of the HiStreamer plug-in. Interested friends quickly learn and participate in the development of the HiStreamer plug-in~

1. Development of plugins
The development of HiStreamer plug-in is mainly divided into two parts: plug-in definition and function implementation.
(1) Plug-in definition
HiStreamer plugins are defined through the PLUGIN_DEFINITION macro. Taking the input plugin FileSource as an example, the definition code is as follows:

 std::shared_ptr<SourcePlugin> FileSourcePluginCreator(const std::string& name)
{
    return std::make_shared<FileSourcePlugin>(name);
}
Status FileSourceRegister(const std::shared_ptr<Register>& reg)
{
    SourcePluginDef definition;
    definition.name = "FileSource";
    definition.description = "File source";
    definition.rank = 100; // 100: max rank
    definition.protocol.emplace_back(ProtocolType::FILE);
    definition.creator = FileSourcePluginCreator;
    return reg->AddPlugin(definition);
}
//PLUGIN_DEFINITION传入四个参数
PLUGIN_DEFINITION(FileSource, LicenseType::APACHE_V2, FileSourceRegister, [] {});

When using the PLUGIN_DEFINITION macro to define the plugin (that is, the last line of code above), four parameters are passed in:
a) Plugin name: "FileSource" in the example.
b) License information: "LicenseType::APACHE_V2" in the example.
c) Plug-in registration function: "FileSourceRegister" in the example, this function describes the basic information of the plug-in, including the plug-in object creation function, and also calls AddPlugin to register the plug-in in the system.
d) Plug-in anti-registration function: can be passed as empty implementation.
(2) Function implementation <br>When implementing the plug-in function, it is necessary to inherit the corresponding plug-in interface class according to the type of plug-in to be implemented, and implement the relevant interface. For example, to implement the input plug-in FileSource, you need to inherit SourcePlugin and implement interfaces such as SetSource and Read. The code is as follows:

 // 定义FileSourcePlugin类继承SourcePlugin类
class FileSourcePlugin : public SourcePlugin {
// 实现SetSource接口, 设置要打开的文件路径
Status SetSource(std::shared_ptr<MediaSource> source) 
{
return OpenFile(source->GetSourceUri());
}
// 实现Read接口，它会读取数据用于后续处理
Status Read(std::shared_ptr<Buffer>& buffer, size_t expectedLen)
{
std::fread(bufData->GetWritableAddr(expectedLen), sizeof(char), expectedLen, fp_);
return Status::OK;
}
}

The complete code of the FileSource plugin can be referred to:
https://gitee.com/openharmony/multimedia_histreamer/tree/master/engine/plugin/plugins/source/file_source

2. Plug-in deployment <br>Use the HiStreamer plug-in defined by PLUGIN_DEFINITION, which can be a single-function plug-in or a plug-in package with multiple functions. Each such plug-in or plug-in package can be independently compiled into .a or .so, corresponding to the following two deployment methods:
● Static deployment: generally used on mini devices, the plug-in is compiled into a static library .a and linked to the system.
● Dynamic deployment: generally used on small/standard devices, the plug-in is compiled into a dynamic library .so, placed in the directory specified by the system, and loaded and run dynamically.

3. Plug-in operation <br>After the plug-in is developed and deployed to the system, the registration of the plug-in will be completed automatically when HiStreamer starts. The next step is to run the plugin.
To run a newly implemented plug-in, the operating conditions of the plug-in need to be met first. For example: FileSource will only run when playing local files; MP3 decoding plugin will only run when playing MP3 files...
Developers can check whether their own plugins are running through log information. If other plug-ins are registered in the system and cause your own plug-ins to fail to run, you can uninstall the plug-ins that cause interference. To uninstall a dynamically deployed plug-in, delete the corresponding .so; to uninstall a statically deployed plug-in, you need to modify the compilation script to cancel the compilation of the corresponding plug-in.

V. Conclusion

OpenHarmony welcomes developers to join the development of HiStreamer plug-ins, expand the media functions they want, and jointly enrich the HiStreamer media ecosystem!
At the same time, a preview to everyone: the next version of HiStreamer will enhance the Standard device with more functions, so stay tuned!
That's it for the introduction to HiStreamer in this issue.
For more HiStreamer information, please refer to:
https://gitee.com/openharmony/multimedia_histreamerhttps://gitee.com/openharmony/multimedia_histreamerhttps://gitee.com/openharmony/multimedia_histreamer