MindSpore Lite, an all-scenario AI inference engine, helps HMS Core video editing service to create a smarter editing experience

The development of the mobile Internet has brought great changes to people's social and entertainment methods, and emerging cultural styles represented by vlogs and short videos are being favored by more and more people. At the same time, with the application of AI intelligence, beauty retouching and other functions in image and video editing apps, video editing efficiency and video effects have been greatly improved, and video application scenarios have become more abundant.

The current editing products have various functions and rich materials, but the development cycle is long and the threshold is high. In order to make the editing software more intelligent, easy to use, and improve the efficiency of developers, HMS Core 6 provides developers with a video editing service (Video Editor Kit), which provides one-stop video import, editing, rendering, export, and media asset management. video processing power. In addition to supporting complete traditional video editing functions, the video editing service also provides rich AI processing capabilities such as exclusive filters, character tracking, and one-click hair coloring to assist in video creation, bringing users more creative inspiration and creating more intelligent Clipping experience.

Figure 1. Display of exclusive filters, character tracking, and one-click hair coloring effects based on AI capabilities

Diversified intelligent video processing capabilities are achieved by neural network models. Due to the large size of the trained model files (the size of a single model is generally a dozen or even tens of megabytes), and the ROM and RAM space of mobile phones and other devices is limited, How to provide developers with richer intelligent video processing capabilities with less space occupation of terminal devices has become a major challenge for mobile application video editing.

To solve the above challenges, the HMS Core video editing service chooses to use Huawei's self-developed AI framework MindSpore Lite for neural network model inference. MindSpore Lite is an all-scenario AI inference engine that supports rapid deployment in different environments of devices, edges, and clouds through a unified API interface. It supports HarmonyOS, Android, iOS, Windows and other operating systems, and supports Ascend, GPU, CPU (x86 , arm...) and other hardware execution. In addition to supporting the model formats trained by MindSpore, MindSpore Lite also supports the conversion and inference of third-party model formats such as TensorFlow, TensorFlow Lite, Caffe, and ONNX.

Figure 2. MindSpore Lite Architecture Diagram

MindSpore Lite provides a high-performance and ultra-lightweight solution for AI model inference: Through efficient kernel algorithms and assembly-level optimization, as well as heterogeneous scheduling of CPU, GPU, and NPU, hardware computing power can be fully utilized to minimize inference time. Delay and power consumption; provide model quantization compression technology, adopt Post-Training Quantization (PTQ), and directly map weight data from floating point to low-bit fixed-point data without a data set, effectively reducing the size of the model. Helps the deployment and execution of AI models in resource-constrained environments.

Figure 3. Introduction to the principle of quantization technology

The quantization for weight data supports both fixed-bit quantization and mixed-bit quantization. Fixed-bit quantization adopts Bit-Packing method, supports weight quantization of 1-16 arbitrary bits, meets the requirements of users in different compression scenarios, and automatically selects the appropriate coding strategy for compression coding according to the data distribution after model quantization So as to achieve the best compression effect.

Figure 4. Fixed-bit quantization compression

According to the different sensitivities of different layers of the neural network to the quantization loss, the hybrid bit quantization adopts the mean square error as the optimization target, and automatically searches for the most suitable bits for the current layer, and achieves a greater compression rate while ensuring the accuracy. At the same time, for the quantized model, finite state entropy (Finite State Entropy, FSE) is used to further compress the quantized weight data by entropy coding, so as to realize the efficient compression of the model, improve the model transmission rate and reduce the model storage space.

Figure 5. Hybrid bit quantization compression

In addition, the Bias Correction method will be used during quantization to minimize its quantization error. Bias Correction will calibrate the weight data during inverse quantization according to the inherent statistical characteristics of the weight data, so that the weight value has the same expectation and variance before and after quantization, which can greatly improve the accuracy of the model.
The AI model in the video editing service adopts the mixed-bit quantization method provided by MindSpore Lite, which finally achieves an average model compression effect of 5x+ while ensuring the accuracy. It solves the difficult deployment problem caused by too many models and too large files.

Figure 6. The quantization effect of the video editing model (from the measured data of MindSpore Lite)

By quantifying and compressing AI models, under the premise of the same ROM space occupation, it is guaranteed that more AI models can be deployed in editing products, and AI capabilities can be fully utilized to provide more special effects application scenarios, making editing functions more powerful and intelligent. . After Huawei's official editing software Petal Editing is connected to the video editing service capabilities, users can use AI video editing functions such as exclusive filters and character tracking (some features will be opened with the upgrade of Petal Editing App) to make video editing more convenient and richer. fun.

MindSpore Lite is committed to creating a high-performance, ultra-lightweight all-scenario AI engine. In addition to high-performance kernel algorithms, hardware heterogeneous scheduling, and quantitative compression, it also provides one-stop training and reasoning capabilities for device-cloud collaboration. The HMS Core video editing service is based on MindSpore Lite, which helps developers create easier-to-use and smarter editing tools.

For more information, please visit the official website

Huawei Developer Alliance HMS Core official website

MindSpore official website

MindSpore Open Source Community