AI target segmentation capability for fast video cutout without green screen

Green screen cutout is a common technical method in the process of film and television production. It is often used to cut out and replace the background in the video, and realize more possibilities for video editing through post-processing. However, the production cost of green screen matting technology is time-consuming and labor-intensive, and it cannot be applied to daily life.

Huawei's video editing service has recently launched the target segmentation capability. It can refine and segment target objects in videos through AI intelligent matting. It is not limited to specific object categories, and can segment the subject and background in videos with clear subjects and relatively simple backgrounds. , a good processing effect can be achieved.

Application scenarios

The target segmentation capability is mostly used in live video, online education, forum meetings and other scenarios. For example, in the scene of live broadcast of selling goods, the background of the live broadcast can be replaced with the carousel screen of the product details page, so that users can understand the product in time; in online forums or video conferences, the background can be replaced with PPT content or office scenes to create learning Serious atmosphere of the office. Similarly, the target segmentation capability can also satisfy users' daily interesting editing. Users can edit portrait videos through the target segmentation capability, and then switch the background to famous scenes of various countries, so that they can "travel around the world" without leaving home.

In addition to changing the video background, the target segmentation capability can also be applied to video playback. Through the target cutout, the bullet screen can avoid blocking the main body of the plot and optimize the user's viewing experience. Combined with other technical capabilities, the segmented target objects can also be copied and deleted, and even the action time of the target objects can be adjusted to create more creative video works.

Implementation principle

So, how is the target segmentation capability achieved?

First, the user needs to select the target object, and then perform AI target segmentation. By giving the corresponding mask of the target object to be segmented in the first frame of the video, the AI model will automatically match the object in each subsequent video frame and try to segment it. In addition, the model will save the result mask with good segmentation effect of the intermediate frame and combine it with the mask information of the first frame, and match it in subsequent video frames, which can accurately outline the edge details of the target object and further improve the accuracy of target segmentation. .

DEMO demo

Integration

1. Development preparation

For detailed preparation steps, please refer to the official website of Huawei Developer Alliance

2. Edit project integration

2.1 Set the authentication information of the application

Application authentication information can be set through api_key or Access Token.

The Access Token is set through the setAccessToken method, which can be initialized and set once when the application starts, and there is no need to set it multiple times.

 MediaApplication.getInstance().setAccessToken("your access token");

The api_key is set by the setApiKey method, which can be initialized and set once when the application starts, and there is no need to set it multiple times.

 MediaApplication.getInstance().setApiKey("your ApiKey");

2.2 Set the unique identification ID, that is, the License ID.

The license ID is a valid certificate for management and control. You must ensure that the license ID is unique.

 MediaApplication.getInstance().setLicenseId("License ID");

2.2.1 Initialize the Editor runtime environment

To create an editing project, you need to first create an Editor object and initialize its runtime environment. When leaving the editing project, the Editor instance should be released.

(1) Create Editor object

 HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());

(2) Specify the layout position of the preview window

The preview window is responsible for the rendering of video images, which is realized by creating SurfaceView inside the video editing atomic capability SDK. Before creating the window, you need to specify the layout position of the preview window in your app.

 <LinearLayout    
    android:id="@+id/video_content_layout"    
    android:layout_width="0dp"    
    android:layout_height="0dp"    
    android:background="@color/video_edit_main_bg_color"    
    android:gravity="center"    
    android:orientation="vertical" />
// 指定预览窗口 
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);

// 设置预览窗口承载的布局 
editor.setDisplay(mSdkPreviewContainer);

(3) Initialize the running environment. If the license authentication fails, a LicenseException will be thrown.

After the Editor object is created, the actual system resources are not occupied at this time, and the timing of its environment initialization needs to be manually selected. At this time, the necessary threads and timers will be created inside the Video Editing Atomic Capability SDK.

 try {
        editor.initEnvironment();
   } catch (LicenseException error) { 
        SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());    
        finish();
        return;
   }

3. "Object segmentation" capability integration

 // 初始化目标分割AI引擎
videoAsset.initSegmentationEngine(new HVEAIInitialCallback() {
        @Override
        public void onProgress(int progress) {
            // 初始化进度
        }

        @Override
        public void onSuccess() {
            // 初始化成功
        }

        @Override
        public void onError(int errorCode, String errorMessage) {
            // 初始化失败
    }
});

// 初始化成功后，选择需要分割的目标进行分割，返回选择分割目标的的处理结果
// bitmap包含需要分割的目标的视频帧图片；timeStamp为视频帧图片在时间线上的时间戳；points为基于视频帧图片的坐标点集合，左上角为原点坐标，坐标点应位于需要分割的目标内，且个数建议大于等于2，需根据坐标点轨迹来确定选择的目标
int result = videoAsset.selectSegmentationObject(bitmap, timeStamp, points);

// 目标分割的处理结果成功后，添加目标分割AI特效
videoAsset.addSegmentationEffect(new HVEAIProcessCallback() {
        @Override
        public void onProgress(int progress) {
            // 目标分割AI特效处理进度
        }

        @Override
        public void onSuccess() {
            // 目标分割AI特效处理成功
        }

        @Override
        public void onError(int errorCode, String errorMessage) {
            // 目标分割AI特效处理失败
        }
});

// 中断目标分割AI特效处理
videoAsset.interruptSegmentation();

// 移除目标分割AI特效
videoAsset.removeSegmentationEffect();

// 释放目标分割AI引擎
videoAsset.releaseSegmentationEngine();

Learn more details>>

Visit the official website of Huawei Developer Alliance
Get development guidance documents
Huawei Mobile Services Open Source Warehouse Address: GitHub , Gitee

Follow us to know the latest technical information of HMS Core for the first time~