1688 Merchants' multi-screen collaborative live broadcast technology solution based on HarmonyOS

Author: Wan He

It has been more than three months since the official release of HarmonyOS 2.0. The latest data shows that more than 100 million devices have been upgraded to the HarmonyOS operating system. However, for the majority of application developers, HarmonyOS seems to be no different from Android. It is more about adapting or migrating some original functions. Regarding the core technology highlights of -1616508530a102 distributed soft bus , it is still unclear how to implement it, let alone how to integrate it with its own business.

On the supply side of the 1688 live broadcast domain, we have been constantly exploring how to improve merchants’ launch capabilities and reduce their launch costs. After learning about the characteristics of the distributed soft bus, we found that these capabilities of HarmonyOS are very suitable for the 1688 merchants’ multi-device launch requirements, so We have developed this multi-device launching program combining distributed soft bus.

In this article, I will combine business scenarios from a technical point of view to share 1688 live broadcast supply side is based on the distributed soft bus technology of HarmonyOS to realize the coordinated broadcast of multiple devices, helping 1688 merchants to reduce the cost of launching and improve the ability of launching.

Different from the usual mobile phone broadcast, our solution involves multiple devices and multiple screens. In addition to recording the anchor, you can also connect additional cameras to record products, display live data and previews on a large screen, and interact with the anchor on the large screen. Function. Let’s learn about the implementation effects of our technical product solutions through a video, please click on the following link to view the video:

1688 HarmonyOS-based multi-screen collaborative live broadcast technology solution for

1. Business background

1688 is the leading B2B e-commerce platform in China. The customers we serve mainly include factory owners, Taobao sellers, physical shop owners, stall merchants, etc. Due to the shrinkage of offline physical businesses due to the epidemic, more and more factories and stall owners are seeking to transform online live broadcast delivery. With the development of our business, this year we have also incubated an app specifically for merchants—1688 Merchant Edition. , To provide merchants with more professional services, including live broadcast, negotiation, workbench, etc. The practical case in this article is exactly in the 1688 merchant version live broadcast domain scene .

1.1 Pain points

Unlike Taobao live broadcasts, most of the anchors are professional MCN teams invited by merchants. Most of the 1688 live broadcasts are the merchants themselves. They treasure their own goods, but they lack professional broadcast capabilities and equipment for e-commerce live broadcasts. How to help merchants lower the threshold and improve the quality of launching under the premise of investing limited resources in 1688 merchants? Through offline visits to merchants, we found several common issues related to launching.

anchor frequently approached the camera to show product details

host interacts with a single device for co-broadcasting

factory environment is simple and portability is difficult

In summary, 1688 live broadcast merchants mainly have the following three pain points in terms of launching equipment:

1. live broadcast lacks specific function equipment

• Lack of commodity cameras, the current camera is far away from the commodity, the anchor needs to frequently approach the broadcasting equipment to display the details of the commodity, which affects the live broadcast perception;
• Lack of large interactive screens, mobile live broadcast interactive screens are small, anchors need to get close to the broadcasting equipment to see the audience's messages and interact with the audience, which affects the live broadcast experience.

2. is difficult to collaborate between live broadcast devices

• It is difficult to coordinate the launching tools, and the recording equipment, interactive equipment used by the host and the central control equipment used by the co-broadcasting are not interoperable and difficult to operate;
• Anchor co-broadcasting is difficult to interact. Usually the co-broadcaster explains the product and co-broadcasts the product coupons. Due to the lack of interaction between the devices of both parties, the communication can only be verbal and the communication lacks privacy.

3. live broadcast equipment investment is low and poor capacity

• Low investment in launching equipment. Many of 1688’s anchors are small and medium-sized businesses. The investment in live broadcasts pursues cost-effectiveness, and the launching equipment is uneven;
• The portability of the broadcasting equipment is poor. In complex scenes such as factory workshops, the live broadcasting and on-the-go broadcasting need to be coordinated to start broadcasting. The equipment is difficult to be portable and lacks the ability to start broadcasting in multiple cameras.

1.2 Current situation

Does the existing equipment meet the merchants' large-screen and multi-camera, collaborative interaction between devices, and portable low threshold ? Let's compare their characteristics first:

Selection and comparison of broadcasting equipment:

1. When the mobile phone is launched, the mobile phone is responsible for streaming and interaction, and has a certain collaborative portability, but there are problems such as small screens and unmatched cameras;
2. The PC is launched, with a configurable camera and a large screen, and the hardware cost is not high, but the device is large in portability and the functions are concentrated in one device and lack of interactivity;
3. The live broadcast all-in-one machine is launched, and the equipment customized for live broadcast has strong large-screen functions, but the hardware threshold is high and the hardware is burned and cannot be customized.

tool provided to merchants for live broadcasting needs to have 1616508530a44f multi-device coordination, large-screen interaction, few connection lines, configurable hardware, and cost-effective . So, is there a start-up program that meets these advantages at the same time?

2. Scheme design

At present, many anchors in our App are HarmonyOS users. For small and medium businesses that already have HarmonyOS devices, we have proposed a multi-device collaborative broadcast solution based on a distributed soft bus.

2.1 Program overview

When the host uses a mobile phone to start broadcasting, when encountering a large-screen device, he can stream the live broadcast capability to the large-screen device with one click. At this time, the host's mobile phone displays the status of the remote control, and the large-screen display shows the real-time data board and real-time interactive information. The capture screen explained with the anchor. If there is a commodity camera in the scene, the anchor mobile phone can also evoke the commodity camera. The commodity camera will transmit the collected audio and video data to the large-screen device for display, and the large-screen device will finally complete the merge and push. Co-broadcasting can obtain real-time audio and video stream playback by connecting to large-screen devices, and realize collaborative interaction with the anchor audience. The solution mainly has two core technical capabilities, namely, live interaction on cross-device migration, , and 1616508530a4e9. The coordinated transmission of audio and video streams on multiple devices is .

2.2 Program features

The program also has the advantages of the above-mentioned launch tools:

1. Efficient collaboration among multiple devices. A camera dedicated to recording the anchor's explanation, a camera dedicated to collecting product images, large-screen equipment to display the anchor's explanation screen and merge and distribute the video stream, the anchor mobile phone controls all the broadcasting equipment, and the co-broadcast mobile phone plays the video stream to watch;
2. Big screen interaction. Large-screen devices display live data and interactive information, the anchor mobile phone can control the large-screen interaction, and the co-broadcast mobile phone can interact with the anchor and the audience on the large screen;
3. There are few portable lines. Complete wireless connection between multiple devices in the same local area network;
4. The equipment can be equipped. Optional configuration of cameras for recording anchors and recording products, and optional configuration of large-screen interactive equipment;
5. Cost-effective equipment. The host and co-broadcast ordinary HarmonyOS mobile phones, the ordinary display is connected to the HarmonyOS device, and the camera can be equipped according to the definition requirements.

Three, technical realization

The core technical capabilities to achieve the above products program is a live interaction dynamic migration flow and cooperative transmission of audio and video streams , both underlying technical capabilities are based on HarmonyOS distributed software package on our bus expansion project . Next, let me briefly introduce the distributed soft bus.

Distributed bus is a concept proposed by Huawei Hongmeng. Its inspiration should come from computer systems. In the computer system, the common channel for transmitting information between CPU, input and output devices is called a bus. The soft bus is to establish a virtual communication connection between multiple devices to complete the interconnection and intercommunication between multiple devices without physical line connection, so that the information transmission function between devices with low latency and high bandwidth makes it possible for each device to be wirelessly realized. Efficient data transmission.

From the above architecture diagram, we can see that the distributed soft bus encapsulates a variety of communication protocols, and uniformly encapsulates the discovery, connection, and transmission of devices into an externally transparent interface. Developers only need to make a simple call. The interconnection and intercommunication between multiple devices do not need to care about the details of Unicom.

The live interaction in our scheme is based on the distributed soft bus to realize the multi-device migration and circulation. The audio and video streams are also transmitted across devices through the soft bus channel. The following is a detailed introduction to the realization of the capabilities of these two technical products.

3.1 Migration and circulation of live interaction

Product Features

There are three main functions of live interaction migration and circulation on multiple devices:

1. Live broadcast real-time data migration large screen;
2. Anchor and co-broadcasting interactively migrate the big screen;
3. The host central station controls and operates the large screen.

Technical solutions

1. The host App transforms the original Android project, adds HarmonyOS Ability dependency, and adds FA for start-up control;
2. The big screen App is a HAP with a big screen added to the original App, including live data FA and real-time interactive FA;
3. The co-broadcasting app is an FA that has been added to the original Android project to interact with anchors and buyers.

The host App first discovers and connects to the nearby big screen, and then seamlessly transfers the live data FA and real-time interactive FA to the big screen device. The host App jumps to the start-up control FA. The host App can operate the big screen and coordination through the middle station control. Broadcasting App, the interaction between devices is realized through IDL communication. It should be noted that the apps of the above three devices are based on the newly added capabilities of the original project transformation, and the three are the same App.

Code

Let's take the anchor app evokes the big-screen app and jumps to the middle station control as an example:

1. Register the transfer equipment at LiveControlAbility;

ContinuationRegisterManager continuationRegisterManager = getContinuationRegisterManager();
continuationRegisterManager.register(getBundleName(), null, callback, requestCallback);

2. After the ContinuationDeviceCallback connects to the connected device successfully, call connectAbility to evoke the large-screen PA ScreenServiceAbility and pass relevant parameters;

private IContinuationDeviceCallback callback = new IContinuationDeviceCallback() {
        @Override
        public void onDeviceConnectDone(String selectDeviceId, String deviceType) {
                connectAa(selectDeviceId);
                continuationRegisterManager.updateConnectStatus(abilityToken, selectDeviceId, DeviceConnectState.CONNECTED.getState(), null);
            }
        }
    };

3. Receive data on the onConnect of the large screen ScreenServiceAbility and pull up the large screen FA ScreenPageAbility;

public IRemoteObject onConnect(Intent intent) {
            jumpScreen();
            return new ScreenRemoteForController();
    }

4. ScreenRemoteForController inherits from HarmonyControllerInterfaceSkeletonScreen and initializes after the OnConnect callback, and receives instructions sent from the host App, such as awakening other devices, and then distributes events through EventHandler.

class ScreenRemoteForController extends HarmonyControllerInterfaceSkeleton {
        ScreenRemoteForController() {
            super("IScreenRemoteInterface");
        }


        @Override
        public void sendCommand(String command) throws RemoteException {
             MyApplication.getHandler().sendEvent(command);   
        }

Difficulties and limitations

Cross-device communication

The current communication method of HarmonyOS devices only supports IDL, and IDL is one-way communication. In order to achieve two-way communication between devices, we adopt the method of adding callbacks in IDL, or create two two-way IDLs. Double pass between devices.

Android and HarmonyOS hybrid compatible development

At present, all of our functions are incremental development of HarmonyOS on the original Android project, and then mixed packaging. The function development here is limited by the fact that there are still many basic libraries that have not yet completed the migration and adaptation of HarmonyOS, and the compatibility and communication between HarmonyOS and Android.

3.2 Cooperative transmission of audio and video streams

Product Features

There are four main functions of coordinated transmission of audio and video streams across devices:

1. The stream of the host’s audio and video collection is transferred to the large-screen camera;
2. Arouse commodity cameras to collect commodity images and transfer them to large-screen devices;
3. The large-screen equipment combines the merchandise and the host screen to play and push the stream;
4. The audio and video streams are transmitted to the co-broadcast mobile phone to play and watch.

Technical solutions

1. The host App will close the audio and video capture and preview functions after connecting to the large-screen device;
2. The large-screen App has added recording preview capability and combined streaming capability. At the same time, it is necessary to increase the encoding to transmit the collected audio and video streams to the co-broadcast App;
3. Added the ability to collect audio and video and transmit to large-screen equipment on the commodity camera;
4. The Co-Broadcast App removed the previous method of pulling streams from the cloud, and changed it to play audio and video streams after local decoding.

After the host App is connected to the large-screen device, it will evoke the camera of the large-screen device to record audio and video and display the preview. After the large-screen device is started, pull up the commodity camera, and at the same time encode the captured audio and video stream, and transmit the stream to the large-screen device Merge and push the stream, and the co-broadcasting mobile phone can get the large-screen audio and video stream to play.

Code

Let's take the example of collecting audio and video and transmitting it to the large screen after the commodity camera is awakened:

1. When the commodity camera is awakened, the SurfaceProvider is first initialized, and the background ScreenServiceAbility service is connected in its surfaceCallback callback, and the camera is turned on at the same time;

protected SurfaceOps.Callback surfaceCallback = new SurfaceOps.Callback() {
        @Override
        public void surfaceCreated(SurfaceOps surfaceOps) {
            previewSurface = surfaceOps.getSurface();


            Intent intent = new Intent();
            Operation operation = new Intent.OperationBuilder()
                .withDeviceId("")
                .withBundleName(mCameraAbility.getBundleName())
                .withAbilityName("com.alibaba.screencamera.ScreenServiceAbility")
                .build();
            intent.setOperation(operation);
            intent.setParam("client_role", "camera");
            mCameraAbility.connectAbility(intent, mAbilityConnection);
            openCamera();


        }
    };

2. The started background service ScreenServiceAbility transmits audio and video data when onConnect;

public IRemoteObject onConnect(Intent intent) {
            return new ScreenRemoteFoSlave() {
                @Override
                public void onPcmReady(byte[] pcmData) throws RemoteException {
                        mControllerCallback.onPcmReady(pcmData);


                }


                @Override
                public void onYuvData(int type, int length, int seq, byte[] cameraData) throws RemoteException {


                        mControllerCallback.onReturnData(type, length, seq, cameraData);


                }
            }
     return null;
}

3. Take the video data collected by the commodity camera as an example, encode the YUV data format and send it;

mCodec.registerCodecListener(new Codec.ICodecListener() {
                @Override
                public void onReadBuffer(ByteBuffer byteBuffer, BufferInfo bufferInfo, int trackId) {
                    byte[] msg = new byte[bufferInfo.size];
                    byteBuffer.clear();
                    byteBuffer.get(msg);
                    sendEncodedDataToRemote(msg, bufferInfo);
                }


            }
        );
       mCodec.setCodecFormat(fmt);
       mCodec.start();

4. Slice the converted YUV video frame data and then send it through the ScreenService background service;

private void sendEncodedDataToRemote(byte[] data, BufferInfo bufferInfo) {
                    byte[] msgTemp = new byte[bufferInfo.size];
                    System.arraycopy(data, position, msgTemp, 0, msgTemp.length);
                    mScreenServiceProxy.onYuvData(CameraUtil.IRemoteMsg.MSG_TYPE_SLICE_END, bufferInfo.size, 0, data);


    }

5. The large-screen device receives the audio and video data sent by the commodity camera, decodes it and plays it.

private ControllerCallbackStub mControllerCallback = new ControllerCallbackStub("com.alibaba.cameraohos.IControlFaCallback") {
        @Override
        public void onPcmReady(byte[] pcmData) throws RemoteException {
            mPlayRecord.playTransData(pcmData);
        }


        @Override
        public void onReturnData(int type, int length, int seq, byte[] cameraData) throws RemoteException {
                    createMyDecoder();
                    initMuxer();
        }
    };

Difficulties and limitations

audio and video stream local codec transmission

Different from the traditional local encoding push, the audio and video streams between devices in this solution are all transmitted in the local area network. Due to the limited bandwidth of the distributed soft bus and the large video data, we use the YUV data to encode the local video stream. Decoding, 720P video stream transmission is achieved through the method of slicing and merging.

Multi-device networking

HarmonyOS equipment networking needs to be based on prerequisites, including the same WiFi, turn on Bluetooth, and temporarily only support logging in to the same Huawei account. The reason for the harsh conditions I think is mainly due to security considerations, but in our scenario, the devices are all services For the same live account, it is reasonable to log in to the same Huawei account and start the broadcast scene.

Fourth, summary and outlook

4.1 Summary

Our plan has the following three characteristics:

1. software solution solves hardware limitations. uses the HarmonyOS distributed soft bus technology to break the hardware limitation of the 1688 live broadcast scene by using a software solution, to break up the application functions to the most matching hardware equipment, and to realize the mutual complementation of hardware resources.
2. Combine scenes to expand the ability to start broadcasting. combines the features of 1688 merchants' live broadcast scenes, based on the characteristics of the distributed soft bus of HarmonyOS, solves the seamless flow of live broadcast interaction, audio and video transmission, and realizes large-screen interaction and multi-device broadcast functions.
3. Low cost and efficient solution to the pain points of merchants. does not basically change the original 1688 launch function. At the software level, the same application can be used to express different capabilities on multiple devices. At the hardware level, suitable equipment can be selected according to the strength of the merchant. The development cost is low and the merchant investment is small.

Distributed soft bus is the unique ability of HarmonyOS operating system. The biggest disadvantage of the current solution is that it cannot be implemented on mobile phones with other operating systems, so only users of HarmonyOS have this ability. In the future, we will explore the implementation of similar functions on Android and iOS. At the same time, we hope that our practice of multi-device collaborative broadcasting in e-commerce live broadcasts can inspire more technical products to use distributed soft bus capabilities to land business scenarios.

4.2 Outlook

Based on our deposited live interactive migration and audio and video collaborative transmission two major technical capabilities, we will further explore in these two directions in the future

1. multi-camera and multi-camera broadcast, enriching live content. As the Group Artc gradually adapts to HarmonyOS and provides more streaming capabilities, we can provide professional high-definition cameras to record products, movable cameras to broadcast factories, multi-angle cameras to capture live images, and further improve the quality of our live content
2. standardized central control box reduces the start-up cost. integrates the capabilities of audio and video stream fusion distribution, screen projection, and start-up control on a central control device. The host only needs to have the device to achieve the above-packaged start-up capabilities

Our current solution is mainly the development of software applications and does not involve the development of hardware. Therefore, the camera and large-screen device capabilities required in actual use are completed by the mobile phone of HarmonyOS. If we have the camera and projection screen for burning OpenHarmony The box will further reduce the cost of merchants and enrich the ability to start broadcasting. With the development of the era of 5G and the Internet of Everything, the prospects within sight are still very broad.

refer to:

• Distributed voice camera https://gitee.com/panthole/harmonyos-codelabs/tree/master/VoiceCamera
• Migrate video content across devices https://gitee.com/panthole/harmonyos-codelabs/tree/master/DistributedVideoCodelab
• HamonyOS video decoding capabilities Play preview screen https://gitee.com/panthole/harmonyos-codelabs/tree/master/CodecDemo

Follow us, 3 mobile technology practices & dry goods for you to think about every week!