Youku Hongmeng Development Practice｜Multi-screen Interactive Development Practice

Author: Yuchai & Yisheng

Youku and Huawei have maintained a good strategic partnership for a long time, aiming to provide consumers with a high-quality audio-visual entertainment experience. The circulation feature of Hongmeng operating system brings a brand new gameplay to multi-screen interaction. This article takes the technical reserves of Youku Play Center as the starting point, combined with the mirroring and circulation characteristics of the Hongmeng system, and introduces in detail the core of normal circulation, free perspective and zoom. The way to practice ability on Hongmeng.

Background introduction

Hongmeng Distributed Experience

The distributed experience defined by Huawei mainly includes two types: continuous experience and collaborative experience.

Continuous experience

When a user initiates an operation on one device and switches to another device to continue the operation, the user can immediately continue the current operation on the new device. The continuity experience includes task connection and audio and video connection.

Collaborative experience

The software and hardware capabilities of multiple devices cooperate with each other to provide users with a more efficient and immersive experience than a single device as a whole. Collaboration experience includes software collaboration and hardware collaboration.

For Youku, we have provided users with the following multi-screen interactive functions.

• The user uses the mobile phone to watch the video to a certain point in time, and then switches to the tablet device to continue watching the video from the point of interruption just now
• The user opens a video with a mobile phone or tablet, and then uses the screen projection function to project the video to the smart screen or TV box, and continue to watch it on the big screen
• When a user encounters functions that are inconvenient to use the client on the large screen, such as login or payment, he can use his mobile phone to directly scan the QR code on the large screen to log in or pay.

The above-mentioned functions can be directly implemented by Youku client using the existing "Huawei One Touch", "Huawei HiPlay" and other functions. We hope to use the special functions provided by Hongmeng OS to realize some "new functions" that are not available in the current Youku client, develop some "pure Hongmeng" proprietary features, and then integrate with the Android Youku host and guest. Hongmeng code and Android code are linked to each other to provide consumers with a variety of cool audio-visual entertainment functions.

Youku clients span multiple software and hardware platforms such as Android, iOS, iPad, OTT, etc., and they can interact with each other through functions such as screen projection, which naturally has a multi-screen interactive scene.

And HarmonyOS uses a variety of distributed technologies, so that the development of application programs has nothing to do with the differences in the form of different terminal devices. This allows developers to focus on the upper-level business logic and develop applications more conveniently and efficiently.

In the Youku Hongmeng version, we developed a new HarmonyOS multi-screen interactive FA based on the distributed bus capability provided by HarmonyOS and combined with Youku's existing screen projection function, which provides multi-screen interaction between multiple Hongmeng devices. .

This FA is also written 100% using Hongmeng API. Users can use the "Video Streaming" button on the playback page of Youku Hongmeng Edition to transfer the video stream being played on the mobile phone to other Hongmeng devices (Hongmeng smart screen, tablet, etc.), and can Use the mobile phone as a remote control to control the broadcast of large-screen devices.

It can not only control the volume, fast forward and rewind, playback speed, clarity, and episodes of the large screen, but also rotate the angle of the free-view video on the large screen.

The biggest difference between "HarmonyOS-based multi-screen interaction" and traditional screencasting is that we use the "device/service discovery mechanism" provided by HarmonyOS to search for peer devices, and use HarmonyOS's "connection establishment" function to establish connections between devices. Two-way communication.

As the "device/service discovery mechanism" and "connection establishment" functions of HarmonyOSOS have been highly optimized, our own experience is that "HarmonyOS multi-screen interaction" is faster than traditional DLNA or Miracast mirroring. Device discovery, connection establishment, and connection After establishment, it is very stable and not easy to disconnect.

Moreover, we can perform high-speed data transmission through this connection, directly streaming video from the control side to the large screen.

Below, we briefly introduce how the distributed bus capability of HarmonyOS is combined with Youku's existing projection function.

Screencasting business overview

Screen projection technology has been more and more used in our daily life and work. Wireless screen displays and extended functions such as mobile phones, tablets, and personal computers have also brought more convenience to families, corporate meetings, product launches, and remote training. More efficient model changes. Corresponding product solutions on the market are also blooming, such as Apple’s Airplay wireless projection solution, WiFi Alliance’s Miracast wireless projection solution, Google’s ChromeCast wireless projection solution, and Intel’s WIDI wireless projection solution And so on, have promoted the upgrade and evolution of multi-screen interactive products at home and office to varying degrees.

The improvement of wireless projection technology and the network environment has actually marked the advent of the era of efficient intelligence. People can use their mobile phones to surf the Internet, watch TV, entertain, and work. In addition to watching TV, smart TVs tend to be smart screens and display screens. Everyone hopes to browse information, shop, play games, etc. on TV. People prefer to enjoy the immersion of watching TV on the big screen. The future development trend is also obvious. The huge potential market demand is the biggest driving force for projection technology, and it will present a "prosperous scene" in full bloom.

As one of the domestic Internet video giants, Youku has accumulated rich technical and product experience in the screencasting business scene, and has achieved very rapid development; in addition, Youku’s unique playback capabilities, such as free perspective, zoom, AI super score, Frame sharing 4K, etc., also provides a powerful expansion space for multi-screen interactive scenes. On the whole, both Youku's projection business and playback capabilities have laid a solid foundation for cooperation with Huawei.

Hongmeng circulation scene

HarmonyOS is a new generation of intelligent terminal operating system, which provides a unified language for the intelligence, interconnection and collaboration of different devices, and brings a simple, smooth, continuous, safe and reliable full-scene interactive experience. Since its inception, HarmonyOS has taken "Internet of Everything" as its mission. Through a set of codes and core distributed technology, it meets the needs of various hardware on different platforms, and connects mobile phones, tablets, computers, cars and smart wearables. This kind of equipment perfectly realizes the circulation between different equipment. It can be said that “circulation” is one of the most core characteristics of Hongmeng ecology.

With the advent of the smart era, circulation is everywhere, whether it is the two-way control on the car or the one-touch connection between the smart home; whether it is the one-touch project of the office file, or the super connection of the mobile phone and the smart screen, there is a screen Wherever possible, circulation can be achieved. On the other hand, the 5G era redefines the public's cultural and entertainment methods. Larger screens, higher bit rates, and smarter circulation have become core demands. In this context, Hongmeng and Youku have naturally come together.

Youku's Projection Capability Layout

Youku's screen projection capabilities not only cover traditional LAN protocols, such as DLNA, AirPlay, etc., but also a new self-developed cloud projection protocol, which completely breaks the limitations of LAN multicast and improves the success rate of device discovery. In addition, in order to increase the new gameplay, the latest magic screen device comes with an NFC sticker, which realizes the one-touch shooting function and introduces a new gameplay. Next, we will introduce youku's screen-casting capabilities in detail.

Capability territory

There are three basic elements of wireless projection: content, equipment, and protocol. Among them, the content can be non-real-time data files, such as movies, pictures, music, etc., or real-time data streams, such as screen mirroring, live streaming, and so on. The equipment includes the content provider that initiates the screen projection (the "sender") and the content presenter that receives the projection (the "receiver"). The common senders include mobile devices such as mobile phones and tablets, which are responsible for obtaining remote or Media resources stored locally and transmitted to the receiving end; the receiving end is responsible for responding to the content and broadcast control instructions from the sending end, and completing subsequent content presentation. The protocol provides an interaction protocol between the sender and receiver, covering core links including device discovery, data transmission, and playback control. Representative protocols include SSDP and SOAP protocols in LAN projection scenarios, cloud casting The MTOP and ACCS protocols of the screen scene, etc.

Multi-screen interactive services are highly complex, with long interactive links, spanning two business lines of large and small screens, involving a large number of business parties including advertising, membership, operators, broadcasting, hardware, and media assets.

After long-term polishing with a large amount of demand, Youku finally formed the following multi-screen interactive layout, which can be divided into core components such as the screen projection protocol layer, the projection screen broadcast control layer, and the business adaptation layer:

Projection protocol layer

1. LAN projection. Including DLNA and AirPlay, which have strong versatility, but are limited by the multicast capability, and there is often a problem of not being able to find the device;

2. Cloud projection. Cloud projection is a self-developed new protocol to solve the problem that the local area network protocol cannot find the device. It can effectively improve the success rate of device discovery. However, it is limited by the self-operated large-screen terminal application, and its versatility is greatly restricted;

3. NFC projection screen. As one of the new features announced by Magic Screen’s new hardware, NFC projection combined with the attached sticker shortens the operation link for projection on the mobile terminal and realizes the "one touch projection" of Youku video.

Projection and broadcast control layer

1. DRM management. Responsible for the management and verification of projection equipment to ensure the correct delivery of copyrighted videos;
2. Format processing. Adaptation and compatibility with different large-screen devices to ensure the playback experience after screen projection;
3. Clarity management. Responsible for adaptive and abnormal degradation of multi-channel clarity, taking into account business scenarios such as advertising;
4. Auxiliary scheduling. Based on the exclusive domain name of the projection screen, it has dynamic flow management and scheduling capabilities for different regions, operators, and device information.

Business adaptation layer

1. Cast screen ads. It includes three combinations: internal injection, internal injection, and external injection. In addition, a delivery system for projected advertisements on both large and small screens is also under development;
2. Point straight and short fusion. The playback adaptation modules of the three scenes of on-demand, live broadcast, and short video are lowered to the internal maintenance of the projection screen, which is of great significance for the unification of the later monitoring link and the outward projection of the projection capability.

Initial practice of Hongmeng system

Ordinary circulation

"Hongmeng circulation" refers to a multi-screen interactive method that uses the characteristics of the Hongmeng operating system. The biggest difference from the traditional screen projection protocol is its dependence on the Hongmeng operating system. Specifically, Hongmeng operating system not only provides a "device/service discovery mechanism" as a device discovery protocol, but also provides a "device connection capability" to support two-way communication between devices.

Similar to the traditional screencasting protocol, Hongmeng Circulation includes URL-based ordinary circulation and streaming-based circulation. Ordinary transfer refers to a multi-screen interactive method based on Hongmeng's transmission capabilities. Unlike general protocols such as DLNA, Hongmeng provides core capabilities including device discovery, data transmission, application query, and file sharing. It not only provides more stability than native protocols According to the business requirements, the self-developed APK (such as Kumiao) can be transferred to the big screen through the small screen and evoked to realize a series of customized operations.

First introduce the general circulation, the timing diagram is as follows.

As shown in the figure above, the implementation of some of the core processes is briefly introduced.

Hongmeng device discovery

It is exactly the same as the discovery process of DLNA projection and cloud projection. Hongmeng’s discovery protocol is implemented in the search method of DlnaDevs:

// 鸿蒙search
    if (HarmonyCastMgr.haveInst()) {
        HarmonyCastMgr.getInst().searchDevs();
        // 自由视角search
        if (HarmonyCastMgr.getInst().hasMirr()){
            HarmonyCastMgr.getInst().searchHarmonyMirrorDevs();
        }
    }

Let's take a look at the implementation details of the Hongmeng discovery process:

    // 搜索设备（异步方式）
    public boolean searchDevs() {
        boolean ret = false;
        if (isHarmonyEnable()) {
            Intent intent = new Intent();
            ComponentName componentName = new ComponentName(PackageContant.PACKAGE_YOUKU, "com.youku.feature.MiddlewareAbility");
            intent.setComponent(componentName);
            intent.putExtra(AbilityUtils.PARAM_KEY_INSTALL_ON_DEMAND, true);
            intent.setAction("action.videoplayer.getdevicelist");
            try {
                mActivity = YoukuContext.getTopActivity();
                ret = AbilityUtils.connectAbility(mActivity, intent, mServiceConnection);
            } catch (RuntimeException e) {
                e.printStackTrace();
                ret = false;
            }
        }
        return ret;
    }

Hongmeng Discovery Protocol relies on the FA channel through AIDL. The mServiceConnection internally parses the IBinder object, maps it to the Device type defined by multi-screen interaction, and calls back the update interface of the device list. The core implementation is as follows:

   public ServiceConnection mServiceConnection = new ServiceConnection() {
        @Override
        public void onServiceConnected(ComponentName name, IBinder service) {
            LogEx.d(tag(), "onServiceConnected");
            HarmonyRCS rcs = new HarmonyRCS(service);
            String[] devices = null;

            try {
                devices = rcs.getDeviceList();
            } catch (RemoteException e) {
                e.printStackTrace();
            }

            // 设备ID和名称数组，以适应鸿蒙AIDL查询格式
            List<String> deviceIdList = new ArrayList<>();
            List<String> deviceNameList = new ArrayList<>();

            for (int i = 0; i < devices.length; i++) {
                if ((i & 1) == 0) {
                    deviceIdList.add(devices[i]);
                } else {
                    deviceNameList.add(devices[i]);
                }
            }

            // 原始数据映射为Client格式
            if (deviceNameList.size() >= deviceIdList.size()) {
                mHarmonyDevs.clear();
                for (int i = 0; i < deviceIdList.size(); i++) {
                    String deviceId = deviceIdList.get(i);
                    String deviceName = deviceNameList.get(i);
                    if (StringUtils.isNotBlank(deviceId) && StringUtils.isNotBlank(deviceName)) {
                        Client client = new Client();
                        
                        ...

                        mHarmonyDevs.put(deviceId, client);
                    }
                }

                // 判断大屏设备是否在线，如果不在线，则停止投屏
                if (DlnaApiBu.api().proj().stat() == DlnaPublic.DlnaProjStat.PLAYING) {
                    Client currentDev = DlnaApiBu.api().proj().req().mDev;
                    if (currentDev != null && currentDev.isHarmonyDev() && !mHarmonyDevs.containsKey(currentDev.getDeviceUuid())) {
                        stopProjEx();
                    }
                }

                // 回调调用方获取设备列表
                if (null != mRcsCallback) {
                    mRcsCallback.onHarmonyDevsChanged();
                }
            }

            try {
                if (mActivity != null) {
                    AbilityUtils.disconnectAbility(mActivity, mServiceConnection);
                }
            } catch (Exception pE) {
                pE.printStackTrace();
            }
        }

        @Override
        public void onServiceDisconnected(ComponentName name) {
            LogEx.d(tag(), "onServiceConnected");
        }
    };

Select the device and start FA

Similar to the cloud projection self-research protocol, Hongmeng ordinary circulation relies on a separate discovery process. Similar to other protocols, HarmonyCastTrunkBiz was created as the Hongmeng protocol management class, and the following basic interfaces were implemented:

    // 开始投屏
    void start();
  
    // 停止投屏
    void stop();

    // 发送播放指令
    void play();

    // 发送暂停指令
    void pause();

    // 发送seek指令
    void seek(int prog);

    // 发送设置音量指令
    void setVolume(int volume);

Since Hongmeng FA provides a customized remote control interface and two-way data transmission channel, all the tasks of sending instructions are taken over by Hongmeng FA. The above-mentioned interface responsible for sending instructions only synchronizes the playback status of the small screen. After the user clicks to select the Hongmeng device, the start method is triggered and the Hongmeng protocol entry is started. The startHarmony of HarmonyCastMgr is responsible for the data interaction with the Hongmeng FA:

    // 投屏所需的播放数据
    HarmonyCastData castData = new HarmonyCastData();
    castData.mDev = req.mDev;
    castData.mUrl = req.mUrl;
    castData.metaData = DlnaMetadata.getInst().getMetadataWithReq(req);
    castData.isFromNowbar = isFromNowbar;

    // FA遥控器所需的UI数据
    HarmonyParameterBean parameterBean = new HarmonyParameterBean();
    parameterBean.deviceUuid = req.mDev.getDeviceUuid();
    parameterBean.mShowTitle = req.mShowTitle;
    parameterBean.projSource = "";
    parameterBean.isYoukuApp = req.mDev.isYoukuApp;
    String parameterData = JSON.toJSONString(parameterBean);
    
    // 按FA数据格式进行封装
    String touchuanData = JSON.toJSONString(castData);
    HarmonyPaBean paBean = new HarmonyPaBean();
    paBean.touchuanData = touchuanData;
    paBean.parameterData = parameterData;
  
    final String harmonyJson = JSON.toJSONString(paBean);
    Activity activity = YoukuContext.getTopActivity();
    Intent jIntent = new Intent();
    ComponentName component = new ComponentName(PackageContant.PACKAGE_YOUKU, "com.youku.feature.MainAbility");
    jIntent.putExtra(AbilityUtils.PARAM_KEY_INSTALL_ON_DEMAND, true);
    jIntent.putExtra("harmonyJson", harmonyJson);
    jIntent.setComponent(component);

    try {
        // 每次开始投屏前检测一次远端是否在线，避免因大屏掉线导致的投屏失败
        boolean remoteValid = true;
        if (HarmonyCastMgr.haveInst()) {
        remoteValid = HarmonyCastMgr.getInst().searchDevs();
    }

        if (remoteValid) {
            AbilityUtils.startAbility(activity, jIntent);
        } else {
            stopProjEx();
        }
    } catch (Exception e) {
        e.printStackTrace();
    }

Free perspective

Free perspective is developed by Ali Entertainment Mocool Lab. It is a playback technology that allows you to slide videos at will and switch viewing angles freely. Free-view video is a further development of multi-view live broadcasting, bringing more initiative to users. Users can rotate their angle of view when watching a ball game and zoom in on part of the screen to watch idol players' slam dunks at close range, realizing a real movie experience comparable to cross-dimension.

The following is the free perspective effect of the "This is Street Dance" variety show:

Please see the original video for the video: Youku Hongmeng Development Practice｜Multi-screen Interactive Development Practice

Different from ordinary video, the bit rate of free-view video is generally higher. For example, in the above-mentioned hip-hop recording scene, a total of more than 40 dedicated cameras are installed, and the formed array capture screen records every wonderful moment in all directions, and realizes millisecond-level synchronization and remote collaborative control. The captured images undergo focus correction and 3D reconstruction. Based on the texture, depth, posture and other parameter information collected by each camera position, the multi-angle scattered pictures are finally spliced into a high-definition video file of up to 6K and 8K.

Therefore, while the free-view video improves the user's perception, it also places very high requirements on the network environment and decoding performance of the playback end. The original free-view implementation scheme is based on the streaming feature. The small screen transmits the M3U8 address or broadcast control command of the free-view video to the large-screen side through the above-mentioned streaming capabilities, and the large-screen side completes the subsequent data download, decoding and rendering. Process. However, considering that the hardware performance of Huawei's smart screen is not enough to support this scale of calculation, the traditional transfer scheme has to be stopped.

To this end, the development team compared all the existing projection protocols on the Hongmeng system, and got the following conclusions:

Hongmeng Screen Projection Agreement	advantage	shortcoming
DLNA	Standard mature protocol, strong versatility	Weak native scalability
Cloud projection	Does not rely on local area network, strong scalability	Poor versatility
Hongmeng Agreement	High system adaptability and strong expansion ability	Rely on Hongmeng system
Cast+ protocol	The system has high adaptability and strong compatibility to meet the needs of streaming transmission	Rely on Huawei ecology

After internal evaluation and discussions with the Huawei R&D team, the development team finally decided to abandon the traditional URL-based projection method and adopt a mirrored projection solution based on Huawei Cast+. Specifically, the free-view video is played and controlled normally on the mobile phone. , And mirror the player view to the TV through Cast+.

Briefly introduce the implementation details of the Huawei Cast+ protocol. As shown in the figure below, the Huawei Cast+ Kit is a mobile phone-centric multi-screen collaboration component developed by Huawei. Based on this, the business side can realize fast, stable, and low-latency mirroring transmission between multiple devices.

Based on the Huawei Cast+ protocol, the projection link of the free-view scene is roughly the same as the normal projection protocol. The overlapping part includes device search, device selection, and connection establishment. The difference is that the last step of the free-view video does not have a playback address. To the large screen, it is to create a virtual screen and complete the display. The timing and method of creating the virtual screen are the most important in the practice process.

The essence of the free-view streaming solution can be summarized in two aspects: first, the remote control page is used as the main screen, and the two-way data channel between it and the playback page is opened to achieve playback control and status synchronization; secondly, on the virtual screen Start playing the related view as the content to be mirrored and transferred to the big screen.

The plan for the main screen was basically determined, but when it was the turn of the virtual screen, I stepped on the pit. The virtual screen rendering object that can be thought of at first is naturally a full-screen playback page. However, after trying it, I found that directly rendering the full-screen playback page to the virtual screen will have the following two problems. The attached analysis report is as follows:

Question 1: The remote control page on the main screen is first mirrored to the big screen.

• Reason: Cast+ will start to capture the contents of the virtual screen after the connection is successfully established and the device is in the PLAYING state. If the virtual screen is not loaded yet, it will capture the contents of the main screen as a bottom line. After testing, the time difference is about 100ms, and the playback page cannot be initialized during the period, so there will be a problem of the main screen content occupying.

Problem 2: After the playback page is entered, there is a logic for switching between horizontal and vertical screens, which causes the screen to undergo a screen transition after mirroring.

• Reason: It belongs to the adaptive logic of the playback page. The initial entry is the vertical screen. After the mirroring, the horizontal screen switching logic will be triggered, which will cause the screen to switch on the side of the large screen.

After a comprehensive internal evaluation, it was decided to adopt a more lightweight method to realize the virtual screen. We found that the player Dialog only contains contents such as SurfaceView and Loading status. If only the Dialog level is mirrored, although more player processing logic is added, the screen upload speed and faster response efficiency are guaranteed. The adjusted program structure is shown in the figure below.

The program has the following advantages:

1. The life cycle of the player is optimized. Every time you switch between free-view streaming and small-screen playback, only the SurfaceView is switched, and the player is not rebuilt. At this time, the player is only in the STOPPED state. After switching to the small-screen playback, skip After the process of initialization and playback address request, it is guaranteed to start broadcasting quickly;
2. Advance the timing of starting the Presentation, and start the page and resume the player when the onDisplayAdded is received, which greatly shortens the screen time.

In summary, the final effect is shown in the following video.

Please see the original video for the video: Youku Hongmeng Development Practice｜Multi-screen Interactive Development Practice

Online performance

Netizens have also shown strong interest in the characteristics of the Hongmeng system and the performance of the Youku Hongmeng version. We have picked some experience videos on the Internet, as follows:

• Edition Experience-Transfer from Knowing": 1617fbaad72b5d https://www.bilibili.com/video/BV1A5411w76Y/
• Youku Cast Screen Loved Love": 1617fbaad72b91 https://www.bilibili.com/s/video/BV1pU4y187t3

Subsequently, after the continuous iteration of the two versions, the stability and experience of Youku Hongmeng Edition have been greatly improved. On the other hand, with the self-improvement of the Hongmeng system, youku Hongmeng Edition's second-phase demand version has begun to be scheduled. I believe that under the strong combination of the two parties, it will continue to bring surprises to the market and users.

Summarize

In the development process of Youku Hongmeng version multi-screen interactive special project, all participants intervened when the Hongmeng interface and debugging environment were not perfect, overcame many difficulties, continued to explore trial and error, and mastered the Hongmeng development technology in a very short time. The stack not only successfully completed large and small development tasks, but also fed back many remaining problems and improvement suggestions to the Huawei technical team. This is another in-depth cooperation between the two parties and a very successful practice.

For Huawei, the Youku Hongmeng version not only brings a groundbreaking new form of video interaction to the Hongmeng ecology, making new users hooked; it also perfectly adapts to the circulation characteristics of the Hongmeng system, paving the way for the subsequent richer new gameplay the way. For Youku, Hongmeng’s multi-screen interactive practice greatly expands the existing multi-screen interactive capabilities and accumulates important experience for the new era of Hongmeng.

, 3 mobile technology practices & dry goods for you to think about every week!