How does the HarmonyOS application framework solve the problem of multi-device interaction?

As the full-scene multi-device lifestyle continues to deepen, users have more and more devices, in addition to the mobile phones we wear every day, smart watches on our wrists, wireless headphones in our pockets, PCs and Pads in our backpacks, and more There are also smart TVs, smart refrigerators and other equipment at home. Over the years, the car has gradually become an electronic device running on the road. If these devices are isolated from each other, the experience must be very poor.

Therefore, if you want to connect everything, you need a new interaction model, and the development of a new interaction model requires the blessing of the application framework. The HarmonyOS application framework solves the interaction problem of the new era. Next, we use user interaction as a starting point to introduce how the HarmonyOS application framework solves the problem of multi-device interaction.

1. The basic interaction model in the era of Internet of Everything

As shown in Figure 1, looking at the history of human-computer interaction, there are the following key points in time:

• In 1868, the emergence of typewriters solved the problem of text input in human-computer interaction
• In 1964, the emergence of the mouse solved the problem of graphical interface interaction in human-computer interaction
• In 2007, the emergence of multi-finger gestures solved the interaction problem of touch screens

Figure 1 Human-computer interaction mode diagram

These three historical time points can be said to be a watershed in the history of human-computer interaction. They provide a new way of interaction to solve the interaction problem of new types of devices. However, these three interaction methods solve the problem of single-machine interaction, and the era of Internet of Everything faces the complex problem of multi-device interaction.

Based on many years of research and analysis of human-computer interaction, we found that in multi-device interaction scenarios, the scenes of users using devices can be divided into two categories, one is users use multiple devices at the same time , and the other is successively Use multiple devices (as shown in Figure 2).

Figure 2 Multi-device interaction model diagram

1. Simultaneous

When we use multiple devices at the same time, in addition to concurrency, the interaction model has two other important characteristics: collaboration and complementarity. Collaboration means that multiple devices interact and coordinate with each other to complete a task. Complementarity refers to the use of the differences in the morphology of the equipment to complement each other to complete a task.

For example, when we can't find a TV remote control at home, the mobile phone can be turned into a remote control, which is a complement of equipment capabilities.

2. Sequential use (Sequential)

When we use multiple devices one after another, continuity and consistency are very important. Continuity means that when I switch from one device to another, the state I just operated on should remain uninterrupted. For example: the video played on the mobile phone can continue to play after it is transferred to the PAD. And consistency means that when we use different devices such as watches, mobile phones, large screens, etc., their operation methods and basic visual elements should be consistent. For example: multi-finger gestures, control styles. Of course, the same does not mean the same. Because of the different screen sizes and shapes of each device, the visual elements need to be adapted in a targeted manner.

In order to meet the above two interaction models, the distributed application framework built by HarmonyOS provides two basic functions, namely: multi-terminal collaboration and cross-terminal migration .

Next, we will introduce how the above two capabilities of HarmonyOS solve the problem of multi-device interaction.

2. How does HarmonyOS solve the problem of multi-device interaction?

First we introduce the HarmonyOS distributed application framework, which can be divided into 5 layers (as shown in Figure 3).

Figure 3 Overall architecture view of distributed application framework

Layer1 is the underlying software , including the kernel, drivers, and soft buses. This layer is relatively far away from developers.

Layer2 is basic capabilities of the platform , it contains a lot of basic services in a distributed environment, such as: distributed task scheduling, distributed data management, distributed hardware management and distributed file management.

Layer3 is an application framework core services , can be roughly divided into two: one is the global package management, a distributed operation and management. Unlike traditional operating systems, the package management on HarmonyOS needs to process the package information of all devices on the entire HyperTerminal, so we call it global package management. As for distributed operation management, it will be highlighted below, which includes a framework for collaboration and migration.

Layer4 is the programming interface layer used by developers. Although the implementation of the bottom layer of the system is very complicated, we provide developers with a concise interface on the fourth layer.

Layer5 is the development and application of user program layer , this layer in close contact with the developer.

Next, we will focus on introducing the two frameworks that implement the multi-device interaction model in Layer 3, which are the multi-terminal collaboration framework and the cross-terminal migration framework .

terminal collaboration framework

The multi-terminal collaboration framework is prepared for scenarios where users use multiple devices at the same time. For example, the mobile phone side application is used as a game controller, the smart screen side application is used as a game display, and a new game experience is formed for users; another example: the tablet side application is used as an answer board, and the smart screen side application is used for live broadcasting, forming a brand new Internet for users Lesson experience.

To realize such a business, we must rely on a multi-terminal collaboration framework (as shown in Figure 4).

Figure 4 Multi-terminal collaboration

Multi-terminal collaboration is a technical solution for the circulation of user applications. It refers to the simultaneous operation of different FA/PAs on multiple terminals, or alternate operations to achieve complete services; or the simultaneous operation of the same FA/PA on multiple terminals to achieve complete services. Multiple devices as a whole provide users with a more efficient and immersive experience than a single device.

Description: FA: Feature Ability, with UI interface, PA: Particle Ability, without UI interface.

When our business needs to span multiple devices, there are two basic capabilities that need to be guaranteed:

One is to be able to establish a cross-device connection path, and to sense changes in the connection status in real time. This is accomplished through IAbilityConnection, which relies on distributed management services and soft buses to manage the underlying connection.

The second is to be able to transfer status and data on the connection channel for business collaboration. This can be achieved through HarmonyOS IDL. These transferred data include not only the data that the system needs to transfer, but also the data that the application needs to transfer.

With these two capabilities, developers can complete the coordination of business logic between different devices and the complementary functions of hardware capabilities between devices.

2. Cross-end migration framework

Cross-terminal migration is used to prepare for the successive use of multiple devices by users, and it is a common scenario we use. For example, when you watch a video on your mobile phone, you feel that the experience is not good enough, and you want to switch to a PAD with a larger screen and better sound effects to continue watching. On systems other than HarmonyOS, you can only unlock the PAD first and find the app , And then find the playback history. If you are lucky, your playback progress has been synchronized, but it is very likely that due to the delay of network synchronization, you have to find the location you just watched first, and then continue watching.

Those who are concerned about the HarmonyOS system will know that at this year's HarmonyOS 2 conference, we released a brand new service center. In this service center, you can manage the entire task list on the HyperTerminal across devices, which allows you to directly drag the tasks to the PAD in one step in the task center on the mobile phone.

To accomplish this, it is necessary to use a cross-terminal migration framework (as shown in Figure 5)

Figure 5 Cross-terminal migration

Cross-terminal migration is a technical solution to realize the transfer of user applications. It is to migrate the FA running on the device 1 to the device 2, and realize business migration through IAbilityContinuation. After the migration is completed, the FA on the device 2 side continues the task, and the application on the device 1 side exits. When the user drags a task from one device to another, the application will receive a call from the system. At this time, the application can inform the system of the state it needs to save, and the system will use distributed task management Pass the data to the target, then unpack the data and restore the application state. In this case, what the user feels is that the task is migrated from one device to another.

Migration and collaboration need the support of developers, but these new features often cannot be fully popularized in a short time. When the application does not adapt to the migration framework, how does the system complete the cross-segment migration task?

① Cross-terminal migration and window management

In order to ensure the consistency of user experience, when the application does not adapt to the migration framework, the system will complete the cross-terminal migration task through the ability of distributed window management. As shown in Figure 6, when the user drags a task across devices, the HarmonyOS system will migrate the task window to a Virtual Window (virtual window) for rendering, and then transfer the rendered result to the ProxyWindow (remote proxy window) through the soft bus. ) To achieve the effect of cross-device display.

Figure 6 Distributed window management diagram

hardware platform and automatic follower

We know that many hardware devices are used in the application process. For example: Camera, microphone or sensor. When we migrate an application from one device to another, it is not only necessary to transfer the user interface and application state. It would be strange if the sound is still played on the original device after the transfer. So, how are these hardware managed in HarmonyOS?

In order to make the application migration process more consistent, we have done two things in the hardware-level system (as shown in Figure 7):

Figure 7 Distributed hardware platform and automatic follow

The first thing is " hardware resource pooling ". That is: the HarmonyOS system will globally number and virtualize the hardware of all devices on the HyperTerminal, which enables the software on any device to use the hardware in all HyperTerminal devices.

The second thing is " automatically follow ". The application actually uses a virtual handle, which is not bound to a certain characteristic hardware. When we migrate a certain software entity from one device to another, the migration decision module in the system will automatically change the hardware Also switch to the target device together, so that the entire user experience is consistent.

After the above introduction, I believe that everyone has understood how the HarmonyOS application framework solves the problem of multi-device interaction.

As a new generation of intelligent terminal operating system, HarmonyOS welcomes developers to join us and become a member of the entire Hongmeng ecosystem. Together, let us build a new business ecosystem and infinite possibilities for billions of users around the world.