Detailed explanation of multi-mode input event distribution mechanism

I. Overview of Multimodal Input <br>Multimodal input service aims to support developers to provide users with a variety of human-computer interaction methods. On the basis of continuous improvement and support for traditional input, multimodal input will also play the role of OpenAtom OpenHarmony (hereinafter referred to as "OpenHarmony") distributed advantages, improve cross-device interaction experience, and provide system-level support capabilities for new scenarios and new services.
The multi-mode input Input component is the OpenHarmony system-level input event management framework; it connects to multiple input devices in the south, gathers multiple input events (keys, touches), and distributes them to consumers (system services, applications) after normalization/standardization. ).
Southbound input device docking includes various types of input devices, such as: touch screen, mouse, keyboard, touch pad, and remote control.
II. Introduction of input event and device status data flow <br>First, let's take a look at the key data flow under the multi-modal input architecture, so as to understand the whole process from the user initiating an interactive request to the system and application giving an interactive response .
As shown in the figure below, there are two types of key data flows:
● Input device status data flow:
The input device state data describes the state change of the input device and its device attribute information, including: device insertion, removal state, device unique identifier, device name, input methods supported by the device, etc.
The input device state data is reported to the input device state management module of the multi-mode input server through the kernel device driver. The input device management module manages and maintains the global input device state, and at the same time, the device state is encapsulated as a monitorable interface and provided to the upper-layer business module to monitor the state of the system input peripherals.
● Interactive input event data flow:
Interactive input event data is used to describe keyboard, mouse, and touch screen input events; keyboard events include: key code, key timestamp, device to which the key belongs; mouse events include: mouse X/Y coordinates, mouse buttons (such as: mouse left | middle|right) events, etc.; touch events include: timestamp, touch position X/Y coordinates, etc.
The input event data is reported by the device driver to the input event receiving module to complete the forwarding of the input event from the kernel space to the user space, and then to the input event preprocessing module to complete the input event standardization processing (key KeyCode mapping standardization, etc.), and finally distributed by the input event The module completes the event distribution according to the system preset distribution mechanism and principle.
Referring to the following data flow diagram, we can clearly understand the whole process of input event reporting and distribution after the user initiates an interaction request through the input device.

Note: The arrows in the data flow diagram are schematic descriptions

Enter the event distribution module for event preprocessing instructions:
1) The input event distribution process will first pass through the input event interception module. When an interceptor is registered, the input event will be terminated and continue to be reported, and the corresponding interceptor will intercept all input events. The event interception feature currently mainly supports accessibility mode.
2) When no interceptor is registered, input events will be reported to the input event monitoring module, and system-level applications (such as system settings, desktop) support system-level features (such as status bar hide/disappear, etc.) by monitoring input events.
3) The monitoring of events by the event monitoring module will not block the continuous reporting of events; while supporting event monitoring, input events will continue to be reported.
4) The key event will be reported to the subscription key distribution module for processing, distributed to the corresponding application for processing, and the event distribution process ends.
5) Other touch screen events and mouse events will not go through the subscription button distribution module and will continue to be reported to the application window for processing.

3. Principle of multi-mode input event distribution

1. Mouse/touch screen event distribution principle <br>Which target the mouse/touch screen coordinates point to, the input event will be distributed to the corresponding target.
Special scene description for mouse/touch screen event distribution:
1) If no button on the mouse is pressed, which target the mouse is currently pointing to, the mouse input event will be distributed to the target whose coordinates are locked.
2) If any button on the mouse is pressed, use the target locked by the mouse coordinates when the first button is pressed as the distribution target, until all the buttons are lifted.
3) When inputting on the touch screen, use the first finger to press the locked target as the input event distribution target until all fingers are lifted.
2. Principle of key event distribution <br>Key event distribution takes the focus in the current user visual interface as the distribution target, and which target the current interface focus is on, the key event is distributed to the corresponding target.

Fourth, OpenHarmony 3.1 version added interface description

In order to better support the real-time detection and processing of input device hot-plug state change events by upper-layer applications and system services, a JS API interface has been added in OpenHarmony 3.1 to support monitoring device hot-plug events through JS API. At the same time, it provides the way to obtain the unique identifier of the hot-plug input device by registering the callback interface. The input device hot-plug monitoring interface cooperates with the inputDevice.getDevice interface to obtain the detailed information of the hot-plug device, including: input device name, input type supported by the device (keyboard|touch screen|mouse|gamepad), etc.
1. Description of new interface for multi-mode input subsystem <br>Input peripheral hot-plug monitoring interface: function on(type: "change", listener: Callback<DeviceListener>): void; input peripheral cancel monitoring interface: function off(type: "change", listener?: CallbackDeviceListener): void;
2. Added interface parameter description 【DeviceListener】

【ChangeType】

3. Application scenario of hot-plug interface of input device <br>Adaptive display of soft keyboard: In the text editing scenario, the input method can adaptively decide whether to display the soft keyboard by monitoring the hot-plug operation of the input device of the physical keyboard. When there is a physical keyboard device, the soft keyboard does not need to be displayed, and the user input operation is completed through the physical keyboard. When there is no physical keyboard, the input method pops up a soft keyboard, and the user completes the input operation through the soft keyboard.
4. Example of using the input device hot-plug interface <br>After having a preliminary understanding of the mouse hot-plug monitor interface, let's learn more about how to use the input device hot-plug interface in actual development:
1) First import the module
import inputDevice from '@ohos.multimodalInput.inputDevice';
2) Implement input device hot-plug event monitoring through the listening interface:

 // 输入法在软键盘显示逻辑中通过订阅物理键盘的状态：插入/拔出
// 根据isPhysicalkeyboardExist的值决定软键盘是否弹出
...
let isPhysicalkeyboardExist = false;
inputDevice.on("change", (callback) => {
  console.log("type: " + callback.type + ", deviceId: " + callback.deviceId);
  inputDevice.getDevice(callback.deviceId, (ret) => {
    console.log("The keyboard type of the device is: " + ret);
    if (ret == keyboard.ALPHABETIC_KEYBOARD && callback.type == 'add') { // 热插拔设备为键盘
      isPhysicalkeyboardExist = true;
    } else if (ret == keyboard.ALPHABETIC_KEYBOARD && callback.type == 'remove') {
      isPhysicalkeyboardExist = false;
    }
  });
});
...

3) Cancel the monitoring of the hot-plug event of the input device by canceling the monitoring interface:

 listener: function(data) {
    console.log("type: " + data.type + ", deviceId: " + data.deviceId);
}
// 单独取消listener的监听。
inputDevice.off("change", this.listener);
// 取消所有监听
inputDevice.off("change");// 取消监听后，软键盘默认都弹出

Note: In the cancel hot-plug event monitoring interface, the input parameter listener is optional; when the input parameter is listener, it means canceling the specific listening callback. When the input parameter does not have a listener, it means that all listener callbacks are cancelled.
Through the above introduction, I believe that you have a comprehensive understanding of the input event processing and distribution mechanism of the OpenHarmony multi-mode input subsystem. At the same time, we also gave a detailed introduction to the new input device hot-plug monitoring interface of OpenHarmony 3.1 version. For more information about the API interface provided by the multi-mode input subsystem for developers, please visit Gitee for details:
https://gitee.com/openharmony/docs/blob/master/en-us/application-dev/reference/apis/js-apis-inputdevice.md

I believe that everyone has the above basic knowledge as a foreshadowing in the follow-up development, and in the follow-up development, you can be more comfortable and develop applications with better interactive experience. We look forward to working with developers to build the ultimate user experience. Finally, I look forward to working together and building together, and we can communicate and discuss in the OpenHarmony community ( https://gitee.com/openharmony ).