It's time to adapt complete input support for all kinds of devices

With the advancement of technology and the popularity of cross-platform applications, your Android applications are no longer limited to running on candy bar touchscreen devices. Richer interactions enable users to use your app with more complex input. So as a developer, it's time to think about providing robust input support for a wide variety of devices. This article prepares you to share about broader, powerful input support, and you are welcome to read it.

If you prefer to see this through video, check it out here:

https://www.bilibili.com/video/BV1WS4y1o716/?aid=723321224&cid=487529112&page=1

△ It's time to adapt complete input support for various devices

For various Android devices, input (input) is one of the key elements that determine the user application experience. Developers often want user interaction to be as simple and intuitive as possible, but if you buy a new foldable device that comes with a keyboard and your favorite app doesn't support standard keystrokes, it's a very frustrating experience. Frustrated.

Many Android devices today have non-touch input built-in, such as Chromebook keyboards, and some even offer it as standard. So in various Android device types such as phones, foldables, tablets, Chromebooks, Chromeboxes with external displays, Chromebases with built-in displays, Android TV, etc., how should developers ensure that different input methods work for their own application.

△ Various Android devices

We know not all users are using a mobile phone touch screen to interact with your application, some users may be using the keyboard and stylus, and even some users have accessibility demand. So every developer needs to take some time to think about how to make the application a pleasant experience for as many users as possible?

Enhanced input (Enhanced input)

△ Standard input method and enhanced input method

For each type of input device, we can divide the functionality of the application into two broad categories of standard use cases and enhanced use cases:

• Standard use cases include user-expected functions such as selection, text entry, long-press and right-click, and the techniques for handling such use cases are relatively simple to implement and in some cases can be automated. If you want to provide more unique and enhanced support for your application, you need to think more about it.
• Some features are not just better, but may be required in enhanced use cases, such as a mobile game that doesn't support gamepads and a text editor that doesn't support standard copy and paste shortcuts. unpopular with users.

In addition to providing basic features, consider adding unique and user-supportive features. This is how developers can truly differentiate their apps. Take, for example, the eDJing app shown below, which adds support for keyboard DJing and trackpad scratching, as well as MIDI DJ controllers, to instantly transform a phone or Chromebook into a DJ workstation.

△ eDJing application usage scenarios

Cubasis, a top-notch digital audio workstation application, has released a new version optimized for the Chrome OS operating system, with the advantages of a large screen and the convenience of connecting a MIDI controller to a Chromebook, not only enhancing the functionality, but also improving the application's usefulness .

△ Cubasis application usage scenarios

Drawing apps are more focused: Bluetooth and USB drawing tablets continue to work properly, and low-latency stylus APIs are implemented in Chrome OS for drawing and painting applications.

△ Drawing application usage scenarios

In short, users expect a unique, enjoyable, and intuitive experience in your app, and you create that experience!

keyboard input supports

Keyboards are built into Chromebooks or become part of the user's daily experience with detachables, tablets, foldables, and TVs. The good news is that most basic keyboard input usually works out of the box, unless you're working on building your own on-screen keyboard or writing your own text input view from scratch.

send key support (KEYCODE_ENTER)

The developer needs to create a new line for the Enter key in the standard EditText view. If your app has chat functionality, feedback forms, newsletter sign-ups, or anything that requires text to be sent, then the default wrapping behavior is not what you expect, don't worry, the sending functionality you expect is easy to implement.

△ Chat to send text

The developer needs to create a new line for the Enter key in the standard EditText view. The code for pressing the Enter key is shown here. The complete code is as follows:

override fun onKeyUp(keyCode: Int, event: KeyEvent?): Boolean {
    when (keyCode) {
        KEYCODE_ENTER -> {     // 撤销 onKeyUp 并监听 KEYCODE_ENTER
            sendMessage()
            return true     // 如果应用已处理按键事件则确保返回 true
        }
    }
    // 如果没有处理事件，则交回系统处理
    return super.onKeyUp(keyCode, event)  // 如果没有则将事件传递给 super
}

Media key support (Media key)

If your app wishes to support media playback, it also needs to include media key support. To do this, replace KEYCODE_ENTER in the Enter key code with the media key code you want to support, for example MEDIA_NEXT and MEDIA_PREV are used here. Also don't forget to use SPACE to control play and pause with the space bar. The complete code looks like this:

override fun onKeyUp(keyCode: Int, event: KeyEvent?): Boolean {
    when (keyCode) {
        KEYCODE_SPACE, KEYCODE_MEDIA_PLAY_PAUSE -> {
              playOrpauseMedia(); 
              return true 
        }
        KEYCODE_MEDIA_NEXT -> {nextTrack(); return true }
        KEYCODE_MEDIA_PREV -> {prevTrack(); return true }
    }
    // 如果没有处理事件，则交回系统处理
    return super.onKeyUp(keyCode, event)
}

Game key support (KEYCODE_W|A|S|D)

For gaming applications, you may want to include support for the arrow keys and the four keys W, A, S, D, which is equally simple. You just need to find the right document in Android key code , and you can listen to these keys. The complete code looks like this:

when (keyCode) {
    KEYCODE_W, DPAD_UP -> { goUp(); return true }
    KEYCODE_A, DPAD_LEFT -> { goLeft(); return true }
    KEYCODE_S, DPAD_DOWN -> { goDown(); return true }
    KEYCODE_D, DPAD_RIGHT -> { goRight(); return true }
}

Note that onKeyUp is usually listened for when providing keyboard support, so you don't have to worry about sending duplicate onKeyDown events when a key is held down. Also if you want to ensure millisecond response time, you can listen to onKeyDown and handle repeated key events yourself. The complete code looks like this:

override fun onKeyDown(keyCode: Int, event: KeyEvent?): Boolean {
    when (keyCode) {
        KEYCODE_F -> {
            shootFireball()
            return true
        }
    }
    return super.onKeyUp(keyCode, event)
}

For more information visit: ChromeOS.dev

Shortcut support (Ctrl+)

In addition to the basic keyboard keys, it is also necessary to consider configuring Ctrl-based shortcut keys, copy, paste, undo, redo, etc., very common shortcut keys for many applications. Excellent keyboard support will help your app achieve more functionality, and some apps even go a step further by putting advanced features at the user's fingertips. Touchpad scratches.

The code for the Ctrl+Z shortcut for undo is shown below, which is similar to the previous onKeyUp and onKeyDown code, but uses a dispatchKeyShortcutEvent to indicate the meta key combination. Note that pressing the Alt, Ctrl or Shift key at this point triggers this action. The complete code looks like this:

override fun dispatchKeyShortcutEvent(event: KeyEvent): Boolean {
    return when (event.getKeyCode()) {
        KEYCODE_Z -> {
            undoAction() // [Ctrl]+z, [Shift]+z, 或 [Alt]+z 触发
            true
        }
        else -> {
            super.dispatchKeyShortcutEvent(event)
        }
    }
}

If you only want to respond to specific meta key combinations, you can use Meta_Alt_On, or you can use methods like isShiftPressed() or isCtrlPressed(). The complete code looks like this:

when {
    event.isCtrlPressed && event.isShiftPressed -> {
        redoAction(); true // [Ctrl]+[Shift]+z 触发
    }
    event.isCtrlPressed -> {
        undoAction(); true //[Ctrl]+z 触发
    }
    else -> { // 交给系统处理
        super.dispatchKeyShortcutEvent(event)
    }
}

According to the code here, the undo operation will only be performed when Ctrl+Z is pressed at the same time, and the redo operation will only be performed when Ctrl+Shift+Z is pressed at the same time, and Alt is not used.

mouse/trackpad input supports

As with keyboards, most mouse and trackpad inputs usually work without any additional code. However, it is still necessary for developers to use the mouse to test all the functions of the application to see if there are any omissions.

△ mouse

Right click supports

Right-clicking is one of the most common omissions. A common example of touch is a long press on the screen to perform a context-based action, but long press mouse clicks are not intuitive. If you want to support right-click, there are several options here. The code to add the context menu is shown here, the full code looks like this:

registerForContextMenu(myView) 
// 首先为上下文菜单注册一个或多个视图，这将自动处理长按和右键点击两种操作。
...
override fun onCreateContextMenu(menu, view, menuInfo) {
    super.onCreateContextMenu(menu, view, menuInfo)
    menuInflater.inflater(R.menu.myContextMenu, menu)
    // 然后撤销 overrideonCreateContextMenu 并扩展正确的菜单布局，同一菜单可用于多个视图。
}
override fun onCreateContextMenu(item: MenuItem): Boolean {
    return when(item.itemId){
        R.id.myMenuItemId -> { menuItemAction(); true }
        else -> { return super.onContextItemSelected(item) }
        // 最后，设置 onContextItemSelected 指示选中特定菜单项时需要执行的操作。
    }
}

For other right-click behavior outside of the context menu, you can set the view with an onContextClickListener and just use it to call the same method used in the long-press use case. The corresponding code is as follows:

myView.setOnContextClickListener {
       performContextAction()
       true
}

Hover response support

When a user uses a mouse or trackpad, they often expect the interface to respond in some way. For example, visual feedback is generated when the mouse cursor hovers over a clickable view, as shown in the figure, the pointer icon may change, or some other visual indication appears, which can be intuitively felt by the user. The code for when the pointer is over the view is shown here, the full code looks like this:

myView.setOnHoverListener { view, motionEvent ->  //为视图设置 onHoverListener
    when (motionEvent.action) {
        ACTION_HOVER_ENTER -> {
            view.pointerIcon = PointerIcon.getSystemIcon(
                appIicationContext, PointerIcon.TYPE_HAND
            )
            view.setBackgroundColor(Color.BLUE)
            true
        } // 监听 HOVER_ENTER 事件并执行相应的操作。根据代码显示指针图标将变为手形且背景颜色将变为蓝色。
        ACTION_HOVER_EXIT -> { resetUI(); }
        else -> { false }
        // 不要忘记设置当 HOVER_EXIT 发生时重置图标和背景颜色。
    }
}

pointer capture support

Pointer capture is another common mouse and trackpad enhancement that is critical for some games and adds specific functionality to some apps. Apps can capture the mouse cursor with pointer capture so that the cursor does not appear on the screen, so that it can receive relative pointer events without moving the cursor to the edge of the screen. First-person games like Minecraft: Education Edition are good examples.

△ Minecraft: Education Edition

To support pointer capture, you can first call requestPointerCapture, and then call releasePointerCapture to release the captured pointer. In your code you can add an OnCapturedPointerListener to use the received pointer data and take advantage of the relative changes in the pointer position to do some cool stuff. The complete code looks like this:

view.requestPointerCapture()
...
view.releasePointerCapture()
...
view.setOnCapturedPointerListener { _, motionEvent ->
    // 计算自从上次 motionEvent 事件后的坐标增量
    mouseChangeX = motionEvent.x
    mouseChangey = motionEvent.y

    // 计算自从上次 motionEvent 事件后的位移
    val totalDistance = hypot(mouseChangeX, mouseChangeY)
    doSomethingCool(totalDistance)
}

Gesture support

In addition, the Android API also provides advanced pointer handling, which can add gestures, pinch and zoom support, etc., as shown in the following figure.

If you want to learn more about the Android API, see the Android Developers site for a getting started guide - Using tap gestures .

stylus input supports

If you've added great support for pointers to your app, the stylus usually works as expected for most use cases. Some of the enhancements to the stylus are worth noting, such as stylus tilt and press support on some devices, which helps you add some great controls and features to your drawing or drawing applications. There's also a low-latency stylus API that gives you the lowest-latency display response in your drawing or drawing applications, and configurable stroke prediction for a pen-on-paper experience. To see it in action, check out an app like Concepts on a supported Chromebook or Android device.

△ Concepts app supports stylus

• For for implementation details , see the Android Developers site documentation on AXIS_PRESSURE and AXIS_TILT.
• For the low latency stylus API library and demo app, see GitHub .

Using the stylus in the Android emulator

We've partnered with Microsoft to bring host stylus support to the Android emulator, and if you're optimizing your app for more advanced stylus support, you'll be able to test precise tilts and presses with the Android emulator on a supported host control.

The Microsoft Surface Duo Developer Experience team worked with Google on an effort to support advanced input development and testing, such as multi-touch analysis and stylus support. So how can I test the app with the stylus while running the app in the emulator?

△ Test the stylus in the simulator

Shown is the Microsoft Duo 2 emulator running on Surface Studio, with two apps running at the same time: The right pane is a sample app that allows you to test pen press sensitivity, pen orientation, Erase the tip and other stylus buttons; the left pane is the Microsoft OneNote app, and the simulator lets you draw, take notes, or erase on the OneNote canvas.

We're excited that on touch-enabled PCs, the Android emulator now also supports multi-touch, allowing you to test interactions with apps that require multiple fingers, such as pinch, zoom, and more Touch interaction.

△ Using Gestures in Google Maps

This foldable emulator with Android Studio built in is running Google Maps, and you can zoom in and out of the map with just two fingers. And these emulators are updated not only to support the use of two fingers, but to support up to 10 touch points if your hardware allows it.

None of these changes you see are specific to the Surface Duo emulator, they apply to other foldable emulators as well. Microsoft has also been using these emulator updates to develop and optimize its own apps, such as testing stylus interactions on a variety of device types including Surface Duo, large screens, and other foldable devices.

handle input supports

If you have a gaming app you need to add gamepad support. Use the appropriate key code to determine whether to perform the action onKeyUp or onKeyDown.

△ Gamepad

The key codes for the gamepad arrow keys and the keyboard arrow keys are the same, just listen to the KEYCODE_DPAD event to handle both. Gamepad buttons also have their own keycodes, and you can listen to these buttons as you did here for the X button. The complete code looks like this:

when (keyCode) {
    BUTTON_X -> {
        engageNitro(); return true
    }
    DPAD_UP, KEYCODE_W -> {
        goForward(); return true
    }
    DPAD_LEFT, KEYCODE_A -> {
        turnLeft(); return true
    }
    ...
}

MIDI input supports

When it comes to specialized hardware, support is required for the device and use case. MIDI support is especially important for musical and creative tools, allowing a wide range of expressive input signals, from pressure-sensitive piano keyboards to devices with sliders, knobs, keyboards, and many more different input triggers. provide support.

If you want to use or learn more about MIDI, the Android Media Samples library provides some handy open source samples to help you get started with MIDI. Check out the MidiSynth project, especially the Synth Engine module in that project. Note that starting from Android 29 you can also use the NDK MIDI API .

Review

With large-screen Android devices emerging and growing in popularity, it's always been important to have great input support on Android, but it's especially important for foldables, tablets, and Chrome OS. Think about your app's input handling and how to increase interactivity, unlock new features, and improve the app experience. Looking forward to your developers putting in the effort to build great Android apps and adding great input handling to them!

You are welcome to to submit feedback to us, or share your favorite content and found problems. Your feedback is very important to us, thank you for your support!