Desktop mode for foldable devices

Expand your video playback experience

Foldable devices provide users with the possibility to do more things with their phones, including desktop mode * and other innovations, that is, when the phone is flat, the hinge is in a horizontal position and the folding screen is in a partially open state. .

The desktop mode is very convenient when you don’t want to hold the phone in your hand. It is very suitable for watching media, making video calls, taking pictures and even playing games.

A good example is the Google Duo team's optimization of its application, so that the application can run well on tablets and foldable devices.

△ Comparison of Duo application before and after optimization

In this article, you will learn a simple and efficient way to adapt the layout of your app when it runs on a foldable device.

This is a simple media player case, it will automatically adjust the size to avoid the folds appearing in the middle of the screen, and adjust the position of the playback control components, from being embedded in the screen when the screen is fully expanded, to being displayed when the screen is partially folded It is a separate panel. As shown in the video:

△ A case of showing desktop mode on Samsung Galaxy Z Fold2 5G mobile phone

*Desktop mode is also called Flex mode on Samsung Galaxy Z series foldable phones.

Preliminary preparation

The sample application uses Exoplayer , which is a very popular open source media player library on the Android platform. The following Jetpack components are also used:

• MotionLayout , which is a subclass ConstraintLayout MotionLayout combines the flexibility of the parent class, and at the same time has the ability to show smooth animation when the view transitions from one pose to another.
• ReactiveGuide , this is an invisible component that will automatically change its position when a SharedValue changes. ReactiveGuide needs to work with the Guideline auxiliary class.
• WindowManager , this is a library that helps application developers to provide support for new device type parameters, and provides a common API interface for different window features.

To use these libraries, you must add the Google Maven library to the project and declare the relevant dependencies:

dependencies {
    ...
    // 成文时使用如下的版本号，Exoplayer 最新版本号详见 https://github.com/google/ExoPlayer/releases
    implementation 'com.google.android.exoplayer:exoplayer-core:2.14.1'
    implementation 'com.google.android.exoplayer:exoplayer-ui:2.14.1'

    implementation 'androidx.constraintlayout:constraintlayout:2.1.0-rc01'
    implementation 'androidx.window:window:1.0.0-beta01'

    ...
}

layout

First consider the layout of the video player Activity. The root element is MotionLayout that contains three subviews.

<androidx.constraintlayout.motion.widget.MotionLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:id="@+id/root"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:background="@color/black"
    app:layoutDescription="@xml/activity_main_scene"
    tools:context=".MainActivity">

    <com.google.android.exoplayer2.ui.PlayerView
        android:id="@+id/player_view"
        android:layout_width="0dp"
        android:layout_height="0dp"
        app:layout_constraintBottom_toTopOf="@+id/fold"
        app:layout_constraintLeft_toLeftOf="parent"
        app:layout_constraintRight_toRightOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        app:use_controller="false" />

    <androidx.constraintlayout.widget.ReactiveGuide
        android:id="@+id/fold"
        app:reactiveGuide_valueId="@id/fold"
        app:reactiveGuide_animateChange="true"
        app:reactiveGuide_applyToAllConstraintSets="true"
        android:orientation="horizontal"
        app:layout_constraintGuide_end="0dp"
        android:layout_height="wrap_content"
        android:layout_width="wrap_content" />

    <com.google.android.exoplayer2.ui.PlayerControlView
        android:id="@+id/control_view"
        android:layout_width="0dp"
        android:layout_height="0dp"
        android:background="@color/black"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/fold" />

</androidx.constraintlayout.motion.widget.MotionLayout>

Two of the views are from the Exoplayer suite, and you can use them to PlayerView (interface for displaying media) and PlayerControlView (container for playback controls).

The third view is a ReactiveGuide . It is placed in the middle of the other two views, and is divided into the other two views in the form of Guideline

The main PlayerView is restricted to always be above the ReactiveGuide. In this way, when you move the ReactiveGuide from the bottom to the folded position, the layout transition will occur.

You may want to limit the playback controls to the bottom of ReactiveGuide. This way, the control will be hidden when the screen is fully expanded, and will appear at the bottom when the screen is partially collapsed.

Note the layout_constraintGuide_end attribute on line 28. It is the value that needs to be changed when you move the reference line. Since ReactiveGuide is horizontal, this attribute refers to the distance from the reference line to the bottom of the parent layout.

Let your app perceive screen folding

Now enter the most important part: how to know when your phone enters desktop mode and get the position of the fold?

When the initialization is complete, the WindowManager library allows you to monitor the layout changes of the WindowLayoutInfo by collecting the data flow WindowInfoRepository.windowLayoutInfo()

override fun onStart() {
        super.onStart()
        initializePlayer()
        layoutUpdatesJob = uiScope.launch {
            WindowInfoRepository.windowLayoutInfo
                .collect { newLayoutInfo ->
                    onLayoutInfoChanged(newLayoutInfo)
                }
        }
    }

    override fun onStop() {
        super.onStop()
        layoutUpdatesJob?.cancel()
        releasePlayer()
    }

If you want to know how to initialize and release an Exoplayer instance, please refer to- Exoplayer codelab .

Whenever you get new layout information, you can query the display characteristics and check whether there are folds or hinges in the current display of the device:

private fun onLayoutInfoChanged(newLayoutInfo: WindowLayoutInfo) {
        if (newLayoutInfo.displayFeatures.isEmpty()) {
            // 如果当前屏幕没有显示特征可用，我们可能正位于副屏观看、
            // 不可折叠屏幕或是位于可折叠的主屏但处于分屏模式。
            centerPlayer()
        } else {
            newLayoutInfo.displayFeatures.filterIsInstance(FoldingFeature::class.java)
                .firstOrNull { feature -> isInTabletopMode(feature) }
                ?.let { foldingFeature ->
                    val fold = foldPosition(binding.root, foldingFeature)
                    foldPlayer(fold)
                } ?: run {
                centerPlayer()
            }
        }
    }

Note that if you don’t want to use Kotlin data stream, starting from 1.0.0-alpha07 version, you can use window-java this tool, which provides a series of Java-friendly APIs to register or unregister callback functions, or Use window-rxjava2 and window-rxjava3 tools to use the API adapted to RxJava.

When the device orientation is horizontal and FoldingFeature.isSeparating() ) returns true, the device may be in desktop mode.

If this is the case, you can calculate the relative position of the fold and move the ReactiveGuide to that position; if the situation is the opposite, you can move it to 0 (bottom of the screen).

private fun centerPlayer() {
        ConstraintLayout.getSharedValues().fireNewValue(R.id.fold, 0)
        binding.playerView.useController = true // 使用内嵌画面的控件
    }

    private fun foldPlayer(fold: Int) {
        ConstraintLayout.getSharedValues().fireNewValue(R.id.fold, fold)
        binding.playerView.useController = false // 使用位于屏幕底部一侧的控件
    }

When you call the function fireNewValue like this, the library function will change the layout_constraintGuide_end . When the device is fully expanded, the entire screen will be used to display the main PlayerView.

Final question: Where should you move ReactiveGuide when the device is folded?

FoldingFeature object has a method bounds () ), it is possible to obtain information in the bounding rectangle is folded at the screen coordinate system.

If you want to implement the horizontal screen function, then most of the time, the border will be represented by a rectangle centered vertically on the screen, which is as wide as the screen and the height is the same as the hinge (for foldable devices, the value is 0, for For dual-screen devices, it will be the distance between the two screens).

If your app is in full-screen mode, you can fix the PlayerView at the top of FoldingFeatures.bounds().top and fix the ControlView at the bottom of FoldingFeatures.bounds().bottom.

In all other cases (not full screen) you need to consider the space occupied by the navigation bar or other UI components on the screen.

In order to move the reference line, you must specify its distance from the bottom of the parent layout. A possible implementation of calculating the ReactiveGuide proper position function is as follows:

    /**
     * 返回折叠处相对于布局的位置 
     */
    fun foldPosition(view: View, foldingFeature: FoldingFeature): Int {
        val splitRect = getFeatureBoundsInWindow(foldingFeature, view)
        splitRect?.let {
            return view.height.minus(splitRect.top)
        }

        return 0
    }

    /**
     * 获取 displayFeature 变换到视图坐标系的边界和它当前在窗口中的位置。
     * 这里的计算中默认会包含内边距。
     */
    private fun getFeatureBoundsInWindow(
        displayFeature: DisplayFeature,
        view: View,
        includePadding: Boolean = true
    ): Rect? {
        // 视图在窗口中的位置要与显示特征在同一坐标空间中。
        val viewLocationInWindow = IntArray(2)
        view.getLocationInWindow(viewLocationInWindow)

        // 将窗口中的 displayFeature 边界矩形与视图的边界矩形相交以裁剪边界。
        val viewRect = Rect(
            viewLocationInWindow[0], viewLocationInWindow[1],
            viewLocationInWindow[0] + view.width, viewLocationInWindow[1] + view.height
        )

        // 如果需要的话，包含内边距
        if (includePadding) {
            viewRect.left += view.paddingLeft
            viewRect.top += view.paddingTop
            viewRect.right -= view.paddingRight
            viewRect.bottom -= view.paddingBottom
        }

        val featureRectInView = Rect(displayFeature.bounds)
        val intersects = featureRectInView.intersect(viewRect)

        // 检查 displayFeature 与目标视图是否完全重叠
        if ((featureRectInView.width() == 0 && featureRectInView.height() == 0) ||
            !intersects
        ) {
            return null
        }

        // 将显示特征坐标偏移至视图坐标空间起始点
        featureRectInView.offset(-viewLocationInWindow[0], -viewLocationInWindow[1])

        return featureRectInView
 }

Summary

In this article, you learned how to improve the user experience of media applications on foldable devices by implementing flexible layouts that support desktop mode.

Stay tuned for the follow-up articles on the development guide for different morphological parameters!

More resources

• Exoplayer Codelab: Use Exoplayer to play video streams
• Desktop mode example application
• for foldable devices
• Build an application for foldable devices
• Jetpack WindowManager
• Use MotionLayout to manage motion and widget animation

You are welcome to submit feedback to us via the QR code below, or share your favorite content or problems you find. Your feedback is very important to us, thank you for your support!