Android breaks through the shackles of screen refresh

background

With the popularity of smart phones, in modern life, we are gradually unable to get rid of our dependence on mobile phones. The needs of all levels of society such as travel, shopping, medical care, housing, and social interaction are inseparable from the use of smartphones to achieve more efficient and convenient purposes. In just over a decade, relying on the development of smart phones, there have also emerged an endless stream of Internet companies, mobile phone brand manufacturers and countless applications. As the related Internet industry market gradually reaches the population bottleneck, competition based on the stock market is also welcome. Here comes the white-hot stage. A corner of the industry's involution is the competition of major mobile phone brands. In addition to "shell replacement" as the main new strategy, major manufacturers have also racked their brains to introduce new products, such as folding screens, high refresh, fast charging, 100 million Software and hardware iterations such as high-level camera pixels are also frequently seen at various new phone conferences. According to incomplete statistics, in 2021 alone, brand manufacturers in the Android camp will release 502 models of devices worldwide (data source: gsmarena ). Let’s not discuss the involution behind the anxiety of reality. The domestic mobile phone brands are exploring new technologies and new hardware and applying them in a slightly radical way. Apple [manual dog head.jpg] that is too lazy to change the shell).

Returning to the theme of this article, as an Android developer, in addition to adapting to various models and systems with serious fragmentation, there is also a key task to make the applications developed by yourself can be used by brand manufacturers. New features to enhance the user experience of the application. Like the adaptation of the folding screen, the feature of multi-screen small windows is used to increase the user's sense of interaction in multitasking.

(Network map, intrusion and deletion)

The adaptation of the camera's ultra-high pixel to enrich the user's experience of taking pictures and the adaptation of applications and games for mobile phone screens that support high refresh rates to increase the user's smooth experience. This article selects some of these new features to focus on the refresh mechanism based on the Android system and how to adapt to some high refresh rate mobile phones currently on the market to obtain a better user experience.

(Network map, intrusion and deletion)

As we all know, the screen refresh rate of most mainstream models on the market is still at 60Hz, that is, the screen refreshes at a speed of about 16.6ms at 1000ms/60. Now some high-end mobile phones can reach 90Hz or even 120Hz. It can also be seen from the previous animation that different refresh rates and higher frequencies give users a smoother sensory experience. A little knowledge is interspersed here. Everyone knows that the movie is also made of continuous pictures, so what is the refresh frame rate of the movie? 24fps! That is to say, as long as a continuous single picture segment is played at a speed of 24 pictures per second, the brain will automatically associate it as a continuous picture. So why did director Ang Lee try to shoot Billy Lynn's Long Halftime Walk at 120fps? From the audience's point of view, 120fps movies are far more shocking and deeply rooted than 24fps, especially for some grand war scenes or narrative scenes, 24fps high-speed pictures will appear blurred during transition (motion blur, motion blur) Phenomenon, but 120fps can make these blurry transitions appear in front of the audience with a clearer picture, and the audience can feel more immersive. The same is true for some games that support high refresh.

screen refresh mechanism

In the Android system, the UI rendering and refreshing pipeline for the screen can be roughly divided into five stages:

• Phase 1: The UI thread of the application processes the input events, invokes the relevant callbacks belonging to the application, and updates the View hierarchy list that records the relevant drawing instructions;
• Stage 2: The rendering thread of the application (RenderThread) sends the processed instructions to the GPU;
• Stage 3: GPU draws the frame data;
• Stage 4: SurfaceFlinger is a system service responsible for displaying different application windows on the screen. It combines the content that the screen should finally display, and submits the frame data to the hardware abstraction layer (HAL) of the screen;
• Stage 5: The screen displays the frame content.
  ​

Android adopts a double-buffering strategy, and uses the vertical synchronization signal vsync to ensure the best exchange timing of the front and rear buffers. There are two concepts mentioned earlier, one is the frame rate and the other is the refresh rate. An ideal state is that the frame rate and the refresh rate are consistent. The GPU has just finished processing a frame, swiped it to the screen, and the next frame is ready. At this time, the frame data before and after is continuous and complete. However, the process of transferring data from the CPU to the GPU is uncontrollable. When the screen is refreshed, if the frame buffer obtained is not in a complete ready state, there will be screen tearing that often occurred in non-smart phones or old TVs a long time ago. (for example, the upper half of the screen is the previous picture, and the lower half is the new picture). The double buffering strategy solves this problem. By maintaining two buffers, the front buffer is responsible for transporting the frame data to the screen, while preparing the rendering object of the next frame in the back buffer, and switching whether or not to use the vsync signal Swap back buffer data to front buffer.

Let's take a look at the source code implementation of Android
（Choreographer$FrameDisplayEventReceiver）：

private final class FrameDisplayEventReceiver extends DisplayEventReceiver implements Runnable {
        
    private boolean mHavePendingVsync;
    private long mTimestampNanos;
    private int mFrame;

    public FrameDisplayEventReceiver(Looper looper, int vsyncSource) {
            super(looper, vsyncSource);
    }
        
    @Override
    public void onVsync(long timestampNanos, long physicalDisplayId, int frame) {
        // 将 vsync 事件通过 handler 发送
        long now = System.nanoTime();
        if (timestampNanos > now) {
            Log.w(TAG, "Frame time is " + ((timestampNanos - now) * 0.000001f)
                        + " ms in the future!  Check that graphics HAL is generating vsync "
                        + "timestamps using the correct timebase.");
            timestampNanos = now;
        }
        // 判断是否还有挂起的信号未被处理
        if (mHavePendingVsync) {
            Log.w(TAG, "Already have a pending vsync event.  There should only be "
                        + "one at a time.");
        } else {
            mHavePendingVsync = true;
        }

        mTimestampNanos = timestampNanos;
        // 替换新的帧数据
        mFrame = frame;
        Message msg = Message.obtain(mHandler, this);
        msg.setAsynchronous(true);
        mHandler.sendMessageAtTime(msg, timestampNanos / TimeUtils.NANOS_PER_MS);
        }

    @Override
    public void run() {
        mHavePendingVsync = false;
        // 帧数据处理
        doFrame(mTimestampNanos, mFrame);
    }
}

Let's intuitively feel the refresh mechanism of Android through the systrace tool from the application point of view:

In the above figure, each vertical gray area is a synchronization of the vsync signal, and each gray area is 16.6ms. In other words, as long as a series of method references and executions of UI thread and RenderThread are completed in a grayscale area, the data rendering of this frame can be completed within 16.6ms. Let's look at the abnormal rendering logic after amplifying these signals in the following figure. The rendering in the red box has obviously exceeded the boundary of a vsync signal, which means that the rendering of this frame of data takes too long, and there is no rendering within a signal cycle. After the execution is completed, then there is a problem with the execution of this frame being reflected in the code execution. Android generally locates the problem by analyzing the stack of related functions. For example, the problem in the red box is caused by the custom view RPDetectCoreView If the application runs for a period of time, this situation occurs frequently, and the resulting user perception is the phenomenon of application freeze and frame drop.

High brush adaptation

After understanding the basic refresh mechanism of Android, let's take a look at how to adapt to the current models with high refresh attributes. An application or game can affect the refresh rate of the screen through the official Android SDK/NDK API. Why is it "affect" rather than "determine"? As mentioned earlier, the writing speed of CPU/GPU is uncontrollable, so these methods can only affect the frame rate of the screen.

Use the SDK that comes with Android

When adapting to the high-speed brush, we will first know which sensors the device itself supports, and then perform specific collection actions, just like registering device sensors. Similarly, for the screen refresh rate, we must first know the refresh rate supported by the screen and the current refresh rate, and then controllably adjust it to the refresh rate desired by the application. <br /

Then there are two ways to obtain the refresh rate, both of which are implemented by registering listeners:

1. DisplayManager.DisplayListener Query the refresh rate through Display.getRefreshRate()

// 注册屏幕变化监听器
public void registerDisplayListener(){
     
    DisplayManager displayManager = (DisplayManager) mContext.getSystemService(Context.DISPLAY_SERVICE);
    displayManager.registerDisplayListener(new DisplayManager.DisplayListener() {
        @Override
        public void onDisplayAdded(int displayId) {

        }

        @Override
        public void onDisplayRemoved(int displayId) {

        }

        @Override
        public void onDisplayChanged(int displayId) {

        }
    }, dealingHandler);
}

// 获取当前刷新率
public double getRefreshRate() {
    
  return ((WindowManager) mContext
      .getSystemService(Context.WINDOW_SERVICE))
      .getDefaultDisplay()
      .getRefreshRate();
}

2. You can also use the NDK's AChoreographer_registerRefreshRateCallback API

void AChoreographer_registerRefreshRateCallback(
  AChoreographer *choreographer,
  AChoreographer_refreshRateCallback,
  void *data
)

void AChoreographer_unregisterRefreshRateCallback(
  AChoreographer *choreographer,
  AChoreographer_refreshRateCallback,
  void *data
)

After obtaining the available refresh rate of the screen, you can try to set the refresh rate according to your business needs. The method is very simple, and no sample code description will be given here:

1. Use the setFrameRate() method of the SDK

   1. Surface.setFrameRate
   2. SurfaceControl.Transaction.setFrameRate

2. Use the _setFrameRate function of the NDK

   1. ANativeWindow_setFrameRate

   2. ASurfaceTransaction_setFrameRate

      FramePacingLibrary (FPL, frame synchronization library)

Here is another introduction to FramePacingLibrary (alias Swappy). Swappy enables smooth rendering and frame synchronization for games based on OpenGL and Vulkan rendering APIs. Here is another concept of frame synchronization that needs to be explained. Frame synchronization, as we mentioned above, the entire rendering pipeline of Android is from CPU to GPU to HAL screen display hardware. The frame synchronization here refers to the logical operation of the CPU and the rendering of the GPU and the display subsystem of the operating system and the underlying display hardware. synchronization between.
Why Swappy is suitable for games, because games contain a lot of CPU computing and rendering work, these computing strategies are obviously a high-cost work through development from 0 to 1, so Swappy is like most game engines on the market (Unity , Unreal, etc.) provides ready-made strategy mechanics to help games develop better and easier.

It can do:

• Add a presentation timestamp to each frame rendering, display the frame on time, and compensate for the stuttering phenomenon caused by the short game frame
• Inject the locking mechanism into the program, so that the displayed pipeline can keep up with the progress, without accumulating too much lag and delay that cause long frames (sync fence)
• Provides frame statistics to debug and profile programs

Let's briefly describe its principle with some pictures: the following picture is an ideal frame synchronization running at 30Hz on a 60Hz device, where each frame (A\B\C\D) is "properly" properly rendered to the screen . But this is not always the case in reality. For some short game frames, such as the C frame in the figure below, because the time-consuming is shorter, the B frame has not fully displayed the number of frames it should be preempted by the C frame , at the same time, the NB signal of the B frame triggers the re-display of the C frame, just like a runner trips over a small stone on the road and falls for a few meters in a fixed posture, the C frame will be displayed subsequently, resulting in a freeze.

The Swappy library solves this problem by adding a presentation timestamp, just like setting an alarm clock for each frame of data, and the alarm clock is not allowed to be displayed on the screen:

Summarize

Although it is not necessary to do too much code adaptation to adapt to the characteristics of high brush, the following aspects must be carefully considered:

1. When setting the screen refresh rate through the setFrameRate() method, there are still cases where the settings cannot take effect, such as Surfaces with higher priority have different frame rate settings, or the device is in power saving mode, etc. Therefore, we must also develop programs. Considering that if the setting fails, the program can also run normally;
2. Avoid calling the setFrameRate() method frequently. During the transition of each frame, if the frequent call will cause the frame drop problem, we need to obtain the necessary information through the API in advance to adjust to the correct frame rate at one time;
3. Do not fix a specific frame rate, but adjust the frame rate setting strategy according to the actual business scenario, with the goal of seamlessly transitioning between high and low screen refresh rates.

refer to

High Refresh Rate Rendering on Android

Frame Pacing Library

Android Frame-rate
​

Author: ES2049 / Dawn

The article can be reproduced at will, but please keep the original link.

You are very welcome to join ES2049 Studio. Please send your resume to [e-mail] .