Graphic composition of Android mouse cursor

I've always been curious about how the mouse cursor is implemented, it responds quickly, with little delay, and is not affected by the Android display system. I happened to be doing related work recently, and I followed the source code to study it.

This article refers to the Android 9.0 source code.

Start with Input

We're not talking about Input, we just want to see how the mouse cursor is drawn. However, Android puts the implementation of the mouse cursor into the Input, which seems reasonable as well. In Input, the cursor is implemented by class Sprite . The interpretation of Sprite in the source code is: graphic objects displayed on top of other layers. It appears that Sprite is not designed for cursors, but where in the source code it is indicated that in Android it also only serves cursors for input devices like mouse or touch. The definition of Sprite also provides only simple graphics operations.

 frameworks/base/libs/input/SpriteController.h

/*
 * A sprite is a simple graphical object that is displayed on-screen above other layers.
 * The basic sprite class is an interface.
 * The implementation is provided by the sprite controller.
 */
class Sprite : public RefBase {
protected:
    Sprite() { }
    virtual ~Sprite() { }

public:
    enum {
        // The base layer for pointer sprites.
        BASE_LAYER_POINTER = 0, // reserve space for 1 pointer

        // The base layer for spot sprites.
        BASE_LAYER_SPOT = 1, // reserve space for MAX_POINTER_ID spots
    };

    /* Sets the bitmap that is drawn by the sprite.
     * The sprite retains a copy of the bitmap for subsequent rendering. */
    virtual void setIcon(const SpriteIcon& icon) = 0;

    inline void clearIcon() {
        setIcon(SpriteIcon());
    }

    /* Sets whether the sprite is visible. */
    virtual void setVisible(bool visible) = 0;

    /* Sets the sprite position on screen, relative to the sprite's hot spot. */
    virtual void setPosition(float x, float y) = 0;

    /* Sets the layer of the sprite, relative to the system sprite overlay layer.
     * Layer 0 is the overlay layer, > 0 appear above this layer. */
    virtual void setLayer(int32_t layer) = 0;

    /* Sets the sprite alpha blend ratio between 0.0 and 1.0. */
    virtual void setAlpha(float alpha) = 0;

    /* Sets the sprite transformation matrix. */
    virtual void setTransformationMatrix(const SpriteTransformationMatrix& matrix) = 0;
};

The class that controls the cursor is called SpriteController and the PointerController uses this class to display the cursor. Here we only care about the composition of the cursor graphics, the real way to display and update the cursor is SpriteController::doUpdateSprites() .

 frameworks/base/libs/input/SpriteController.cpp

void SpriteController::doUpdateSprites() {
    // 从invalidatedSprites 中收集需要更新的 Sprite
    Vector<SpriteUpdate> updates;
    size_t numSprites;
    { // acquire lock
        AutoMutex _l(mLock);

        numSprites = mLocked.invalidatedSprites.size();
        for (size_t i = 0; i < numSprites; i++) {
            const sp<SpriteImpl>& sprite = mLocked.invalidatedSprites.itemAt(i);

            updates.push(SpriteUpdate(sprite, sprite->getStateLocked()));
            sprite->resetDirtyLocked();
        }
        mLocked.invalidatedSprites.clear();
    } // release lock

    // surfaces 未创建或丢失时，重新创建 surface
    bool surfaceChanged = false;
    for (size_t i = 0; i < numSprites; i++) {
        SpriteUpdate& update = updates.editItemAt(i);

        if (update.state.surfaceControl == NULL && update.state.wantSurfaceVisible()) {
            update.state.surfaceWidth = update.state.icon.bitmap.width();
            update.state.surfaceHeight = update.state.icon.bitmap.height();
            update.state.surfaceDrawn = false;
            update.state.surfaceVisible = false;
            // 创建 Surface，我们这次的关注点
            update.state.surfaceControl = obtainSurface(
                    update.state.surfaceWidth, update.state.surfaceHeight);
            if (update.state.surfaceControl != NULL) {
                update.surfaceChanged = surfaceChanged = true;
            }
        }
    }

    // 如果需要，重新调整 sprites 大小
    SurfaceComposerClient::Transaction t;
    bool needApplyTransaction = false;
    for (size_t i = 0; i < numSprites; i++) {
        ......
            if (update.state.surfaceWidth < desiredWidth
                    || update.state.surfaceHeight < desiredHeight) {
                needApplyTransaction = true;

                t.setSize(update.state.surfaceControl,
                        desiredWidth, desiredHeight);
                ......
            }
        }
    }
    if (needApplyTransaction) {
        t.apply();
    }

    // 如果需要，重画 sprites
    for (size_t i = 0; i < numSprites; i++) {
        SpriteUpdate& update = updates.editItemAt(i);

        if ((update.state.dirty & DIRTY_BITMAP) && update.state.surfaceDrawn) {
            update.state.surfaceDrawn = false;
            update.surfaceChanged = surfaceChanged = true;
        }

        if (update.state.surfaceControl != NULL && !update.state.surfaceDrawn
                && update.state.wantSurfaceVisible()) {
            sp<Surface> surface = update.state.surfaceControl->getSurface();
            ANativeWindow_Buffer outBuffer;
            ......
                // 使用 SKIA 画图
                SkBitmap surfaceBitmap;
                ssize_t bpr = outBuffer.stride * bytesPerPixel(outBuffer.format);
                surfaceBitmap.installPixels(SkImageInfo::MakeN32Premul(outBuffer.width, outBuffer.height),
                                            outBuffer.bits, bpr);

                SkCanvas surfaceCanvas(surfaceBitmap);

                SkPaint paint;
                paint.setBlendMode(SkBlendMode::kSrc);
                surfaceCanvas.drawBitmap(update.state.icon.bitmap, 0, 0, &paint);

                if (outBuffer.width > update.state.icon.bitmap.width()) {
                    paint.setColor(0); // transparent fill color
                    surfaceCanvas.drawRect(SkRect::MakeLTRB(update.state.icon.bitmap.width(), 0,
                            outBuffer.width, update.state.icon.bitmap.height()), paint);
                }
                if (outBuffer.height > update.state.icon.bitmap.height()) {
                    paint.setColor(0); // transparent fill color
                    surfaceCanvas.drawRect(SkRect::MakeLTRB(0, update.state.icon.bitmap.height(),
                            outBuffer.width, outBuffer.height), paint);
                }
                ......
    }

    // 根据 dirty 值来设置 Surface
    needApplyTransaction = false;
    for (size_t i = 0; i < numSprites; i++) {
        SpriteUpdate& update = updates.editItemAt(i);

        bool wantSurfaceVisibleAndDrawn = update.state.wantSurfaceVisible()
                && update.state.surfaceDrawn;
        bool becomingVisible = wantSurfaceVisibleAndDrawn && !update.state.surfaceVisible;
        bool becomingHidden = !wantSurfaceVisibleAndDrawn && update.state.surfaceVisible;
        if (update.state.surfaceControl != NULL && (becomingVisible || becomingHidden
                || (wantSurfaceVisibleAndDrawn && (update.state.dirty & (DIRTY_ALPHA
                        | DIRTY_POSITION | DIRTY_TRANSFORMATION_MATRIX | DIRTY_LAYER
                        | DIRTY_VISIBILITY | DIRTY_HOTSPOT))))) {
        ......
    }

    if (needApplyTransaction) {
        status_t status = t.apply();
        if (status) {
            ALOGE("Error applying Surface transaction");
        }
    }
......
}

A cursor update will involve so much code logic, it is really not easy to see the UI. No other logic lines, this time we only care about the layer of the cursor. The above code creates Surface by obtainSurface() .

 frameworks/base/libs/input/SpriteController.cpp

sp<SurfaceControl> SpriteController::obtainSurface(int32_t width, int32_t height) {
    ensureSurfaceComposerClient();

    sp<SurfaceControl> surfaceControl = mSurfaceComposerClient->createSurface(
            String8("Sprite"), width, height, PIXEL_FORMAT_RGBA_8888,
            ISurfaceComposerClient::eHidden |
            ISurfaceComposerClient::eCursorWindow);
    if (surfaceControl == NULL || !surfaceControl->isValid()) {
        ALOGE("Error creating sprite surface.");
        return NULL;
    }
    return surfaceControl;
}

Here we need to focus on the parameters in the createSurface() method flags . This flags in the Sprite has set eHidden and eCursorWindow , which indicate that the created Surface is hidden and marked for use by Cursor.

Come to Surface

A Surface is created for the cursor in Input, and it is identified as a Surface used by Cursor. After that, Surface will do special processing to the cursor layer according to the situation. The keyword here is Cursor .

We still use the cursor layer as the main line for tracking, let's continue to look at createSurface() . After a series of Binder calls and Message transmission, the layer creation is finally completed through SurfaceFlinger's createLayer() .

 frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

status_t SurfaceFlinger::createLayer(const String8& name, const sp<Client>& client, uint32_t w,
                                     uint32_t h, PixelFormat format, uint32_t flags,
                                     int32_t windowType, int32_t ownerUid, sp<IBinder>* handle,
                                     sp<IGraphicBufferProducer>* gbp,
                                     const sp<IBinder>& parentHandle,
                                     const sp<Layer>& parentLayer) {
    ......
    switch (flags & ISurfaceComposerClient::eFXSurfaceMask) {
        // 普通图层
        case ISurfaceComposerClient::eFXSurfaceNormal:
            result = createBufferLayer(client,
                    uniqueName, w, h, flags, format,
                    handle, gbp, &layer);

            break;
        // 纯色图层
        case ISurfaceComposerClient::eFXSurfaceColor:
            result = createColorLayer(client,
                    uniqueName, w, h, flags,
                    handle, &layer);
            break;
        default:
            result = BAD_VALUE;
            break;
    }
    ......
    // Client中通过Layer管理Surface，将创建的Layer加入到LayerStack中
    result = addClientLayer(client, *handle, *gbp, layer, parentHandle, parentLayer);
    if (result != NO_ERROR) {
        return result;
    }
    mInterceptor->saveSurfaceCreation(layer);

    setTransactionFlags(eTransactionNeeded);
    return result;
}

createLayer() , the cursor is a normal layer, so just call createBufferLayer() to create it.

 frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

status_t SurfaceFlinger::createBufferLayer(const sp<Client>& client,
        const String8& name, uint32_t w, uint32_t h, uint32_t flags, PixelFormat& format,
        sp<IBinder>* handle, sp<IGraphicBufferProducer>* gbp, sp<Layer>* outLayer)
{
    ......
    // 创建一个BufferLayer
    sp<BufferLayer> layer = new BufferLayer(this, client, name, w, h, flags);
    // 设置Buffer属性
    status_t err = layer->setBuffers(w, h, format, flags);
    if (err == NO_ERROR) {
        *handle = layer->getHandle(); // 获取Layer的句柄
        *gbp = layer->getProducer(); // 获取GraphicBufferProducer对象 
        *outLayer = layer;
    }

    ALOGE_IF(err, "createBufferLayer() failed (%s)", strerror(-err));
    return err;
}

Among them layer->setBuffers() set the properties of the BufferLayer. It can be seen that when applying for a Cursor layer, mPotentialCursor is set to true , indicating that the BufferLayer is used as a Cursor.

 frameworks/native/services/surfaceflinger/BufferLayer.cpp

status_t BufferLayer::setBuffers(uint32_t w, uint32_t h, PixelFormat format, uint32_t flags) {
    ......
    mFormat = format;

    mPotentialCursor = (flags & ISurfaceComposerClient::eCursorWindow) ? true : false;
    mProtectedByApp = (flags & ISurfaceComposerClient::eProtectedByApp) ? true : false;
    mCurrentOpacity = getOpacityForFormat(format);

    mConsumer->setDefaultBufferSize(w, h);
    mConsumer->setDefaultBufferFormat(format);
    mConsumer->setConsumerUsageBits(getEffectiveUsage(0));

    return NO_ERROR;
}

Cursor operation in SurfaceFlinger

As mentioned above, the most core properties of Cursor Layer mPotentialCursor , createSurface() just set this property, which is really used in the SurfaceFlinger rendering process. Then I found that if I want to understand this thing, I need to figure out the Android graphics synthesis first, which is a huge project. Borrow a picture for your own research if you are interested.

However, time is limited, what to do? My solution is to search for keywords. After searching the keyword Cursor , you can get some related operations. After SurfaceFlinger receives the Vsync signal, it will call handleMessageRefresh() to refresh the display.

 frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

void SurfaceFlinger::handleMessageRefresh() {
    ......
    preComposition(refreshStartTime); //合成预处理
    rebuildLayerStacks(); //重新构建LayerStacks
    setUpHWComposer(); //更新HWComposer的图层和属性
    doDebugFlashRegions(); //图形绘制的debug模式
    doTracing("handleRefresh");
    logLayerStats();
    doComposition(); //合成所有图层
    postComposition(refreshStartTime); //合成后处理
    ......
}

We still only care about the operation of the Cursor, which is in the layer controlled by the HWComposer.

 frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

void SurfaceFlinger::setUpHWComposer() {
    ......
    // 遍历所有的DisplayDevice，为绘制做准备
    for (size_t dpy=0 ; dpy<mDisplays.size() ; dpy++) {
        ......
        mDisplays[dpy]->beginFrame(mustRecompose);

        if (mustRecompose) {
            mDisplays[dpy]->lastCompositionHadVisibleLayers = !empty;
        }
    }

    // 设置HWC Layer
    if (CC_UNLIKELY(mGeometryInvalid)) {
        mGeometryInvalid = false;
        for (size_t dpy=0 ; dpy<mDisplays.size() ; dpy++) {
            ......
                for (size_t i = 0; i < currentLayers.size(); i++) {
                    const auto& layer = currentLayers[i];
                    // 尝试创建HWC Layer，如果失败则强制OpenGL渲染
                    if (!layer->hasHwcLayer(hwcId)) {
                        if (!layer->createHwcLayer(getBE().mHwc.get(), hwcId)) {
                            layer->forceClientComposition(hwcId);
                            continue;
                        }
                    }

                    // 设置HWC Layer的显示区域、合成模式、Alpha、Order等
                    layer->setGeometry(displayDevice, i);
                    // HWC被禁止或绘制debug模式时，强制OpenGL渲染
                    if (mDebugDisableHWC || mDebugRegion) {
                        layer->forceClientComposition(hwcId);
                    }
        ......
    }

    // 准备HWC需要渲染的数据
    for (size_t displayId = 0; displayId < mDisplays.size(); ++displayId) {
        auto& displayDevice = mDisplays[displayId];
        const auto hwcId = displayDevice->getHwcDisplayId();
        ......
            //调用 setPerFrameData方法
            layer->setPerFrameData(displayDevice);
        ......
    }
    ......
    for (size_t displayId = 0; displayId < mDisplays.size(); ++displayId) {
        ......
        // 尝试进行显示
        status_t result = displayDevice->prepareFrame(*getBE().mHwc);
        ......
    }
}

Among them setPerFrameData() Complete the relevant settings of HWComposer and prepare for display.

 frameworks/native/services/surfaceflinger/BufferLayer.cpp

void BufferLayer::setPerFrameData(const sp<const DisplayDevice>& displayDevice) {
    ......
    // 设置可见区域
    auto error = hwcLayer->setVisibleRegion(visible);
    ......
    // 设置刷新区域
    error = hwcLayer->setSurfaceDamage(surfaceDamageRegion);
    ......
    // Sideband layers设置
    if (getBE().compositionInfo.hwc.sidebandStream.get()) {
        setCompositionType(hwcId, HWC2::Composition::Sideband);
        error = hwcLayer->setSidebandStream(getBE().compositionInfo.hwc.sidebandStream->handle());
        ......
        return;
    }

    if (mPotentialCursor) {
        // Cursor layers设置
        setCompositionType(hwcId, HWC2::Composition::Cursor);
    } else {
        // Device layers设置
        setCompositionType(hwcId, HWC2::Composition::Device);
    }

    // 设置色彩空间
    error = hwcLayer->setDataspace(mCurrentDataSpace);
    if (error != HWC2::Error::None) {
        ALOGE("[%s] Failed to set dataspace %d: %s (%d)", mName.string(), mCurrentDataSpace,
              to_string(error).c_str(), static_cast<int32_t>(error));
    }

    // 获取HDR数据并设置到HWC中
    const HdrMetadata& metadata = mConsumer->getCurrentHdrMetadata();
    error = hwcLayer->setPerFrameMetadata(displayDevice->getSupportedPerFrameMetadata(), metadata);
    ......
    // 获取渲染的数据buffer和Fence，设置到HWC中
    sp<GraphicBuffer> hwcBuffer;
    hwcInfo.bufferCache.getHwcBuffer(getBE().compositionInfo.mBufferSlot,
                                     getBE().compositionInfo.mBuffer, &hwcSlot, &hwcBuffer);

    auto acquireFence = mConsumer->getCurrentFence();
    error = hwcLayer->setBuffer(hwcSlot, hwcBuffer, acquireFence);
    ......
}

We finally found what we wanted to see mPotentialCursor , which tells HWC2 that this is a CursorLayer. In addition, the operation of CursorLayer is no different from DeviceLayer. Therefore, SurfaceFlinger expects HWComposer to process differently according to the type of Layer. The Layer types currently supported by HWC2 are:

1. HWC2_COMPOSITION_CLIENT: Do not composite layers through HWC hardware. The GPU composites such layers into an image buffer, which is then passed to the HWC.
2. HWC2_COMPOSITION_DEVICE: Use HWC hardware to composite layers.
3. HWC2_COMPOSITION_SOLID_COLOR: used to process ColorLayer data, if HWC does not support it, use CLIENT method instead.
4. HWC2_COMPOSITION_CURSOR: used to process CursorLayer data, the location is set asynchronously by setCursorPosition . If HWC does not support it, use CLIENT or DEVICE mode instead.
5. HWC2_COMPOSITION_SIDEBAND: For this Layer, content updates need to be provided by an external mechanism, such as video data for TV signals. If HWC does not support it, use CLIENT or DEVICE instead, but it may not display correctly.

The Cursor Layer also has an important operation, setCursorPosition() , this method is used to set the position of the Cursor, and the specific implementation is still in HWComposer. When the user process updates the Surface graphics, SurfaceFlinger will send INVALIDATE message to the corresponding Layer. The message handler calls handleTransaction() and handlePageFlip() to update the Layer object. handleTransaction()

Used to handle Layer and display device changes, it continues to call handleTransactionLocked() .

 frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

void SurfaceFlinger::handleTransactionLocked(uint32_t transactionFlags)
{
    ......
    // 处理Layer的变化
    if (transactionFlags & eTraversalNeeded) {
        ......
    }

    // 处理显示设备的变化
    if (transactionFlags & eDisplayTransactionNeeded) {
        processDisplayChangesLocked();
        processDisplayHotplugEventsLocked();
    }

    // 设置transform hint
    if (transactionFlags & (eDisplayLayerStackChanged|eDisplayTransactionNeeded)) {
        ......
   }

   //处理Layer的增减
   if (mLayersAdded) {
        ......
    }

    if (mLayersRemoved) {
        ......
    }

    commitTransaction();

    // 更新光标位置
    updateCursorAsync();
}

We found the place where Cursor is updated, and SurfaceFlinger updates the cursor position synchronously when updating the graphics. After that, when Vsync arrives, the update display of the image is completed.

 frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp

void SurfaceFlinger::updateCursorAsync()
{
    for (size_t displayId = 0; displayId < mDisplays.size(); ++displayId) {
        ......
        // 调用Layer的对应方法
        for (auto& layer : displayDevice->getVisibleLayersSortedByZ()) {
            layer->updateCursorPosition(displayDevice);
        }
    }
}

 frameworks/native/services/surfaceflinger/Layer.cpp

void Layer::updateCursorPosition(const sp<const DisplayDevice>& displayDevice) {
    // HWC Layer不存在或者不是Cursor Layer，不做处理
    auto hwcId = displayDevice->getHwcDisplayId();
    if (getBE().mHwcLayers.count(hwcId) == 0 ||
        getCompositionType(hwcId) != HWC2::Composition::Cursor) {
        return;
    }
    ......
    // 获取图层的位置
    Rect bounds = reduce(win, s.activeTransparentRegion);
    Rect frame(getTransform().transform(bounds));
    frame.intersect(displayDevice->getViewport(), &frame);
    if (!s.finalCrop.isEmpty()) {
        frame.intersect(s.finalCrop, &frame);
    }
    auto& displayTransform(displayDevice->getTransform());
    auto position = displayTransform.transform(frame);

    // 调用HWC的方法来设置图层位置
    auto error = getBE().mHwcLayers[hwcId].layer->setCursorPosition(position.left, position.top);
}

Arrive at HWComposer

A lot of code has been analyzed above, but not much is really related to Cursor. The real implementation of CursorLayer is still in HWComposer. But the implementation of HWComposer is platform-dependent, and different platforms may implement CursorLayer differently. The efficient way is to use an independent hardware OSD to display the CursorLayer, and then superimpose the CursorLayer to the UI display layer by means of hardware synthesis. Using this method, the moving efficiency of the cursor is also very high, as long as the position displayed by the hardware OSD can be changed. If there is no separate hardware OSD to use, you can only do software overlay on the standard display layer, or use the GPU to overlay.

Due to the privacy of the implementation related to the platform, the analysis will not be continued here.

Reference documentation:

SurfaceFlinger of Android GUI system )

Android SurfaceFlinger Learning - HWComposer Composition Workflow

Talking about Mobile Display Technology