In-depth optimization of Flutter streaming scene

Author: Jiang Zejun (Yi Yi)

Taote uses Flutter in many business scenarios, and the business scenario itself has a certain complexity, which makes Flutter freeze and frame skipping during the sliding browsing process of low-end machine streaming scenes compared to the use of native (Android/iOS) development obvious. After analyzing the performance problems of the business layer at each stage of the Flutter rendering process and performing a series of in-depth optimizations, the average frame rate has reached 50 frames and surpassed the native performance, but the stall rate is still not up to the maximum. For a good experience, we have encountered bottlenecks and technical challenges that are difficult to break through, and we need to make technical attempts and breakthroughs.

This article will describe the underlying principles, optimization ideas, optimization strategies of actual scenarios, core technology implementation, optimization results, etc., and hope to bring you some inspiration and help. We also welcome more exchanges and corrections to build a beautiful Flutter technical community.

Rendering mechanism

Native vs Flutter

Flutter itself is based on the native system, so the rendering mechanism is very close to Native. Quoting Google Flutter team Xiao Yu shared [1], as shown in the following figure:

Rendering process

As shown in the left, Flutter goes through 8 stages after receiving the VSync signal. After the Compositing stage, the data is submitted to the GPU.

In the Semantics phase, the information that the RenderObject marked needs to be semantically updated is passed to the system to realize auxiliary functions. The semantic interface can help users with visual impairments to understand the UI content, and it is not related to the overall drawing process.

The Finalize Tree stage will unmount all the inactive Elements added to _inactiveElements, which is not related to the overall drawing process.

Therefore, the overall rendering process of Flutter mainly focuses on the stage in the right center of the above figure:

GPU Vsync

After the Flutter Engine receives the vertical synchronization signal, it will notify the Flutter Framework to begin Frame and enter the Animation stage.

Animation

The transientCallbacks callback is mainly executed. Flutter Engine will notify Flutter Framework to drawFrame and enter the Build phase.

Build

Construct the data structure of the UI component tree to be presented, that is, create the corresponding Widget and the corresponding Element.

Layout

The purpose is to calculate the actual size of the space occupied by each node for layout, and then update the layout information of all dirty render objects.

Compositing Bits

Perform an update operation on the RenderObject that needs to be updated.

Paint

Generate Layer Tree. It cannot be used directly to generate Layer Tree. It also needs Compositing to synthesize into a Scene and perform Rasterize rasterization processing. The reason for level merging is because there are generally many levels of Flutter, and it is very inefficient to directly pass each layer to the GPU, so composite will be used first to improve efficiency. After rasterization, it will be handed over to Flutter Engine for processing.

Compositing

Synthesize the Layout Tree into a Scene, and create a raster image of the current state of the scene, that is, perform Rasterize rasterization, and then submit it to the Flutter Engine. Finally, Skia submits the data to the GPU through the Open GL or Vulkan interface, and the GPU is processed for display.

core rendering stage

Widget

Most of what we write is Widget. Widget can actually be understood as a data structure of a component tree, which is the main part of the Build phase. Among them, the depth of Widget Tree, the reasonableness of StatefulWidget's setState, whether there is unreasonable logic in the build function, and the use of related Widgets that call saveLayer often become performance issues.

Element

Associate Widget and RenderObject to generate the Widget corresponding to the Element to store the context information. Flutter traverses the Element to generate the RenderObject view tree to support the UI structure.

RenderObject

RenderObject determines the layout information in the Layout phase, and generates the corresponding Layer in the Paint phase, which shows its importance. So most of the graphics performance optimization in Flutter happens here. The data constructed by the RenderObject tree will be added to the LayerTree required by the Engine.

Performance optimization ideas

Understanding the underlying rendering mechanism and core rendering stages, optimization can be divided into three layers:

I will not specifically expand on the optimization details of each layer here. This article mainly describes from the actual scene.

Streaming scene

Principles of flow components

Under native development, RecyclerView/UICollectionView is usually used to develop list scenes; under Flutter development, Flutter Framework also provides ListView components, which are actually SliverList.

Core source code

We analyze from the core source code of SliverList:

class SliverList extends SliverMultiBoxAdaptorWidget {

  @override
  RenderSliverList createRenderObject(BuildContext context) {
    final SliverMultiBoxAdaptorElement element = context as SliverMultiBoxAdaptorElement;
    return RenderSliverList(childManager: element);
  }
}

abstract class SliverMultiBoxAdaptorWidget extends SliverWithKeepAliveWidget {

  final SliverChildDelegate delegate;

  @override
  SliverMultiBoxAdaptorElement createElement() => SliverMultiBoxAdaptorElement(this);

  @override
  RenderSliverMultiBoxAdaptor createRenderObject(BuildContext context);
}

By looking at the source code of SliverList, we can see that SliverList is a RenderObjectWidget with the following structure:

Let's first look at the core source code of its RenderObject:

class RenderSliverList extends RenderSliverMultiBoxAdaptor {

  RenderSliverList({
    @required RenderSliverBoxChildManager childManager,
  }) : super(childManager: childManager);

  @override
  void performLayout(){
    ...
    //父节点对子节点的布局限制
    final SliverConstraints constraints = this.constraints;
    final double scrollOffset = constraints.scrollOffset + constraints.cacheOrigin;
    final double remainingExtent = constraints.remainingCacheExtent;
    final double targetEndScrollOffset = scrollOffset + remainingExtent;
    final BoxConstraints childConstraints = constraints.asBoxConstraints();
    ...
    insertAndLayoutLeadingChild(childConstraints, parentUsesSize: true);
    ...
    insertAndLayoutChild(childConstraints,after: trailingChildWithLayout,parentUsesSize: true);
    ...
    collectGarbage(leadingGarbage, trailingGarbage);
    ...
  }
}

abstract class RenderSliverMultiBoxAdaptor extends RenderSliver ...{
  @protected
  RenderBox insertAndLayoutChild(BoxConstraints childConstraints, {@required RenderBox after,...}) {
    _createOrObtainChild(index, after: after);
    ...
  }

  RenderBox insertAndLayoutLeadingChild(BoxConstraints childConstraints, {@required RenderBox after,...}) {
    _createOrObtainChild(index, after: after);
    ...
  }

  @protected
  void collectGarbage(int leadingGarbage, int trailingGarbage) {
    _destroyOrCacheChild(firstChild);
    ...
  }

  void _createOrObtainChild(int index, { RenderBox after }) {
    _childManager.createChild(index, after: after);
    ...
  }

  void _destroyOrCacheChild(RenderBox child) {
    if (childParentData.keepAlive) {
      //为了更好的性能表现不会进行keepAlive，走else逻辑.
      ...
    } else {
      _childManager.removeChild(child);
      ...
    }
  }
}

Looking at the source code of RenderSliverList, it is found that the creation and removal of child are carried out through its parent class RenderSliverMultiBoxAdaptor. The RenderSliverMultiBoxAdaptor is carried out through _childManager, namely SliverMultiBoxAdaptorElement, and the layout size is limited by the parent node during the entire SliverList drawing process.

In the streaming scenario:

• During the sliding process, SliverMultiBoxAdaptorElement.createChild is used to create a new child that enters the visual area; (that is, each item card in the business scene)
• In the sliding process, SliverMultiBoxAdaptorElement.removeChild is used to remove the old child that is not in the visible area.

Let's take a look at the core source code of SliverMultiBoxAdaptorElement:

class SliverMultiBoxAdaptorElement extends RenderObjectElement implements RenderSliverBoxChildManager {
  final SplayTreeMap<int, Element> _childElements = SplayTreeMap<int, Element>();

  @override
  void createChild(int index, { @required RenderBox after }) {
    ...
    Element newChild = updateChild(_childElements[index], _build(index), index);
    if (newChild != null) {
      _childElements[index] = newChild;
    } else {
      _childElements.remove(index);
    }
    ...
  }

  @override
  void removeChild(RenderBox child) {
    ...
    final Element result = updateChild(_childElements[index], null, index);
    _childElements.remove(index);
    ...
  }

  @override
  Element updateChild(Element child, Widget newWidget, dynamic newSlot) {
    ...
    final Element newChild = super.updateChild(child, newWidget, newSlot);
    ...
  }
}

By looking at the source code of SliverMultiBoxAdaptorElement, we can find that the operation of the child is actually carried out through the updateChild of the parent class Element.

Next, let's take a look at the core code of Element:

abstract class Element extends DiagnosticableTree implements BuildContext {
  @protected
  Element updateChild(Element child, Widget newWidget, dynamic newSlot) {
    if (newWidget == null) {
      if (child != null)
        deactivateChild(child);
      return null;
    }
    Element newChild;
    if (child != null) {
      ...
      bool hasSameSuperclass = oldElementClass == newWidgetClass;;
      if (hasSameSuperclass && child.widget == newWidget) {
        if (child.slot != newSlot)
          updateSlotForChild(child, newSlot);
        newChild = child;
      } else if (hasSameSuperclass && Widget.canUpdate(child.widget, newWidget)) {
        if (child.slot != newSlot)
          updateSlotForChild(child, newSlot);
        child.update(newWidget);
        newChild = child;
      } else {
        deactivateChild(child);
        newChild = inflateWidget(newWidget, newSlot);
      }
    } else {
      newChild = inflateWidget(newWidget, newSlot);
    }
    ...
    return newChild;
  }

  @protected
  Element inflateWidget(Widget newWidget, dynamic newSlot) {
    ...
    final Element newChild = newWidget.createElement();
    newChild.mount(this, newSlot);
    ...
    return newChild;
  }

  @protected
  void deactivateChild(Element child) {
    child._parent = null;
    child.detachRenderObject(); 
    owner._inactiveElements.add(child); // this eventually calls child.deactivate() & child.unmount()
    ...
  }
}

You can see that the mount and detachRenderObject of Element are mainly called. Here we look at the source code of these two methods of RenderObjectElement:

abstract class RenderObjectElement extends Element {
  @override
  void mount(Element parent, dynamic newSlot) {
    super.mount(parent, newSlot);
    ...
    _renderObject = widget.createRenderObject(this);
    attachRenderObject(newSlot);
    ...
  }

  @override
  void attachRenderObject(dynamic newSlot) {
    ...
    _ancestorRenderObjectElement = _findAncestorRenderObjectElement();
    _ancestorRenderObjectElement?.insertChildRenderObject(renderObject, newSlot);
    ...
  }

  @override
  void detachRenderObject() {
    if (_ancestorRenderObjectElement != null) {
      _ancestorRenderObjectElement.removeChildRenderObject(renderObject);
      _ancestorRenderObjectElement = null;
    }
    ...
  }
}

By looking at the traceability of the above source code, we can see:

In the streaming scenario:

• The creation of a new child that enters the visual area during the sliding process is by creating a brand new Element and mounting it to the Element Tree; then the corresponding RenderObject is created, and _ancestorRenderObjectElement?.insertChildRenderObject is called;
• During the sliding process, the old child that is not in the visible area is removed, and the corresponding Element is removed and mounted from the Element Tree unmount; then _ancestorRenderObjectElement.removeChildRenderObject is called.

In fact, this _ancestorRenderObjectElement is SliverMultiBoxAdaptorElement, let's look at SliverMultiBoxAdaptorElement:

class SliverMultiBoxAdaptorElement extends RenderObjectElement implements RenderSliverBoxChildManager {

  @override
  void insertChildRenderObject(covariant RenderObject child, int slot) {
    ...
    renderObject.insert(child as RenderBox, after: _currentBeforeChild);
    ...
  }

  @override
  void removeChildRenderObject(covariant RenderObject child) {
    ...
    renderObject.remove(child as RenderBox);
  }
}

In fact, all the methods of ContainerRenderObjectMixin are called. Let's take a look at ContainerRenderObjectMixin:

mixin ContainerRenderObjectMixin<ChildType extends RenderObject, ... {
  void insert(ChildType child, { ChildType after }) {
        ...
    adoptChild(child);// attach render object
    _insertIntoChildList(child, after: after);
  }

  void remove(ChildType child) {
    _removeFromChildList(child);
    dropChild(child);// detach render object
  }
}

ContainerRenderObjectMixin maintains a doubly linked list to hold the current children RenderObject, so the creation and removal during the sliding process will be synchronously added and removed in the doubly linked list of ContainerRenderObjectMixin.

finally sums it up:

• The creation of a new child that enters the visual area during the sliding process is by creating a brand new Element and mounting it to the Element Tree; then creating the corresponding RenderObject, attaching it to the Render Tree by calling SliverMultiBoxAdaptorElement.insertChildRenderObject, and simultaneously adding the RenderObject to the In the double-linked list of the mixin of SliverMultiBoxAdaptorElement;
• Remove the old child that is not in the visible area during the sliding process, remove and mount the corresponding Element from the Element Tree unmount; then use SliverMultiBoxAdaptorElement.removeChildRenderObject to remove the corresponding RenderObject from the double-linked list of the mixin and synchronize it RenderObject is detached from Render Tree.

Rendering principle

Through the analysis of the core source code, we can classify the elements of the streaming scene as follows:

Let’s look at the overall rendering process and mechanism when the user swipes up to view more product cards and triggers the loading of the next page of data for display:

• When sliding upwards, the top 0 and 1 cards move out of the Viewport area (Visible Area + Cache Area). We define it as entering the Detach Area. After entering the Detach Area, the corresponding RenderObject is detached from the Render Tree, and the corresponding Element is removed from the Element Tree unmount removes the mount and synchronously removes it from the doubly linked list;
• By monitoring the sliding calculation position of the ScrollController to determine whether it is necessary to start loading the next page of data, then the loading Footer component at the bottom will enter the visible area or the cache area, and the childCount of the SliverChildBuilderDelegate needs to be +1, and the last child returns to the Loading Footer component and calls at the same time setState refreshes the entire SliverList. update will call performRebuild to rebuild, and all update operations will be performed in the user's visual area in the middle part; then the Loading Footer component will be created corresponding to the new Element and RenderObject, and will be added to the doubly linked list synchronously;
• When the loading is finished and the data returns, setState will be called again to refresh the entire SliverList, update will call performRebuild to rebuild, and the middle part will be updated in the user view area; then the Loading Footer component will detach the corresponding RenderObject from the Render Tree Unmount, and remove and mount the corresponding Element from Element Tree unmount, and remove it from the doubly linked list synchronously;
• The new item at the bottom will enter the visual area or the cache area. New Element and RenderObject need to be created and added to the doubly linked list synchronously.

Optimization Strategy

The scenario where the user swipes up to view more product cards and triggers the loading of the next page of data for display can be optimized from five directions:

Load More

Calculate continuously by monitoring the sliding of ScrollController. It is best to automatically recognize the need to load the next page of data and initiate the loadMore() callback without judgment. New ReuseSliverChildBuilderDelegate adds loadMore and footerBuilder at the same level as item Builder, and includes the Loading Footer component by default. In SliverMultiBoxAdaptorElement.createChild(int index,...) it is judged whether it is necessary to dynamically call back loadMore() and automatically build the footer component.

partial refresh

Refer to the smoothness optimization of the long list before Xianyu [2], call setState to refresh the entire SliverList after the next page of data comes back, causing the middle part to be updated in the user's visual area, and actually only need to refresh the newly created Part, optimize the part of SliverMultiBoxAdaptorElement.update(SliverMultiBoxAdaptorWidget newWidget) to achieve partial refresh, as shown below:

[]()

Element & RenderObject reuse

With reference to Xianyu’s previous smoothness optimization in long lists [2] and Google Android RecyclerView ViewHolder reuse design [3], when a new item is created, a ViewHolder similar to Android RecyclerView can be used to hold and reuse components . Based on the analysis of the principle of the rendering mechanism, Widget in Flutter can actually be understood as a data structure of a component tree, that is, more of a data expression of the component structure. We need to cache and hold the Element and RenderObject component types of the removed item. When creating a new item, take priority from the cache and hold for reuse. At the same time, it does not destroy Flutter's own design of Key. When the item uses Key, only the Element and RenderObject with the same Key are reused. However, the data in the streaming scene list are all different data, so if the Key is used in the streaming scene, it cannot be reused. If you reuse Element and RenderObject, it is not recommended to use Key for the item component.

We add a cache state to the classification of Element in the original streaming scene:

[]()

As shown below:

[]()

GC inhibits

Dart has its own GC mechanism, similar to Java's generational collection, which can suppress GC during the sliding process and customize the GC collection algorithm. For this discussion with Google’s Flutter experts, in fact, unlike Java, Dart does not have multi-threaded switching for garbage collection. Single-threaded (main isolate) garbage collection is faster and lighter. At the same time, it requires in-depth transformation of the Flutter Engine. , Considering that the benefits are not significant, we will not proceed for the time being.

Asynchronization

Flutter Engine restricts non-Main Isolate from calling Platform-related APIs, and puts all the logic that does not interact with Platform Thread into the new isolate. Frequent Isolate creation and recycling will also have a certain impact on performance. Flutter compute<Q, R>( isolates.ComputeCallback<Q, R> callback, Q message, {String debugLabel }) A new Isolate will be created every time it is called, and it will be recycled after the task is executed. A thread pool-like Isolate is implemented to process non-view tasks. After the actual test, the improvement is not obvious, so I will not start to talk about it.

Core technology realization

We can classify the code of the call link as follows:

[]()

All the rendering cores are in the SliverMultiBoxAdaptorElement inherited from RenderObjectElement, without destroying the original function design and the structure of the Flutter Framework. The ReuseSliverMultiBoxAdaptorElement element is added to implement the optimization strategy, and it can be used directly with the original SliverList's RenderSliverList or customized Use of RenderObject of flow components (for example: waterfall flow components).

Partial refresh

call link optimization

In the update method of ReuseSliverMultiBoxAdaptorElement, it is judged whether it is a partial refresh. If it is not a partial refresh, performRebuild is still used; if it is a partial refresh, only the newly generated item is created.

[]()

core code

@override
void update(covariant ReuseSliverMultiBoxAdaptorWidget newWidget) {
  ...
  //是否进行局部刷新
  if(_isPartialRefresh(oldDelegate, newDelegate)) {
      ...
      int index = _childElements.lastKey() + 1;
      Widget newWidget = _buildItem(index);
      // do not create child when new widget is null
      if (newWidget == null) {
        return;
      }
      _currentBeforeChild = _childElements[index - 1].renderObject as RenderBox;
      _createChild(index, newWidget);
    } else {
       // need to rebuild
       performRebuild();
    }
}

Element & RenderObject reuse

Call link optimization

• Create: In the createChild method of ReuseSliverMultiBoxAdaptorElement, read the _cacheElements corresponding to the component type cached Element for reuse; if there is no reusable Element of the same type, create a new Element and RenderObject corresponding to it.
• Removal: Remove the removed RenderObject from the double-linked list in the removeChild method of ReuseSliverMultiBoxAdaptorElement, and do not deactive and detach the RenderObject of the Element, and update the _slot of the corresponding Element to null, so that it can be reused normally next time. Then cache the corresponding Element to the linked list of _cacheElements corresponding to the component type.

[]()

Note: It can be achieved without calling the deactive Element, but it cannot be done directly without detaching the RenderObject. You need to add a new method removeOnly in the object.dart file of the Flutter Framework layer, which is to only remove the RenderObject from the double-linked list. To detach.

core code

• create

//新增的方法，createChild会调用到这个方法
_createChild(int index, Widget newWidget){
  ...
  Type delegateChildRuntimeType = _getWidgetRuntimeType(newWidget);
  if(_cacheElements[delegateChildRuntimeType] != null
      && _cacheElements[delegateChildRuntimeType].isNotEmpty){
    child = _cacheElements[delegateChildRuntimeType].removeAt(0);
  }else {
    child = _childElements[index];
  }
  ...
  newChild = updateChild(child, newWidget, index);
  ...
}

• Remove

@override
void removeChild(RenderBox child) {
 ...
 removeChildRenderObject(child); // call removeOnly
 ...
 removeElement = _childElements.remove(index);
 _performCacheElement(removeElement);
 }

Load More

Call link optimization

At the time of createChild, it is judged whether it is to construct a footer for processing.

[]()

core code

@override
void createChild(int index, { @required RenderBox after }) {
    ...
    Widget newWidget;
    if(_isBuildFooter(index)){ // call footerBuilder & call onLoadMore
      newWidget = _buildFooter();
    }else{
      newWidget = _buildItem(index);
    }
    ...
    _createChild(index, newWidget);
    ...
}

Overall structure design

• Cohesion of the core optimization capabilities on the Element layer to provide the underlying capabilities;
• Use ReuseSliverMultiBoxAdaptorWidget as the base class to return the optimized Element by default;
• Unify the capabilities of loadMore and FooterBuilder to the upper layer exposed by ReuseSliverChildBuilderDelegate inherited from SliverChildBuilderDelegate;
• If you have your own individually customized streaming component Widget, you can directly change the inheritance relationship from RenderObjectWidget to ReuseSliverMultiBoxAdaptorWidget, such as custom single-list component (ReuseSliverList), waterfall flow component (ReuseWaterFall), etc.

Optimization results

Based on the previous series of in-depth optimization and switching the Flutter Engine to UC Hummer, the optimization variables of the streaming scene were individually controlled, and PerfDog was used to obtain fluency data, and fluency test comparisons were carried out:

It can be seen that the overall performance data has been optimized and improved. On average, combined with the test data before replacing the Engine, the frame rate has been increased by 2-3 frames, and the stall rate has dropped by 1.5%.

Summarize

How to use

The same way as the native SliverList is used, the Widget is replaced with the corresponding components that can be reused (ReuseSliverList/ReuseWaterFall/CustomSliverList). If the delegate needs footer and loadMore, use ReuseSliverChildBuilderDelegate; if you don't need to directly use the native SliverChildBuilderDelegate.

requires paging scene

return ReuseSliverList( // ReuseWaterFall or CustomSliverList
delegate: ReuseSliverChildBuilderDelegate(
  (BuildContext context, int index) {
    return getItemWidget(index);
  }, 
  //构建footer
  footerBuilder: (BuildContext context) {
    return DetailMiniFootWidget();
  },
  //添加loadMore监听
  addUnderFlowListener: loadMore,
  childCount: dataOfWidgetList.length
)
);

No paging required scene

return ReuseSliverList( // ReuseWaterFall or CustomSliverList
delegate: SliverChildBuilderDelegate(
  (BuildContext context, int index) {
    return getItemWidget(index);
  }, 
  childCount: dataOfWidgetList.length
)
);

be careful

When using item/footer components, do not add Key, otherwise it will be considered that only the same Key will be reused. Because the Element is reused, although the Widget that expresses the result of the component tree data will be updated every time, the State of the StatefulElement is generated when the Element is created and will be reused at the same time. It is consistent with the design of Flutter itself, so it is necessary In didUpdateWidget(covariant OldWidget), get the data cached in State from Widget again.

Reuse Element Lifecycle

Call back the status of each item, and the upper layer can do logic processing and resource release, for example, previously in didUpdateWidget (covariant OldWidget) the data cached in the State was retrieved from the Widget and can be placed in onDisappear or automatically played video stream, etc.;

/// 复用的生命周期
mixin ReuseSliverLifeCycle{

  // 前台可见的
  void onAppear() {}

  // 后台不可见的
  void onDisappear() {}
}

Reference

[1]: Google Flutter Team Xiao Yu: Flutter Performance Profiling and Theory: https://files.flutter-io.cn/events/gdd2018/Profiling_your_Flutter_Apps.pdf

[2]: Xianyu Yuncong: He doubled the fluency of the long list of Xianyu APP

[3]：Google Android RecyclerView.ViewHolder:RecyclerView.Adapter#onCreateViewHolder：https://developer.android.com/reference/androidx/recyclerview/widget/RecyclerView.Adapter#onCreateViewHolder(android.view.ViewGroup,%20int)

Follow the [Alibaba Mobile Technology] official public number , 3 mobile technology practices & dry goods for you to think about every week!