Flutter's image library is new

Author: Wang Zhenhui (Shinjuku)

Last year, Xianyu Image Library achieved good results in large-scale applications, but it also encountered some problems and demands, and needs further evolution to adapt to more business scenarios and the latest flutter features. For example, because the native ImageCache is completely abandoned, some low-frequency pictures will occupy the cache when mixed with native pictures; for example, we cannot display pictures on the simulator; for example, we need to display pictures in the album. Build a picture channel outside the library.

This time, we cleverly combined external textures and FFi solutions to get closer to the native design and solve a series of business pain points. That's right, there's a new addition to the Power series, and we've named our new image gallery "PowerImage"!

We will add the following core capabilities:

• Supports the ability to load ui.Image. In the solution based on external texture last year, the user could not get the real ui.Image to use, which caused the image library to be powerless in this special usage scenario;
• Support image preloading capability. Just like native precacheImage. This is very useful in some scenarios that require high image display speed;
• Added texture cache to connect with native image library cache! Unified image cache to avoid memory problems caused by mixing native images;
• Emulators are supported. Before flutter-1.23.0-18.1.pre, the simulator could not display Texture Widget;
• Improve the custom image type channel. Solve business custom image acquisition demands;
• Perfect exception capture and collection;
• Support animation.

Flutter native solution

Before we start our new solution, let's briefly recall the flutter native image solution.

The native Image Widget first obtains the ImageStream through the ImageProvider, and displays various states by monitoring its state. For example, frameBuilder and loadingBuilder will rebuild RawImage after the image is loaded successfully, and RawImage will be drawn by RenderImage. The core of the whole drawing is ui.Image in ImageInfo.

• Image: Responsible for the display of various states of image loading, such as loading, failure, and successful loading to display images, etc.;
• ImageProvider: Responsible for the acquisition of ImageStream, such as the built-in NetworkImage, AssetImage, etc.;
• ImageStream: The object loaded by the image resource.

After sorting out the flutter native image solution, we found that there is a chance to connect the flutter image and native in a native way at some point?

new program

We cleverly combined the FFi solution with the external texture solution to solve a series of business pain points.

FFI

As mentioned at the beginning, there are some things that the Texture solution cannot do, which requires other solutions to complement each other. The core of which is ui.Image. We pass the native memory address, length and other information to the flutter side for generating ui.Image.

First, the native side first obtains the necessary parameters (take iOS as an example):

    _rowBytes = CGImageGetBytesPerRow(cgImage);
    
    CGDataProviderRef dataProvider = CGImageGetDataProvider(cgImage);
    CFDataRef rawDataRef = CGDataProviderCopyData(dataProvider);
    _handle = (long)CFDataGetBytePtr(rawDataRef);
    
    NSData *data = CFBridgingRelease(rawDataRef);
    self.data = data;
    _length = data.length;

After getting it on the dart side,

@override
  FutureOr<ImageInfo> createImageInfo(Map map) {
    Completer<ImageInfo> completer = Completer<ImageInfo>();
    int handle = map['handle'];
    int length = map['length'];
    int width = map['width'];
    int height = map['height'];
    int rowBytes = map['rowBytes'];
    ui.PixelFormat pixelFormat =
        ui.PixelFormat.values[map['flutterPixelFormat'] ?? 0];
    Pointer<Uint8> pointer = Pointer<Uint8>.fromAddress(handle);
    Uint8List pixels = pointer.asTypedList(length);
    ui.decodeImageFromPixels(pixels, width, height, pixelFormat,
        (ui.Image image) {
      ImageInfo imageInfo = ImageInfo(image: image);
      completer.complete(imageInfo);
      //释放 native 内存
      PowerImageLoader.instance.releaseImageRequest(options);
    }, rowBytes: rowBytes);
    return completer.future;
  }

We can get the native memory through ffi to generate ui.Image. There is a problem here. Although the native memory can be directly obtained through ffi, since decodeImageFromPixels will have memory copy, when copying the decoded image data, the memory peak will be more serious.

There are two optimization directions here:

• The image data before decoding is sent to flutter, which is decoded by the decoder provided by flutter, thereby reducing the memory copy peak;
• Discuss with flutter official and try to reduce this memory copy internally.

This method of FFI is suitable for light use and special scenarios. Supporting this method can solve the problem that ui.Image cannot be obtained, and can also display pictures on the simulator (flutter <= 1.23.0-18.1.pre), and pictures The cache will be completely managed by ImageCache.

Texture

It is difficult to combine the Texture scheme with the native one. It involves no ui.Image but only textureId. There are several issues to address here:

Question 1: Image Widget needs ui.Image to build RawImage to draw, which is also mentioned in the introduction of Flutter native solution earlier in this article.

Question 2: ImageCache relies on the width and height of ui.Image in ImageInfo for cache size calculation and pre-cache verification.

Question 3: Texture life cycle management on the native side

There are solutions:

Problem 1: Solve by customizing Image, revealing imageBuilder to allow external custom image widgets

Question 2: Customize ui.image for Texture, as follows:

import 'dart:typed_data';
import 'dart:ui' as ui show Image;
import 'dart:ui';

class TextureImage implements ui.Image {
  int _width;
  int _height;
  int textureId;
  TextureImage(this.textureId, int width, int height)
      : _width = width,
        _height = height;

  @override
  void dispose() {
    // TODO: implement dispose
  }

  @override
  int get height => _height;

  @override
  Future<ByteData> toByteData(
      {ImageByteFormat format = ImageByteFormat.rawRgba}) {
    // TODO: implement toByteData
    throw UnimplementedError();
  }

  @override
  int get width => _width;
}

In this case, TextureImage is actually a shell, only used to calculate the cache size. In fact, ImageCache calculates the size, there is no need to directly touch ui.Image, you can directly find ImageInfo to get it, so there is no such problem. This question can be found in @haoan's ISSUE[1] and PR[2].

Question 3: About the release timing of the native side perception of flutter image

• After flutter 2.2.0, ImageCache provides a release opportunity, which can be reused directly without modification;
• < Version 2.2.0, ImageCache needs to be modified to obtain the timing when the cache is discarded. When the cache is discarded, notify the native to release it.

The modified ImageCache is released as follows (part of the code):

typedef void HasRemovedCallback(dynamic key, dynamic value);

class RemoveAwareMap<K, V> implements Map<K, V> {
  HasRemovedCallback hasRemovedCallback;
  ...
}
//------
  final RemoveAwareMap<Object, _PendingImage> _pendingImages = RemoveAwareMap<Object, _PendingImage>();
//------
void hasImageRemovedCallback(dynamic key, dynamic value) {
    if (key is ImageProviderExt) {
      waitingToBeCheckedKeys.add(key);
    }
    if (isScheduledImageStatusCheck) return;
    isScheduledImageStatusCheck = true;
    //We should do check in MicroTask to avoid if image is remove and add right away
    scheduleMicrotask(() {
      waitingToBeCheckedKeys.forEach((key) {
        if (!_pendingImages.containsKey(key) &&
            !_cache.containsKey(key) &&
            !_liveImages.containsKey(key)) {
          if (key is ImageProviderExt) {
            key.dispose();
          }
        }
      });
      waitingToBeCheckedKeys.clear();
      isScheduledImageStatusCheck = false;
    });
  }

Overall structure

We have combined two solutions very elegantly:

We abstracted the PowerImageProvider. For external (ffi) and texture, we can produce our own ImageInfo respectively. It will provide unified loading and releasing capabilities by calling PowerImageLoader.

The ImageExt with the blue solid line is the custom Image Widget, which reveals the imageBuilder for the texture method.

The blue dotted line ImageCacheExt is the extension of ImageCache, which is only required in flutter < 2.2.0 version, it will provide a callback for ImageCache release timing.

This time, we have also designed super expansion capabilities. In addition to supporting network graphs, local maps, flutter resources, and native resources, we provide channels for custom image types. Flutter can pass any custom parameter combination to native, as long as native registers the corresponding type of loader, such as "album" In the scene, the user can customize the imageType as album, and the native uses its own logic to load the image. With this custom channel, even image filters can be refreshed using PowerImage.

In addition to the extension of the image type, the rendering type can also be customized. For example, as mentioned in ffi above, in order to reduce the peak problem caused by memory copying, the user can decode on the flutter side. Of course, this requires the native image library to provide data before decoding.

Data comparison

FFI vs Texture

Model: iPhone 11 Pro, Image: 300 network graphs, Behavior: Manually scroll to the bottom and then to the top in the listView, native Cache: 100MB, flutter Cache: 100MB

There are two phenomena here:

1. Texture: 395MB fluctuates, memory is smoother
2. FFI: 480MB fluctuates, memory glitches

The Texture scheme outperforms FFI in terms of memory, in terms of memory water level and glitches:

• Memory water level: Since the cache on the flutter side of the Texture solution is a space-occupying empty shell and does not actually occupy memory, only one copy exists in the memory cache of the native image library, so the memory cache on the flutter side is actually 100MB less than the ffi solution;
• Glitch: Since the ffi scheme cannot avoid memory copying on the flutter side, there will be a process of first copying and then releasing, so there will be glitches.

in conclusion:

1. Texture is suitable for daily scenes, preferred;
2. FFI is more suitable for

a. In flutter <= 1.23.0-18.1.pre version, display pictures on the simulator

b. Get ui.Image image data

c. Decoding on the flutter side, the impact of data copy before decoding is small. (For example, the external decoding library of the group Hummer)

Scrolling Fluency Analysis

Device: Android OnePlus 8t, CPU and GPU frequency locked.

case: GridView has 4 pictures per row, 300 pictures, from top to bottom, then from bottom to top, the sliding range is from 500, 1000, 1500, 2000, 2500, 5 rounds of sliding. Repeat 20 times.

Method: for i in {1..20}; do flutter drive --target=test_driver/app.dart --profile; done Run data, get TimeLine data and analyze.

in conclusion:

• UI thread time-consuming texture method is the best, PowerImage is slightly better than IFImage, and FFI method fluctuates greatly.
• Raster thread time-consuming PowerImage is better than IFImage. The original method of Origin is better because the image is resized, and the original image is loaded in other methods.

leaner code

The code on the dart side has been greatly reduced. This is due to the fact that the technical solution fits the native design of flutter, and we share a lot of code with native images.

The FFI scheme complements the insufficiency of external textures and follows the design specifications of native Image, which not only allows us to enjoy the unified management brought by ImageCache, but also brings more streamlined code.

future

I believe many people have noticed that the animation part is missing from the above. The current animation part is under development. In the internal Pre Release version, the one that is returned when loading is actually OneFrameImageStreamCompleter. For the animation, we will replace it with MultiFrameImageStreamCompleter. How to do it later is just some strategic issues, not difficult. By the way, there is another solution: you can decode and render the data before the decoding of the moving image to the flutter side, but the supported formats are not as rich as the native ones.

We hope to contribute PowerImage to the community. In order to achieve this goal, we provide detailed design documents, access documents, and performance reports. In addition, we are also improving unit testing. Unit testing will be performed after code submission or CR. .

References

[1] ISSUE: https://github.com/flutter/flutter/issues/86402\
[2] PR: https://github.com/flutter/flutter/pull/86555

Follow [Alibaba Mobile Technology] WeChat public account, 3 mobile technology practices & dry goods per week for you to think about!