Flutter image library is heavily open source!

Author: Shinjuku

background

Last year, PowerImage, a new generation of image library of Xianyu technical team, has been fully and stably applied to Xianyu after a series of grayscale, problem fixes, and code tuning.

Compared with the previous generation IFImage, PowerImage has undergone further evolution, adapted to more business scenarios and the latest flutter features, and solved a series of pain points: It will cause some low-frequency pictures to occupy the cache instead; for example, we cannot display pictures on the simulator; for example, in the album, we need to build a picture channel outside the picture library.

For details, you can read: "Flutter Image Library High Fire New Debut "

PowerImage related links:

GitHub: (✅star🌟)

https://github.com/alibaba/power_image

Flutter pub: (✅like👍)

https://pub.dev/packages/power_image

Introduction

PowerImage is a flutter image library that makes full use of native native image library capabilities and is highly scalable. We cleverly combined external textures and ffi solutions to be closer to the native design and solved a series of business pain points.

Ability characteristics

• Supports the ability to load ui.Image. In the solution based on external texture, the user cannot get the real ui.Image to use, which makes the image library powerless in this special usage scenario.
• Support image preloading capability. Just like native precacheImage. This is very useful in some scenarios that require high image display speed.
• Added texture cache to connect with native image library cache! Unified image cache to avoid memory problems caused by mixing native images.
• Emulators are supported. Before flutter-1.23.0-18.1.pre, the simulator cannot display Texture Widget.
• Improve the custom image type channel. Solve the demand for business custom image acquisition.
• Perfect exception capture and collection.
• Support animation. (PR from Tao Te)

Flutter native solution

Before introducing the new scheme, let’s briefly recall the native image scheme of flutter.

The native Image Widget first obtains the ImageStream through the ImageProvider, and displays various states by monitoring its state. frameBuilder 、 loadingBuilder ，最终在图片加载成功后，会rebuild出RawImage ， RawImage RenderImage to draw, the core of the whole drawing is ImageInfo in ui.Image .

• Image: Responsible for the display of various states of image loading, such as loading, failure, and successful loading to display images, etc. \
• ImageProvider: Responsible for the acquisition of ImageStream, such as the built-in NetworkImage, AssetImage, etc.
• ImageStream: The object loaded by the image resource.

After sorting out the flutter native image solution, we found that there is a chance to connect the flutter image and native in a native way at some point?

New generation solution

We cleverly combined the FFi solution with the external texture solution to solve a series of business pain points.

FFI

As mentioned at the beginning, there are some things that the Texture solution cannot do, which requires other solutions to complement each other. The core of which is ui.Image. We pass the native memory address, length and other information to the flutter side for generating ui.Image.

First, the native side first obtains the necessary parameters (take iOS as an example):

dart side after getting it

We can get the native memory through ffi to generate ui.Image. There is a problem here, although the native memory can be directly obtained through ffi, but since decodeImageFromPixels will have a memory copy, when copying the decoded image data, the memory peak will be more serious.

There are two optimization directions here:

1. The image data before decoding is sent to flutter, which is decoded by the decoder provided by flutter, thereby reducing the memory copy peak;
2. Discuss with flutter official and try to reduce this memory copy internally.

This method of FFI is suitable for light use and special scenarios. Supporting this method can solve the problem that ui.Image cannot be obtained, and can also display pictures on the simulator (flutter <= 1.23.0-18.1.pre), and pictures The cache will be completely managed by ImageCache.

Texture

It is a bit difficult to combine the Texture scheme with the native one, which involves no ui.Image but only textureId. There are several issues to address here:

1. Question 1: Image Widget needs ui.Image to build RawImage to draw, which is also mentioned in the introduction of Flutter native solution earlier in this article;
2. Question 2: ImageCache relies on the width and height of ui.Image in ImageInfo for cache size calculation and pre-cache verification;
3. Question 3: The texture life cycle management on the native side.

There are solutions for each:

Problem 1: Solve by customizing Image, revealing imageBuilder to allow external custom image widgets

Question 2: Customize ui.Image for Texture, as follows:

In this case, TextureImage is actually a shell, only used to calculate the cache size. In fact, ImageCache calculates the size, there is no need to directly touch ui.Image, you can directly find ImageInfo to get it, so there is no such problem. \

Question 3: About the native side perception of flutter image release timing.

The modified ImageCache is released as follows (part of the code):

Overall structure

We have combined two solutions very elegantly:

We abstracted the PowerImageProvider. For external (ffi) and texture, we can produce our own ImageInfo respectively. It will provide unified loading and releasing capabilities by calling PowerImageLoader.

The ImageExt with the blue solid line is the custom Image Widget, which reveals the imageBuilder for the texture mode.

The blue dotted line ImageCacheExt is the extension of ImageCache, which is only required in flutter < 2.2.0 version, it will provide a callback for ImageCache release timing.

This time, we have also designed super expansion capabilities. In addition to supporting network maps, local maps, flutter resources, and native resources, we provide channels for custom image types. Flutter can pass any custom parameter combination to native, as long as native registers the corresponding type of loader, such as "album" In the scene, the user can customize the imageType as album, and the native uses its own logic to load the image. With this custom channel, even image filters can be refreshed using PowerImage.

In addition to the extension of the image type, the rendering type can also be customized. For example, as mentioned in ffi above, in order to reduce the peak problem caused by memory copying, the user can decode on the flutter side. Of course, this requires the native image library to provide data before decoding.

data

FFI vs Texture

Model: iPhone 11 Pro; Image: 300 network graphs; Behavior: Manual scroll to bottom and then top in listView; \
native Cache: 20 maxMemoryCount; flutter Cache: 30MB

flutter version 2.5.3; in release mode

There are two phenomena here:

FFI: 186MB fluctuating
Texture: 194MB fluctuation

In version 2.5.3, the Texture scheme and FFI have little difference in the memory water level, and the memory fluctuation is opposite to the conclusion of flutter 1.22.

The checkerboard diagram in the figure is shown after opening checkerboardRasterCacheImages. It can be seen that the ffi scheme will cache the entire cell, while the texture scheme only caches the text in the cell. RasterCache will give ffi a certain advantage in fluency.

Scrolling Fluency Analysis

Device: Android OnePlus 8t, CPU and GPU frequency locked. \
case: GridView has 4 pictures per row, 300 pictures, from top to bottom, then from bottom to top, the sliding range is from 500, 1000, 1500, 2000, 2500, 5 rounds of sliding. Repeat 20 times.

Method: for i in {1..20}; do flutter drive --target=test_driver/app.dart --profile; done Run data, get TimeLine data and analyze.

in conclusion:

• UI thread time-consuming texture method is the best, PowerImage is slightly better than IFImage, and FFI method fluctuates greatly.
• Raster thread time-consuming PowerImage is better than IFImage. The original method of Origin is better because the image is resized, and the original image is loaded in other methods.

leaner code

The code on the dart side has been greatly reduced. This is due to the fact that the technical solution fits the native design of flutter, and we share a lot of code with native images.

The FFI scheme complements the insufficiency of external textures and follows the design specifications of native Image, which not only allows us to enjoy the unified management brought by ImageCache, but also brings more streamlined code.

single test

In order to ensure the stability of the core code, we have a relatively complete single test, and the line coverage rate is close to 95%.

About open source

We look forward to making PowerImage more complete and powerful through the power of the community, and hope that PowerImage can bring benefits to everyone in engineering research and development.

Issues

Regarding issues, we hope that you will actively communicate when you encounter problems and demands using PowerImage, and provide detailed information as much as possible when raising issues to reduce communication costs. Please make sure to read the readme before filing an issue.

For bug issues, we have customized a template (Bug report), which can easily fill in some necessary information. For other types, you can choose Open a blank issue.

We will spend part of the week dealing with issues in a unified manner, and look forward to your discussions and PRs.

PR

In order to maintain the stability of the core functions of PowerImage, we have a complete single test with a line coverage rate of 95% (power_image library).

When submitting PR, please ensure that the submitted code is covered by the unit test, and the involved unit test code should be submitted at the same time.

Thanks to the Actions capability of Github, when we push the code in the main branch and perform PR operations on the main branch, the flutter test task will be triggered, and only the single test can be merged.

future

Open source is the beginning, not the end, of PowerImage. There are many more interesting and rich things that PowerImage can do. For example, how to implement the loadingBuilder described in the first issue? For example, how does the ffi scheme support animation? Another example is Kotlin and Swift...

PowerImage will continue to evolve in the future. When the current texture scheme and ffi scheme coexist, along with the iteration of flutter itself, we will be more inclined to develop towards ffi. As in the comparison above, the ffi scheme can naturally enjoy the benefits of the raster cache. the advantage of fluency.

PowerImage will continue to follow in the footsteps of flutter, in order to always adhere to the original design concept and make continuous progress. We hope that more students will join in and grow together.

PowerImage related links:

GitHub: (✅star🌟)

https://github.com/alibaba/power_image

Flutter pub: (✅like👍)

https://pub.dev/packages/power_image

Pay attention to [Alibaba Mobile Technology], Ali's cutting-edge mobile dry goods & practice will give you thoughts!