Perform YUV to RGB conversion for CameraX ImageAnalysis

CameraX is a Jetpack support library designed to help developers simplify camera application development. It supports a variety of scenarios such as ImageCapture, Preview and ImageAnalysis that can be seamlessly combined with ML Kit or TensorFlow Lite This provides the possibility for the development of applications such as text recognition and image tagging, and even supports the use of the TensorFlow Lite model trained by the developer for object recognition and detection. However, the work of image format conversion between CameraX and these libraries is relatively time-consuming and laborious. In this article, we will introduce the new features recently brought to CameraX ImageAnalysis, which supports the conversion from YUV to RGB, we will introduce some background knowledge, why this feature is introduced, and will introduce how to use it with a small amount of sample code.

background

CameraX uses YUV420_888 to generate images. This format has three channels of 8-bit Luma(Y), Chroma(U, V) and Paddings(P). YUV is a universal and flexible format, which supports OEM variants on different devices, which covers many ImageAnalysis usage scenarios. However, many applications still rely on the RGB format. In our developer community, YUV to RGB conversion is one of the most popular features, because the RGB format is popular and easy to use, and sometimes needs to be used in TensorFlow Lite models. Let's take a look at the YUV and RGB formats first.

YUV_420_888 format

The YUV format can also be called "YCbCr", which includes planar (such as I420), semi-planar (such as NV21/NV12) and packed (such as UYVY) formats. YUV_420_888 is a general YCbCr format, it can represent any 4:2:0 chroma sub-sampling plane or half plane buffer (but not completely interlaced), each color sample has 8 bits. And it can be guaranteed that the Y plane will not be interleaved with the U/V plane (and the pixel step is always 1), and the U/V plane always has the same row step and pixel step.

RGBA_8888 format

RGBA_8888 is a standard RGB format with red, green, blue and alpha channels, each channel has 8 bits. The main conversion object is the RGB color space. RGB is relatively simple because of less variation in color difference.

API implementation

We evaluated three methods to convert YUV to RGB:

1. Use Java/Kotlin
2. Render script using Renderscript
3. Native solution (using C/C++ and NDK)

Using Java/Kotlin to process images requires a long time of calculation and faces the pressure of garbage collection. The Renderscript is a candidate for computationally intensive tasks (such as converting from YUV to RGB format). However, starting from Android 12, this method has been abandoned .

Considering the future scalability and compatibility, we decided to use the native solution (libyuv + NDK). Libyuv is an open source project, which contains the functions of zooming, converting and rotating YUV. Taking all factors together, from a macro point of view, the pipeline of CameraX color conversion is as follows:

For backward compatibility, we still use ImageProxy as output. ImageProxy is a wrapper class of media.image, which is an image buffer provided in the Android framework. The Java/Kotlin layer can Surface through dequeueInputImage() ), and then use ImageReader and ImageWriter ImageWriter to write the converted data into it. Since ImageWriter was added in API 23, we use ANativeWindow and its buffer to generate output images in RGBA format to support more API levels.

For input data, we support different variants of the YUV_420_888 format (I420, NV12, NV21, etc.) inside CameraX. For output data, we now support the RGBA format, but will expand to more other RGB formats in the future.

Since we use libyuv as the new dependent library, our library size has increased by approximately 50 KB .

API uses

Starting from the CameraX 1.1.0-alpha08 version, the application can select the image output format of YUV_420_888 or RGBA_8888 by using setOutputImageFormat in the ImageAnalysis configuration.

Once RGBA_8888 is selected, the output image format will be PixelFormat.RGBA_8888 , which has only one image plane with padding (R, G, B, A pixels one by one). In principle, the image buffer format supported by the Android framework is a subset of PixelFormat and ImageFormat.

In contrast, if YUV_420_888 is selected, the output image format will be ImageFormat.YUV_420_888 , which has 3 independent image planes (Y, U, V).

Performance

We did some performance tests and compared them with the results Renderscript Generally speaking, in different resolutions and Android system versions, the pipeline using libyuv is better than the implementation using Renderscript.

summary

We support YUV to RGB conversion in CameraX ImageAnalysis pipeline. Users can now simply select an output format (YUV_420_888 or RGBA_8888) for an ImageAnalysis use case and use it in other libraries. And this is just the beginning. We also plan to add more image processing functions to the CameraX ImageAnalysis pipeline and extend it to other use cases (such as ImageCapture or Preview, etc.). If you have any functional requirements, please contact us.

The sample code of YUV to RGB conversion can be viewed in GitHub . For more information about CameraX, please refer to official document . To learn about the latest developments of CameraX, you can join CameraX discussion forum . In addition, your feedback is very valuable to us, please feel free to leave a message in the CameraX discussion area or give us feedback Issue Tracker

Related references

• CameraX Release Notes
• Start using CameraX
• CameraX Github Use Case
• YUV format Wiki

Welcome to click here to submit feedback to us, or share your favorite content or problems found. Your feedback is very important to us, thank you for your support!