What new features are released in CameraX 1.1?

CameraX is a Jetpack support library designed to help you simplify camera application development. It provides a consistent and easy-to-use API interface for most Android devices and is backward compatible up to Android 5.0 (API level 21). In this article, we will introduce several features of CameraX 1.1, such as video capabilities.

If you prefer to see this through video, check it out here:

https://www.bilibili.com/video/BV1z3411Y7UJ/?aid=423331610&cid=486916612&page=1
△ What new features are released in CameraX 1.1?

CameraX Overview

CameraX is a support library designed to simplify writing camera applications. It provides high-level APIs that allow developers to focus on interacting with the user rather than the internal implementation of the camera. We've been exploring and fixing the complex compatibility issues behind it, making each new version run stably on more devices.

When to use CameraX or Camera2, it depends on whether you expect faster development speed or want a higher degree of customization.

• CameraX can easily realize the shooting function of ordinary photos and videos, while Camera2 can perform special control of the shooting process, such as multiple exposures or full manual capture;
• CameraX is designed to eliminate the differences between different devices and has been tested on different devices, while Camera2 requires an application to manage the differences between different devices and test its behavior;
• CameraX speeds up code development, allowing you to focus more on the user interface and experience flow, while Camera2 is used for deeper development to create custom camera-based features;
• CameraX releases new versions frequently, while Camera2 is updated with the Android version;
• CameraX can be developed even if you are unfamiliar with cameras, while Camera2 requires you to have a deeper understanding of camera expertise.

CameraX is built on top of the main usage scenarios, such as previewing the camera in real time, retrieving buffers for analysis, and taking photos, with video capture added in CameraX version 1.1. Let's look at a simple CameraX example:

fun bindPreview(cameraProvider : ProcessCameraProvider) {
    // 使用 CameraX 创建 Preview 用例
    var preview : Preview = Preview.Builder().build()

    // 创建 cameraSelector，它会在设备上搜索所需的相机
    var cameraSelector : CameraSelector = CameraSelector.Builder()
        // 在本例中，我们选择搜索后置相机
        .requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

    // 从 CameraX 的 CameraView 包中获取 previewView 的句柄
    // 利用此方法可以轻松的将相机内容添加到视图上
    preview.setSurfaceProvider(previewView.getSurfaceProvider())

    // 将 preview 与其生命周期绑定
    var camera = cameraProvider.bindToLifecycle(this as LifecycleOwner,
                                cameraSelector, preview)
}

△ CameraX code example

CameraX is a lifecycle-aware component, which means it automatically handles the application's lifecycle events to start, stop, pause, and resume. A live preview now appears on the screen when the app starts.

We have released the 1.0 stable version in May 2021 and are currently working on a 1.1 Alpha version and will soon enter the Beta phase. And, as always, we're rolling out compatibility fixes for new devices, such as 1.0.1 and 1.0.2.

In CameraX version 1.1 we added features that have been highly requested by developers, specifically, in this article we will focus on:

• video shooting
• YUV to RGB conversion
• Beta Extensions API
• Some other features to know

video shot

In CameraX 1.1 version, we added the video shooting function, the video shooting API (still in Alpha stage, the details may change, but the overall structure will basically remain the same) provides basic functions such as recording to file, can automatically adapt to each Quality Setting API for each device, and Lifecycle Management API. Next, let's first understand how to set the video shooting function. The code example is as follows:

// 创建 Recorder
val recorder = Recorder.Builder()
                       // 我们可以在此处使用 setQualitySelector 设置视频质量
                       .setQualitySelector(...)
                       .build()

// 使用新创建的 Recorder 创建 VideoCapture
val videoCapture = VideoCapture.withOutput(recorder)

// 将其与生命周期绑定
cameraProvider.bindToLifecycle(
    this, CameraSelector.DEFAULT_BACK_CAMERA, preview, videoCapture)

// 设定 VideoRecordEvent 监听器
val videoRecordEventListener = Consumer<VideoRecordEvent>{
    when (it) {
        is VideoRecordEvent.Start -> {}
        is VideoRecordEvent.Finalize -> {}
        is VideoRecordEvent.Status -> {
            // status 事件将会在录制时持续更新
            val stats: RecordingStats = it.recordingStats
            // RecordingStats 中包含录制文件的尺寸和时长
        }
        is VideoRecordEvent.Pause -> {}
        is VideoRecordEvent.Resume -> {}

// 指定输出
val mediaStoreOutput = MediaStoreOutputOptions.Builder(this.contentResolver,
        MediaStore.Video.Media.EXTERNAL_CONTENT_URI)
    .setContentValues(contentValues)
    .build()

// 准备录制
val activeRecording = videoCapture.output.prepareRecording(this, mediaStoreOutput)
    // 关联事件监听器
    .withEventListener(ContextCompat.getMainExecutor(this), videoRecordEventListener)
    // 启用音频 (前提是此应用已获得音频权限)
    .withAudioEnabled()
    // 开始录制
    .start()

△ Video shooting example

videoCapture will be ready when the application starts, the application can use videoRecordEventListener response start, end, pause and resume other shooting incidents, which Status event will provide, including file size and duration RecordingStats . Video capture can be output to File, FileDescriptor or MediaStore , in this case we choose MediaStore . If you choose to enable audio, this app needs to have audio permission. Calling start() start recording gives us the activeRecording handle, which can be used to pause, resume or stop the recording. You can try out these APIs in version 1.1.

YUV to RGB conversion

Another highly requested feature is YUV to RGB conversion, let's take a look at this feature.

△ YUV format (left) converted to RGB format (right)

Cameras typically generate data in YUV420 format, which includes Luminance (Y), Chroma (Chroma, U, V) and some padding bytes to align lines with a valid memory stride. But images in this format can be cumbersome to process, and now CameraX can convert the output of ImageAnalysis to the more familiar RGBA for easy processing. Next we look at an example:

val imageAnalysis = ImageAnalysis.Builder()
        .setTargetResolution(Size(1280, 720))
        .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
        .build()

△ Get RGB output from ImageAnalysis

In the sample code, we create an ImageAnalysis instance, specify the desired resolution and backpressure strategy for the image buffer, and call the new setOutputImageFormat method to request output in RGBA 8888 format. Now, ImageAnalysis outputs frames as RGBA 8888 data instead of YUV format.

The YUV to RGB conversion in CameraX is based on libyuv. Additionally, in CameraX version 1.1 the data itself can be converted to the target resolution. Converting data with an image size of 640x480 to 1080p takes about 5-10ms on a midrange device, depending on the device. In addition, the APK will be slightly increased by about 50KB.

fix single pixel drift

YUV conversion also fixes a single-pixel drift issue on some devices. On these devices, the YUV output is barrel-shifted by one pixel, causing the rightmost column of data to appear at the left edge of the image. On devices where this is known to happen, doing YUV to RGB conversion and outputting YUV or RGB will be fixed, and CameraX will continue to make fixes for more devices that need it.

△ Fix single pixel drift

To learn more, please see our previous tweet " be converted into YUV to RGB ImageAnalysis CameraX ."

CameraX Extensions API

camera effects

The CameraX Extensions API in CameraX 1.1 can fully utilize the powerful functions of the device.

CameraX Extensions include some of the most common built-in camera effects:

• BOKEH (Bokeh Bokeh) : Make foreground subjects sharper when taking photos in portrait mode.
• HDR (High Dynamic Range) : Use different Auto Exposure (AE) configurations when taking pictures for best results.
• NIGHT (Night) : Captures the best still images in low light conditions (usually at night).
• FACE RETOUCH (Face Photo Restoration) : Retouch facial skin tones, contours, etc. when shooting still images.
• AUTO : Automatically adjust the final image to the surrounding scenery.

Let's see how to use the CameraX Extensions API:

// 获取后置相机列表
val cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA

// 检查所有的后置相机中是否有支持焦外虚化
if (extensionsManager.isExtensionAvailable(
    cameraProvider,
    cameraSelector,
    ExtensionMode.BOKEH
)) {
    // 创建扩展 cameraSelector，我们提供了相机并指定焦外虚化模式
    // 它将开始在后台搜索支持焦外虚化的后置相机
    val bokehCameraSelector = extensionsManager.getExtensionCameraSelector(
        cameraProvider,
        cameraSelector,
        ExtensionMode.BOKEH
    )

    // 创建 imageCapture 和 preview
    val imageCapture = ImageCapture.Builder().builder()
    val preview = Preview.Builder().build()

    // 使用 bokehCameraSelector 将它们绑定到生命周期
    cameraProvider.bindToLifecycle(lifecycleOwner,
        bokehCameraSelector,
        imageCapture,
        preview
    )
}

△ Capture and preview images with BOKEH effect

In the example above, imageCapture will have a bokeh effect, and if the device supports it, the preview will also preview the bokeh effect.

For more details, see our previous post " Applying Effects to Photos Using the CameraX Extensions API ".

Exposure Compensation

CarmeraX 1.1 also added the Exposure Compensation API, which helps users better capture overexposed or underexposed areas.

As shown in the picture, we are in a scene that is bright outside the window and dark indoors. At this time, the exposure compensation can be adjusted to better capture the bright outdoor or dim indoor scene. Let's look at an example:

// 创建变量来跟踪 exposureIndex 值
var exposureIndex = 0
// 使用 cameraSelector 将 imageCapture 和 preview 绑定到生命周期
val camera = cameraProvider.bindToLifecycle(
    lifecycleOwner,
    getCameraSelector(),
    preview,
    imageCapture
)
 
// 为视图中的按钮添加点击事件
evButton.setOnclickListener {
 
    // 检查有效的范围以防止可能的异常
    val range = camera.cameraInfo.exposureState.exposureCompensationRange
 
    if (range.contains(exposureIndex + 1)) {
        // 调用 camera.cameraControl 的 setExposureCompenstation() 方法来设置曝光补偿
        camera.cameraControl.setExposureCompenstation(++exposureIndex)
        // 使用 exposureCompensationStep 来实现从 index 到 EV 到转换
        val ev = camera.cameraInfo.exposureState.exposureCompensationStep.toFloat() * exposureIndex
        Log.i("CameraXLog", "EV: $ev")
    }
}

△ Adjust the exposure by button

where exposureIndex is a device-independent number that will increment or decrement the exposure value in the smallest step the hardware allows, so it can work in a similar way on different devices. If you want to show the EV value to the user, you can get exposureCompensationStep to do the conversion.

For background on how the Exposure Compensation API in CameraX is used and how to call it, please refer to our previous post " CameraX Exposure Compensation API Getting Started Guide ".

smooth scaling

In CameraX 1.1, we also added smooth zooming. There are devices with multiple lenses including wide and telephoto, CameraX can detect if these devices support the SMOOTH_ZOOM frame, and when using CameraX's zoom controls on supported devices, it will automatically use all cameras to achieve larger Zoom range. If you're already using this zoom control, your app should have access to all cameras on these devices when you compile with version 1.1.

More from CameraX 1.1

More features we've added in 1.1 follow.

The CameraState API can now provide more information about the camera state, such as whether another app is using the camera or is in Do Not Disturb mode, allowing apps to design better UI and user experience flows around different camera times. Image Analysis can now provide images in excess of 1080p. The Logging API enables more verbose debug logging and improved error reporting. The Coordinate Transformation API can correlate coordinates across use cases, and if you locate a point of interest in the imageAnalysis buffer, you can easily find it in the image capture output or preview. You can use the CameraFilter API to specify detailed rules to select the appropriate camera. If the application only needs the front or rear camera, the AvailableCamerasLimiter can be used to speed up the startup time. More details on camera capabilities are available in CameraControllerInfo.

Device Compatibility

CameraX continuously pays attention to device compatibility so that the app works well on a wide range of devices. We fixed a number of issues such as image stretching, incorrect scaling, upside-down images, and unexpectedly outputting green graphics when closing the camera. Such fixes are added in every release or patch release of CameraX, the latest stable version being 1.0.2.

You can see the detailed changes in each release release record issues in the 161f0dc62e858f issue tracker .

More info

• CameraX Overview
• CameraX Getting Started Codelab
• CameraX Github sample code
• Android CameraX discussion group

I hope this brief introduction to CameraX version 1.1 was helpful, and I'm really looking forward to seeing what you can build with CameraX!

You are welcome click here to submit feedback to us, or share your favorite content and problems found. Your feedback is very important to us, thank you for your support!