OpenHarmony Camera source code analysis

1. Introduction

At present, open source is playing an increasingly important role in technological progress and industrial development. OpenAtom OpenHarmony ("OpenHarmony" for short) gives developers the seeds of innovation, and also opens up a field for the development of the digital industry. Shenzhen Kaihong is a firm practitioner of open source, focusing on the research and development and continuous innovation of intelligent IoT operating system (KaihongOS) technology based on OpenHarmony.
OpenHarmony Camera is an important module in the multimedia subsystem. Camera provides functions such as preview, photo and video of OpenHarmony camera. As an OS system development engineer of Shenzhen Kaihong, I have been committed to the research and development of the OpenHarmony framework layer for a long time, and have accumulated some experience in the photography, preview and video recording of the OpenHarmony camera module. I will focus on these three core functions. Camera source code for detailed analysis.

2. OpenHarmony camera subsystem

(1) System introduction <br>The camera component supports the development of camera business. Developers can access, operate and develop new functions of camera hardware through the open interface. The most common operations are: preview, photo and video.

Architecture Diagram <br>The camera frame mainly includes session management, device input and data output. The device input and data output configuration is completed in the acquisition session, and the session management module manages the camera device input and data output. When the application layer calls the camera function, it first needs to create an acquisition session, and during the process of configuring the session, the created device input and data output will be added to the acquisition session.
Several important concepts in the camera frame
Session management : The management of the life cycle, parameter configuration, input and output of camera acquisition.
Device input : The main input device is the camera. Set the input parameters of the camera, such as setting the flash mode.
Data output : The output of the camera includes photo output, preview output and video output, which correspond to three different classes, so the upper layer needs to create different data outputs according to different scenarios.

The camera bottom layer function diagram The camera driver framework model implements the camera HDI interface on the top, and implements the camera Pipeline model on the bottom to manage the hardware devices of the camera. The underlying hardware provides camera device functions, such as camera device management, including camera device enumeration, camera device capability query, stream creation management, and image capture.
(2) Functional modules
Session Management Module <br>The main function of the session management module is to configure the input (device input) and output (data output) of the session, as well as control the start and end of the session. The main interfaces are:

Device input module <br>Camera input is mainly to set the input of the device for the session. The device input module can set and obtain the parameters of the input device, such as flash mode, zoom ratio, focus mode, etc. The main interfaces are:

Data output module <br>Data output module is divided into photo output, preview output and video output according to different scenarios. The output of the photo is provided by the Capture interface of PhotoOutput to provide the photo function, and the preview and video are realized by the interface provided by StreamRepeat. The main interfaces are:

(3) Functional characteristics or application scenarios <br>Related functional interfaces: camera photography, camera preview, camera recording. The main application scenarios of the camera are photographing, previewing and video recording. The following is an analysis of the process for these three scenarios.

The application files in the camera\_standard\interfaces\inner_api\native\test directory are used to take pictures, preview and video functions, and perform source code analysis.
Photo source code analysis <br>The photo function is analyzed according to the main method in the camera\_capture.cpp file. The main calling steps in the main method are listed below, and the functions of the core code are introduced in the comments.

In the process of taking pictures, first obtain a camera manager instance and obtain a list of camera objects, then create and configure a capture session (including configuring camera input, creating consumer Surface, monitoring events, and configuring camera output), and finally taking pictures and releasing resources.
The following is the sequence diagram of the photographing process. The process only analyzes the CameraService. The subsequent operations are called through the Camera's Service and HDI interfaces, and finally called to the Camera's underlying HDF implementation. The following is a detailed analysis of several core codes.

① Create an acquisition session
The App side first calls the CreateCaptureSession interface of CameraManager.

There is a serviceProxy\_ variable in CameraManager, which is assigned when CameraManager is initialized.

CameraManager calls CreateCaptureSession through serviceProxy\_, which actually calls the CreateCaptureSession interface of HcameraService, creates a new HCaptureSession object, and returns it through the CreateCaptureSession parameter.

② Create a consumer Surface and register a listener to listen for buffer updates and call the Surface's CreateSurfaceAsConsumer interface.

Create a ConsumerSurface object, and then initialize the object. Init mainly creates and initializes the BufferQueue, and uses the BufferQueue as a parameter to create the Producer and Consumer of the BufferQueue as data producers and consumers.

Create a CaptureSurfaceListener object. CaptureSurfaceListener inherits the IbufferConsumerListener abstract class and implements the OnBufferAvailable interface. After the capture is successful, the frame data is obtained through the surface's AcquireBuffer method in this interface, and the image is saved.

The next step is to register the listener by calling the RegisterConsumerListener interface of the ConsumerSurface.

Call the RegisterConsumerListener interface of BufferQueueConsumer, and finally call the RegisterConsumerListener interface of BufferQueue.

③ When taking a photo, call the Capture interface of PhotoOutput to realize the photo function.

The Capture interface of HstreamCapture is called through streamCapture\_.

The calling process comes to the Capture interface of StreamOperatorProxy. StreamOperatorProxy is the client side of the HDI module. The client side of the HDI module calls the HDI server side through IPC to perform specific operations. This belongs to the underlying calling module and will not be analyzed for the time being.
Preview source code analysis <br>The preview function is analyzed according to the main method in the camera\_capture.cpp file. The main calling steps in the main method are listed below, and the functions of the core code are introduced in the comments.

In the preview process, first obtain the camera manager instance and obtain the camera object list, then create and configure the acquisition session (including configuring camera input, creating consumer Surface and listening events, configuring preview output), and finally start preview, stop preview, release resource.
The following is the sequence diagram of the preview process. The process only analyzes the CameraService. The subsequent operations are called through the Camera's Service and HDI interfaces, and finally called to the Camera's underlying HDF implementation. Since the preview and photographing processes are partially the same, we only analyze the differences in the process.

① Create preview output <br>First call the CreateCustomPreviewOutput interface of CameraManager.

Call the interface of CreateCustomPreviewOutput of HcameraService. This interface will create an HStreamRepeat object and assign the object to the parameters of streamRepeat, and then create a PreviewOutput object according to streamRepeat and return it. PreviewOutput is used as the output of the preview.

②Start preview <br>First call the Start interface of CaptureSession.

Then call the Start interface of HCaptureSession.

In the invocation of HCaptureSession, the Start interface of HStreamRepeat is called, and the object created by HStreamRepeat when creating the preview output above is finally called to the StartPreview interface of HStreamRepeat.

Call the Capture interface of streamOperator\_ in the StartPreview method, and pass in true as the third parameter, indicating that data is continuously captured. The invocation of streamOperator\_ is an operation in HDI, which is a low-level operation and will not be analyzed for the time being.
Video source code analysis <br>The video function is analyzed according to the main method in the camera\_video.cpp file. The main calling steps in the main method are listed below, and the functions of the core code are introduced in the comments.

During the recording process, first obtain an instance of the camera manager and obtain a list of camera objects, then create and configure a capture session (including configuring camera input, creating video output), and finally record, pause, resume and stop the video.
The following is the sequence diagram of the recording process. The process only analyzes the CameraService. The subsequent operations are called through the Camera's Service and HDI interfaces, and finally called to the Camera's underlying HDF implementation. The video recording process is basically the same as the above process, and the relevant source code analysis is performed for the difference process.

①Create video output <br>First call the CreateVideoOutput interface of CameraManager, and the CameraManager will call the CreateVideoOutput of the serviceProxy\_ variable.

serviceProxy\_ will eventually call the CreateVideoOutput interface of HcameraService, and an HStreamRepeat object will be created in HcameraService. After the creation is successful, the object will be assigned to the second parameter streamRepeat of CreateVideoOutput, which will be used in CameraManager as a parameter for creating VideoOutput.

②Start recording <br>First call the Start interface of VideoOutput, and then call the Start interface of streamRepeat\_.

Finally, the function of recording is realized by the StartVideo interface.

Call the Capture interface of streamOperator to record, where the second parameter captureInfoVideo is the video related information parameter.

3. Summary

This paper firstly analyzes the code of the application layer of the camera preview, photo and video functions, then sorts out the process of the framework layer, and finally analyzes the framework source code. I hope that this article can help developers initially master the entire process of OpenHarmony Camera source code work.关于OpenHarmony多媒体子系统方面的内容，我已经发表过《 http://mp.weixin.qq.com/s?__biz=MzkzNTIyNTY3Mg==&mid=2247498809&idx=1&sn=a424ecfd1425465e8e5209e14fae4546&chksm=c2b38c1ef5c40508e58d86fb827190ef04821ef028f56eb0a71ca3566122aa5a229e06c1c7fe&scene=21#wechat_redirect 》《 http: //mp.weixin.qq.com/s?__biz=MzkzNTIyNTY3Mg==&mid=2247498048&idx=1&sn=8f6e5bfb0f3bba6c17614a4c6e69f6a5&chksm=c2b38967f5c40071dd00b5252a379fc8e405c023784f26a2eebfe094db6af1dddc64247aa788&scene=21#wechat_redirect 》两篇文章，感兴趣的朋友可以点击阅读，希望大家通过学习能够掌握更How multiple OpenHarmony multimedia subsystems work.