JavaCV's camera combat six: save as mp4 file (with sound)

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: All original articles are categorized and summarized and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

Overview of this article

• This article is the sixth part of "JavaCV's Camera Actual Combat". In the article "JavaCV's Camera Actual Combat III: Save as an mp4 File" , we record the content of the camera as an mp4 file. Strands of Flaws: No Sound
• Although the theme of "JavaCV's Camera Combat" series is camera processing, it is clear that audio and video soundness is the most common situation, so this article fills in the deficiencies of in the previous section of : coding to realize camera and microphone recording

About audio capture and recording

• The code of this article is based on the source code of "JavaCV's Camera Practice III: Save as mp4 File" with an audio processing part added
• Before coding, let's analyze the changes in the specific code logic after adding audio processing.
• The operation of saving only the video, compared with saving the audio, the difference between the steps is shown in the figure below, and the dark block is the newly added operation:

• In contrast, when the application ends, when all resources are released, the operation of audio and video is more than when there is only video. As shown in the following figure, dark color is the operation of releasing audio-related resources:

• In order to make the code simpler, I put the audio-related processing in a class named <font color="blue">AudioService</font>, that is to say, the code of the dark part of the above two pictures is in AudioService In .java, the main program uses this class to complete audio processing
• Next start coding

Develop audio processing class AudioService

• The first is the AudioService.java mentioned just now. The main content is the function of the dark block in the previous figure. There are several points to pay attention to later:

package com.bolingcavalry.grabpush.extend;

import lombok.extern.slf4j.Slf4j;
import org.bytedeco.ffmpeg.global.avcodec;
import org.bytedeco.javacv.FFmpegFrameRecorder;
import org.bytedeco.javacv.FrameRecorder;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.TargetDataLine;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.ShortBuffer;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

/**
 * @author willzhao
 * @version 1.0
 * @description 音频相关的服务
 * @date 2021/12/3 8:09
 */
@Slf4j
public class AudioService {

    // 采样率
    private final static int SAMPLE_RATE = 44100;

    // 音频通道数，2表示立体声
    private final static int CHANNEL_NUM = 2;

    // 帧录制器
    private FFmpegFrameRecorder recorder;

    // 定时器
    private ScheduledThreadPoolExecutor sampleTask;

    // 目标数据线，音频数据从这里获取
    private TargetDataLine line;

    // 该数组用于保存从数据线中取得的音频数据
    byte[] audioBytes;

    // 定时任务的线程中会读此变量，而改变此变量的值是在主线程中，因此要用volatile保持可见性
    private volatile boolean isFinish = false;

    /**
     * 帧录制器的音频参数设置
     * @param recorder
     * @throws Exception
     */
    public void setRecorderParams(FrameRecorder recorder) throws Exception {
        this.recorder = (FFmpegFrameRecorder)recorder;

        // 码率恒定
        recorder.setAudioOption("crf", "0");
        // 最高音质
        recorder.setAudioQuality(0);
        // 192 Kbps
        recorder.setAudioBitrate(192000);

        // 采样率
        recorder.setSampleRate(SAMPLE_RATE);

        // 立体声
        recorder.setAudioChannels(2);
        // 编码器
        recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
    }

    /**
     * 音频采样对象的初始化
     * @throws Exception
     */
    public void initSampleService() throws Exception {
        // 音频格式的参数
        AudioFormat audioFormat = new AudioFormat(SAMPLE_RATE, 16, CHANNEL_NUM, true, false);

        // 获取数据线所需的参数
        DataLine.Info dataLineInfo = new DataLine.Info(TargetDataLine.class, audioFormat);

        // 从音频捕获设备取得其数据的数据线，之后的音频数据就从该数据线中获取
        line = (TargetDataLine)AudioSystem.getLine(dataLineInfo);

        line.open(audioFormat);

        // 数据线与音频数据的IO建立联系
        line.start();

        // 每次取得的原始数据大小
        final int audioBufferSize = SAMPLE_RATE * CHANNEL_NUM;

        // 初始化数组，用于暂存原始音频采样数据
        audioBytes = new byte[audioBufferSize];

        // 创建一个定时任务，任务的内容是定时做音频采样，再把采样数据交给帧录制器处理
        sampleTask = new ScheduledThreadPoolExecutor(1);
    }

    /**
     * 程序结束前，释放音频相关的资源
     */
    public void releaseOutputResource() {
        // 结束的标志，避免采样的代码在whlie循环中不退出
        isFinish = true;
        // 结束定时任务
        sampleTask.shutdown();
        // 停止数据线
        line.stop();
        // 关闭数据线
        line.close();
    }

    /**
     * 启动定时任务，每秒执行一次，采集音频数据给帧录制器
     * @param frameRate
     */
    public void startSample(double frameRate) {

        // 启动定时任务，每秒执行一次，采集音频数据给帧录制器
        sampleTask.scheduleAtFixedRate((Runnable) new Runnable() {
            @Override
            public void run() {
                try
                {
                    int nBytesRead = 0;

                    while (nBytesRead == 0 && !isFinish) {
                        // 音频数据是从数据线中取得的
                        nBytesRead = line.read(audioBytes, 0, line.available());
                    }

                    // 如果nBytesRead<1，表示isFinish标志被设置true，此时该结束了
                    if (nBytesRead<1) {
                        return;
                    }

                    // 采样数据是16比特，也就是2字节，对应的数据类型就是short，
                    // 所以准备一个short数组来接受原始的byte数组数据
                    // short是2字节，所以数组长度就是byte数组长度的二分之一
                    int nSamplesRead = nBytesRead / 2;
                    short[] samples = new short[nSamplesRead];

                    // 两个byte放入一个short中的时候，谁在前谁在后？这里用LITTLE_ENDIAN指定拜访顺序，
                    ByteBuffer.wrap(audioBytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(samples);
                    // 将short数组转为ShortBuffer对象，因为帧录制器的入参需要该类型
                    ShortBuffer sBuff = ShortBuffer.wrap(samples, 0, nSamplesRead);

                    // 音频帧交给帧录制器输出
                    recorder.recordSamples(SAMPLE_RATE, CHANNEL_NUM, sBuff);
                }
                catch (FrameRecorder.Exception e) {
                    e.printStackTrace();
                }
            }
        }, 0, 1000 / (long)frameRate, TimeUnit.MILLISECONDS);
    }
}

• There are two things to note in the above code:

1. Focus on <font color="blue">recorder.recordSamples</font>, which saves the audio to an mp4 file
2. The timed task is executed in a new thread, so when the main thread finishes recording, the while loop in the timed task needs to be interrupted, so a volatile variable isFinish is added to help the code in the timed task determine whether to end the while loop immediately

Modify the code that originally only saved the video

• Next is the transformation of <font color="blue">RecordCameraSaveMp4.java</font> in "JavaCV's Camera Actual Combat III: Save as mp4 File" , in order not to affect the code in the previous chapter on github, here I added a new class <font color="blue">RecordCameraSaveMp4WithAudio.java</font>, the content is exactly the same as RecordCameraSaveMp4.java, then let's transform this RecordCameraSaveMp4WithAudio class
• First add a member variable of the AudioService type:

    // 音频服务类
    private AudioService audioService = new AudioService();

• Next is the key. The initOutput method is responsible for the initialization of the frame recorder. Now we need to add audio-related initialization operations, and start a timed task to collect and process audio. As shown below, the three methods of AudioService are called here. , note that the start of the timed task should be placed after the frame recorder is initialized:

    @Override
    protected void initOutput() throws Exception {
        // 实例化FFmpegFrameRecorder
        recorder = new FFmpegFrameRecorder(RECORD_FILE_PATH,        // 存放文件的位置
                                           getCameraImageWidth(),   // 分辨率的宽，与视频源一致
                                           getCameraImageHeight(),  // 分辨率的高，与视频源一致
                                            0);                      // 音频通道，0表示无

        // 文件格式
        recorder.setFormat("mp4");

        // 帧率与抓取器一致
        recorder.setFrameRate(getFrameRate());

        // 编码格式
        recorder.setPixelFormat(AV_PIX_FMT_YUV420P);

        // 编码器类型
        recorder.setVideoCodec(avcodec.AV_CODEC_ID_MPEG4);

        // 视频质量，0表示无损
        recorder.setVideoQuality(0);

        // 设置帧录制器的音频相关参数
        audioService.setRecorderParams(recorder);

        // 音频采样相关的初始化操作
        audioService.initSampleService();

        // 初始化
        recorder.start();

        // 启动定时任务，采集音频帧给帧录制器
        audioService.startSample(getFrameRate());

• The output method is saved as it is and only processes video frames (audio processing is in the timing task)

    @Override
    protected void output(Frame frame) throws Exception {
        // 存盘
        recorder.record(frame);
    }

• In the method of releasing resources, the operation of releasing audio resources is added:

    @Override
    protected void releaseOutputResource() throws Exception {
        // 执行音频服务的资源释放操作
        audioService.releaseOutputResource();

        // 关闭帧录制器
        recorder.close();
    }

• So far, the function of saving camera video and microphone audio as mp4 files has been developed, and then write the main method. Note that the parameter <font color="blue">30</font> means that the capture and recording operations are executed for 30 seconds. Note that this is the duration of the program execution, <font color="red"> is not the duration of the recorded video </font>:

    public static void main(String[] args) {
        // 录制30秒视频
        new RecordCameraSaveMp4WithAudio().action(30);
    }

• Run the main method and wait until the console outputs the content of the red box in the figure below, indicating that the video recording is complete:

• Open the directory where the mp4 file is located, as shown in the figure below, the red box is the file and related information just generated, pay attention to the content of the blue box, which proves that the file contains video and audio data:

• Use VLC to play the verification, the result video and sound are normal
• So far, we have completed the function of saving audio and video files. Thanks to the power of JavaCV, the whole process is so easy and pleasant. Next, please continue to pay attention to Xinchen Originals. The series of "JavaCV's Camera Actual Combat" will show more richness. Applications;

Source code download

• The complete source code of "Camera Combat of JavaCV" can be downloaded from GitHub. The address and link information are shown in the following table ( https://github.com/zq2599/blog_demos ):

name	Link	Remark
Project homepage	https://github.com/zq2599/blog_demos	The project's homepage on GitHub
git repository address (https)	https://github.com/zq2599/blog_demos.git	The warehouse address of the source code of the project, https protocol
git repository address (ssh)	[e-mail]:zq2599/blog_demos.git	The warehouse address of the project source code, ssh protocol

• There are multiple folders in this git project. The source code of this article is in the <font color="blue">javacv-tutorials</font> folder, as shown in the red box below:

• There are multiple sub-projects in <font color="blue">javacv-tutorials</font>. The code of "JavaCV Camera Actual" series is in <font color="red"> simple-grab-push </font> Under the project:

You are not alone, Xinchen Original is with you all the way

Search "Programmer Xinchen", I'm Xinchen, I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos