We know that Camera collects and returns YUV data, and AudioRecord is PCM. We need to encode (compress and encode) these data. Here we are talking about the pit that cannot be escaped in audio and video codec on Android-MediaCodec
MediaCodec
PSMediaCodec can be used encoding / decoding audio / video.A brief introduction to MediaCodec
The MediaCodec class can be used to access low-level media codecs, that is, encoder/decoder components. It is part of the Android low-level multimedia support infrastructure (usually used with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface and AudioTrack). For the description of MediaCodec, please refer to the official introduction MediaCodec
Broadly speaking, a codec processes input data to generate output data. It processes data asynchronously and uses a set of input and output buffers. In simple cases, you request (or receive) an empty input buffer, fill it with data and send it to the codec for processing. The codec ran out of data and converted it to one of the empty output buffers. Finally, you request (or receive) the filled output buffer, use its content and release it back to the codec.
If PS readers still have an impression of the producer-consumer model, then the operating mode of MediaCodec is actually not difficult to understand.
Below is a simple class diagram of MediaCodec
MediaCodec state machine
In the life cycle of MediaCodec, the codec is conceptually in one of the following three states: Stopped, Executing or Released. The collective state of Stopped is actually a collection of three states: Uninitialized, Configured and Error, while the Executing state conceptually passes through three sub-states: Flushed, Running and Stream-of-Stream.
When creating a codec using one of the factory methods, the codec is in an uninitialized state. First, you need to configure it through configure(...) to make it enter the configured state, and then call start() to move it to the execution state. In this state, you can process data through the above buffer queue operations.
The execution state has three sub-states: Flushed, Running and Stream-of-Stream. After start(), the codec is in the Flushed sub-state immediately, which contains all buffers. Once the first input buffer is dequeued, the codec will move to the "Running" sub-state, where most of the time will be spent. When you queue the input buffer with the end-of-stream marker, the codec will transition to the End-of-Stream sub-state. In this state, the codec will no longer accept other input buffers, but will still generate output buffers until the end of the stream is reached at the output. In the execution state, you can use flush() to return to the "flush" sub-state at any time.
Call stop() to return the codec to the Uninitialized state, which can then be configured again. After completing the operation with the codec, it must be released by calling release().
In rare cases, the codec may encounter an error and enter the "error" state. Use invalid return values from queued operations or sometimes through exceptions to convey this information. Call reset() to make the codec available again. You can call it from any state to move the codec back to the "Uninitialized" state. Otherwise, call release() to move to the "Released" state of the terminal.
PSMediaCodec data processing mode can be divided into synchronous and asynchronous, we will analyze one by one below
MediaCodec synchronization mode
Upload code
public H264MediaCodecEncoder(int width, int height) {
//设置MediaFormat的参数
MediaFormat mediaFormat = MediaFormat.createVideoFormat(MIMETYPE_VIDEO_AVC, width, height);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, width * height * 5);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30);//FPS
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
try {
//通过MIMETYPE创建MediaCodec实例
mMediaCodec = MediaCodec.createEncoderByType(MIMETYPE_VIDEO_AVC);
//调用configure,传入的MediaCodec.CONFIGURE_FLAG_ENCODE表示编码
mMediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
//调用start
mMediaCodec.start();
} catch (Exception e) {
e.printStackTrace();
}
}
Call putData to add raw YUV data to the queue
public void putData(byte[] buffer) {
if (yuv420Queue.size() >= 10) {
yuv420Queue.poll();
}
yuv420Queue.add(buffer);
}
//开启编码
public void startEncoder() {
isRunning = true;
ExecutorService executorService = Executors.newSingleThreadExecutor();
executorService.execute(new Runnable() {
@Override
public void run() {
byte[] input = null;
while (isRunning) {
if (yuv420Queue.size() > 0) {
//从队列中取数据
input = yuv420Queue.poll();
}
if (input != null) {
try {
//【1】dequeueInputBuffer
int inputBufferIndex = mMediaCodec.dequeueInputBuffer(TIMEOUT_S);
if (inputBufferIndex >= 0) {
//【2】getInputBuffer
ByteBuffer inputBuffer = null;
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.LOLLIPOP) {
inputBuffer = mMediaCodec.getInputBuffer(inputBufferIndex);
} else {
inputBuffer = mMediaCodec.getInputBuffers()[inputBufferIndex];
}
inputBuffer.clear();
inputBuffer.put(input);
//【3】queueInputBuffer
mMediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, getPTSUs(), 0);
}
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
//【4】dequeueOutputBuffer
int outputBufferIndex = mMediaCodec.dequeueOutputBuffer(bufferInfo, TIMEOUT_S);
if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = mMediaCodec.getOutputFormat();
if (null != mEncoderCallback) {
mEncoderCallback.outputMediaFormatChanged(H264_ENCODER, newFormat);
}
if (mMuxer != null) {
if (mMuxerStarted) {
throw new RuntimeException("format changed twice");
}
// now that we have the Magic Goodies, start the muxer
mTrackIndex = mMuxer.addTrack(newFormat);
mMuxer.start();
mMuxerStarted = true;
}
}
while (outputBufferIndex >= 0) {
ByteBuffer outputBuffer = null;
//【5】getOutputBuffer
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.LOLLIPOP) {
outputBuffer = mMediaCodec.getOutputBuffer(outputBufferIndex);
} else {
outputBuffer = mMediaCodec.getOutputBuffers()[outputBufferIndex];
}
if (bufferInfo.flags == MediaCodec.BUFFER_FLAG_CODEC_CONFIG) {
bufferInfo.size = 0;
}
if (bufferInfo.size > 0) {
// adjust the ByteBuffer values to match BufferInfo (not needed?)
outputBuffer.position(bufferInfo.offset);
outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
// write encoded data to muxer(need to adjust presentationTimeUs.
bufferInfo.presentationTimeUs = getPTSUs();
if (mEncoderCallback != null) {
//回调
mEncoderCallback.onEncodeOutput(H264_ENCODER, outputBuffer, bufferInfo);
}
prevOutputPTSUs = bufferInfo.presentationTimeUs;
if (mMuxer != null) {
if (!mMuxerStarted) {
throw new RuntimeException("muxer hasn't started");
}
mMuxer.writeSampleData(mTrackIndex, outputBuffer, bufferInfo);
}
}
mMediaCodec.releaseOutputBuffer(outputBufferIndex, false);
bufferInfo = new MediaCodec.BufferInfo();
outputBufferIndex = mMediaCodec.dequeueOutputBuffer(bufferInfo, TIMEOUT_S);
}
} catch (Throwable throwable) {
throwable.printStackTrace();
}
} else {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
});
}
The time-consuming operation of PS encoding and decoding must be completed in a separate thread. We have a buffer queueArrayBlockingQueue<byte[]> yuv420Queue = new ArrayBlockingQueue<>(10);, which is used to receive byte[] YUV data passed in from the Camera callback. We have created a new ready-made slave buffer Queueyuv420Queuecyclically read data and submit it to MediaCodec for encoding. The format of the encoding ismMediaCodec = MediaCodec.createEncoderByType(MIMETYPE_VIDEO_AVC);. The output here is currently the most widely used formatH264
For the complete code, please see H264MediaCodecEncoder
MediaCodec asynchronous mode
@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
public H264MediaCodecAsyncEncoder(int width, int height) {
MediaFormat mediaFormat = MediaFormat.createVideoFormat(MIMETYPE_VIDEO_AVC, width, height);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, width * height * 5);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30);//FPS
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
try {
mMediaCodec = MediaCodec.createEncoderByType(MIMETYPE_VIDEO_AVC);
mMediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
//设置回调
mMediaCodec.setCallback(new MediaCodec.Callback() {
@Override
/**
* Called when an input buffer becomes available.
*
* @param codec The MediaCodec object.
* @param index The index of the available input buffer.
*/
public void onInputBufferAvailable(@NonNull MediaCodec codec, int index) {
Log.i("MFB", "onInputBufferAvailable:" + index);
byte[] input = null;
if (isRunning) {
if (yuv420Queue.size() > 0) {
input = yuv420Queue.poll();
}
if (input != null) {
ByteBuffer inputBuffer = codec.getInputBuffer(index);
inputBuffer.clear();
inputBuffer.put(input);
codec.queueInputBuffer(index, 0, input.length, getPTSUs(), 0);
}
}
}
@Override
/**
* Called when an output buffer becomes available.
*
* @param codec The MediaCodec object.
* @param index The index of the available output buffer.
* @param info Info regarding the available output buffer {@link MediaCodec.BufferInfo}.
*/
public void onOutputBufferAvailable(@NonNull MediaCodec codec, int index, @NonNull MediaCodec.BufferInfo info) {
Log.i("MFB", "onOutputBufferAvailable:" + index);
ByteBuffer outputBuffer = codec.getOutputBuffer(index);
if (info.flags == MediaCodec.BUFFER_FLAG_CODEC_CONFIG) {
info.size = 0;
}
if (info.size > 0) {
// adjust the ByteBuffer values to match BufferInfo (not needed?)
outputBuffer.position(info.offset);
outputBuffer.limit(info.offset + info.size);
// write encoded data to muxer(need to adjust presentationTimeUs.
info.presentationTimeUs = getPTSUs();
if (mEncoderCallback != null) {
//回调
mEncoderCallback.onEncodeOutput(H264_ENCODER, outputBuffer, info);
}
prevOutputPTSUs = info.presentationTimeUs;
if (mMuxer != null) {
if (!mMuxerStarted) {
throw new RuntimeException("muxer hasn't started");
}
mMuxer.writeSampleData(mTrackIndex, outputBuffer, info);
}
}
codec.releaseOutputBuffer(index, false);
}
@Override
public void onError(@NonNull MediaCodec codec, @NonNull MediaCodec.CodecException e) {
}
@Override
/**
* Called when the output format has changed
*
* @param codec The MediaCodec object.
* @param format The new output format.
*/
public void onOutputFormatChanged(@NonNull MediaCodec codec, @NonNull MediaFormat format) {
if (null != mEncoderCallback) {
mEncoderCallback.outputMediaFormatChanged(H264_ENCODER, format);
}
if (mMuxer != null) {
if (mMuxerStarted) {
throw new RuntimeException("format changed twice");
}
// now that we have the Magic Goodies, start the muxer
mTrackIndex = mMuxer.addTrack(format);
mMuxer.start();
mMuxerStarted = true;
}
}
});
mMediaCodec.start();
} catch (Exception e) {
e.printStackTrace();
}
}
For the complete code, please see H264MediaCodecAsyncEncoder
MediaCodec summary
MediaCodec is used to encode and decode audio and video (some articles in this process are also called hard solution), through MediaCodec.createEncoderByType(MIMETYPE_VIDEO_AVC) function to create an audio or video encoder, similarly through MediaCodec.createDecoderByType(MIMETYPE_VIDEO_AVC) create an audio or video decoder. For the different parameters required in the audio and video codec, use MediaFormat to specify
summary
This article analyzes MediaCodec in detail, and readers can perform actual drills according to the corresponding Demo of the blog
Put the demo address detail Demo
android
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。