Android audio and video-EGL source code analysis and C++ implementation

OpenGL is a cross-platform API, and different operating systems (Windows, Android, IOS) have their own screen rendering implementations. So OpenGL defines an intermediate interface layer EGL (Embedded Graphics Library) standard, and the specific implementation is left to each operating system itself

EGL

Simply put, EGL is an intermediate interface layer and a specification. Due to the cross-platform nature of OpenGL, this specification is particularly important. No matter how the various operating systems are running, they cannot deviate from the specification defined by me.

Some basic knowledge of EGL

• EGLDisplay

An abstract system display class defined by EGL for operating device windows.

• EGLConfig

EGL configuration, such as rgba digits

• EGLSurface

Rendering cache, a memory space, all image data to be rendered on the screen must be cached on EGLSurface.

• EGLContext

OpenGL context, used to store OpenGL drawing state information and data.

The process of initializing EGL can be said to be the process of configuring the above information.

OpenGL ES drawing complete process

When we use Java GLSurfaceView, we actually just customize the Render, which implements the GLsurfaceView.Renderer interface, and then the three methods in the custom Render will get callbacks. The Android system actually saved me many steps. . So here we look at complete process (1). obtain a display device (corresponding to the above EGLDisplay)

/*
 * Get an EGL instance */
 mEgl = (EGL10) EGLContext.getEGL();
 
/*
 * Get to the default display. */
 mEglDisplay = mEgl.eglGetDisplay(EGL10.EGL_DEFAULT_DISPLAY);

(2). initialize EGL

int[] version = new int[2];
//初始化屏幕
if(!mEgl.eglInitialize(mEglDisplay, version)) {
    throw new RuntimeException("eglInitialize failed");
}

(3). select Config (use EGLConfig to configure parameters)

//这段代码的作用是选择EGL配置， 即可以自己先设定好一个你希望的EGL配置，比如说RGB三种颜色各占几位，你可以随便配，而EGL可能不能满足你所有的要求，于是它会返回一些与你的要求最接近的配置供你选择。
if (!egl.eglChooseConfig(display, mConfigSpec, configs, numConfigs,
 num_config)) {
    throw new IllegalArgumentException("eglChooseConfig#2 failed");
}

(4). creates EGLContext

//从上一步EGL返回的配置列表中选择一种配置，用来创建EGL Context。
egl.eglCreateContext(display, config, EGL10.EGL_NO_CONTEXT,
 mEGLContextClientVersion != 0 ? attrib_list : null);

(5). get EGLSurface

//创建一个窗口Surface，可以看成屏幕所对应的内存
 egl.eglCreateWindowSurface(display, config, nativeWindow, null)

PS NativeWindow here is the surfaceHolder of GLSurfaceView

(6). binds the rendering environment to the current thread

/*
 * Before we can issue GL commands, we need to make sure * the context is current and bound to a surface. */
 if (!mEgl.eglMakeCurrent(mEglDisplay, mEglSurface, mEglSurface, mEglContext)) {
    /*
    * Could not make the context current, probably because the underlying * SurfaceView surface has been destroyed. */ 
     logEglErrorAsWarning("EGLHelper", "eglMakeCurrent", mEgl.eglGetError());
     return false;
 }

loop drawing

loop:{
    //绘制中....
    //(7).交换缓冲区
    mEglHelper.swap();
}

public int swap() {
    if (! mEgl.eglSwapBuffers(mEglDisplay, mEglSurface)) {
        return mEgl.eglGetError();
    }
    return EGL10.EGL_SUCCESS;
}

Java - GLSurfaceView/GLTextureView

Above we introduced some basic knowledge of EGL, and then we look at the specific implementation of EGL in GLSurfaceView/GLTextureView, let's analyze the Android system EGL and GL threads from the source code.

GLThread

Let's take a look at GLThread. GLThread is also inherited from the ordinary Thread class. In theory, it is an ordinary thread. Why does it have OpenGL drawing capabilities? Continuing to look down, the most important part is the guardedRun() method.

static class GLThread extends Thread {
    ...
    @Override
    public void run() {
      
        try {
                guardedRun();
         } catch (InterruptedException e) {
                // fall thru and exit normally
         } finally {
                sGLThreadManager.threadExiting(this);
         }
    }
}

Let's take a look at what's in the guardedRun() method and what's roughly done in guardedRun():

private void guardedRun() throws InterruptedException {
    while(true){
        //if ready to draw
        ...
        mEglHelper.start();//对应于上面完整流程中的(1)(2)(3)(4)
        
        ...
        mEglHelper.createSurface()//对应于上面完整流程中的(5)(6)
        
        ...
        回调GLSurfaceView.Renderer的onSurfaceCreated();
        ...
        回调GLSurfaceView.Renderer的onSurfaceChanged();
        ...
        回调GLSurfaceView.Renderer的onDrawFrame();
        
        ...
         mEglHelper.swap();//对应于上面完整流程中的(5)(7)
    }
}

From our analysis above, we know that the GLThread in GLSurfaceView is a normal thread, but it operates correctly in accordance with the complete flow of OpenGL drawing, so it has OpenGL drawing capabilities. So, if we create a thread ourselves and follow this method of operation, can we also draw in the thread we created? The answer is yes (this is not exactly the interface meaning of EGL), I will give the implementation of EGL in Native C/C++ below.

Native - EGL

There is no ready-made EGL environment in the Android Native environment, so we must implement the EGL environment ourselves when developing OpenGL NDK. Then how to achieve it, we only need to refer to the writing of GLThread in GLSurfaceView to achieve EGL in Native .

PS

The following content may require you to be familiar with C/C++ and NDK

Step 1 Implement a function similar to GLThread in Java GLSurfaceView

gl_render.h

class GLRender {
    private:
         const char *TAG = "GLRender";
         //OpenGL渲染状态
         enum STATE {
             NO_SURFACE, //没有有效的surface
             FRESH_SURFACE, //持有一个为初始化的新的surface
             RENDERING, //初始化完毕，可以开始渲染
             SURFACE_DESTROY, //surface销毁
             STOP //停止绘制
         };
         JNIEnv *m_env = NULL;
         //线程依附的jvm环境
         JavaVM *m_jvm_for_thread = NULL;
         //Surface引用，必须要使用引用，否则无法在线程中操作
         jobject m_surface_ref = NULL;
         //本地屏幕
         ANativeWindow *m_native_window = NULL;
         //EGL显示表面
         EglSurface *m_egl_surface = NULL;
         int m_window_width = 0;
         int m_window_height = 0;
         
         // 绘制代理器
         ImageRender *pImageRender;
         
         //OpenGL渲染状态
         STATE m_state = NO_SURFACE;
         // 初始化相关的方法
         void InitRenderThread();
         bool InitEGL();
         void InitDspWindow(JNIEnv *env);
         // 创建/销毁 Surface void CreateSurface();
         void DestroySurface();
         // 渲染方法
         void Render();
         void ReleaseSurface();
         void ReleaseWindow();
         // 渲染线程回调方法
         static void sRenderThread(std::shared_ptr<GLRender> that);
    public:
         GLRender(JNIEnv *env);
         ~GLRender();
         //外部传入Surface
         void SetSurface(jobject surface);
      
         void Stop();
         void SetBitmapRender(ImageRender *bitmapRender);
        // 释放资源相关方法
         void ReleaseRender();
         
         ImageRender *GetImageRender();
};

gl_render.cpp

//构造函数
GLRender::GLRender(JNIEnv *env) {
     this->m_env = env;
     //获取JVM虚拟机，为创建线程作准备
     env->GetJavaVM(&m_jvm_for_thread);
     InitRenderThread();
}
//析构函数
GLRender::~GLRender() {
    delete m_egl_surface;
}

//初始化渲染线程
void GLRender::InitRenderThread() {
    // 使用智能指针，线程结束时，自动删除本类指针
     std::shared_ptr<GLRender> that(this);
     std::thread t(sRenderThread, that);
     t.detach();
}

//线程回调函数
void GLRender::sRenderThread(std::shared_ptr<GLRender> that) {
    JNIEnv *env;
     //(1) 将线程附加到虚拟机，并获取env
     if (that->m_jvm_for_thread->AttachCurrentThread(&env, NULL) != JNI_OK) {
            LOGE(that->TAG, "线程初始化异常");
            return; 
     }
     // (2) 初始化 EGL 
    if (!that->InitEGL()) {
         //解除线程和jvm关联
         that->m_jvm_for_thread->DetachCurrentThread();
         return; 
     }
     
     //进入循环
    while (true) {
            //根据OpenGL渲染状态进入不同的处理
            switch (that->m_state) {
                //刷新Surface，从外面设置Surface后m_state置为该状态，说明已经从外部(java层)获得Surface的对象了
                case FRESH_SURFACE:
                     LOGI(that->TAG, "Loop Render FRESH_SURFACE")
                     // (3) 初始化Window
                     that->InitDspWindow(env);
                     // (4) 创建EglSurface
                     that->CreateSurface();
                     // m_state置为RENDERING状态进入渲染
                     that->m_state = RENDERING;
                     break; 
                 case RENDERING:
                    LOGI(that->TAG, "Loop Render RENDERING")
                    // (5) 渲染
                    that->Render();
                    break; 
               
                 case STOP:
                    LOGI(that->TAG, "Loop Render STOP")
                    //(6) 解除线程和jvm关联
                     that->ReleaseRender();
                     that->m_jvm_for_thread->DetachCurrentThread();
                     return; 
                case SURFACE_DESTROY:
                    LOGI(that->TAG, "Loop Render SURFACE_DESTROY")
                    //(7) 释放资源
                    that->DestroySurface();
                    that->m_state = NO_SURFACE;
                    break; 
                case NO_SURFACE:
                default:
                    break;
     }
    usleep(20000);
 }
}

The steps in each process code of our defined GLRender have been marked. Although the amount of code is relatively large, our c++ class analysis is similar to that of java.

PS The steps (3)(4) in the above figure correspond to the step comments in the code

(1) attach the thread to the virtual machine and get the env

This step is simple and clear, let’s look down

EGL package preparation

We knew some basic knowledge of EGL in the previous article, EGLDiaplay , EGLConfig , EGLSurface , EGLContext , we need to encapsulate these basic classes, then how to encapsulate, let’s take a look at the custom ones in our previous article What is needed for the GLRender class gl_render.h

//Surface引用，必须要使用引用，否则无法在线程中操作
jobject m_surface_ref = NULL;
//本地屏幕
ANativeWindow *m_native_window = NULL;
//EGL显示表面 注意这里是我们自定义的EglSurface包装类而不是系统提供的EGLSurface哦
EglSurface *m_egl_surface = NULL;

For gl_render, the is an external Surface object, ours here is jobject m_surface_ref , then the output needs to be ANativeWindow , EglSurface

For ANativeWindow you can check the official document ANativeWindow

What about EglSurface ,

egl_surface.h

class EglSurface {
private:
    const char *TAG = "EglSurface";
    //本地屏幕
     ANativeWindow *m_native_window = NULL;
     //封装了EGLDisplay EGLConfig EGLContext的自定义类
     EglCore *m_core;
     //EGL API提供的 EGLSurface
     EGLSurface m_surface;
}

You can see that the idea of our definition above is also the separation of V (View) and C (Controller).

egl_core.h

class EglCore {
private:
    const char *TAG = "EglCore";
     //EGL显示窗口
     EGLDisplay m_egl_dsp = EGL_NO_DISPLAY;
     //EGL上下文
     EGLContext m_egl_context = EGL_NO_CONTEXT;
     //EGL配置
     EGLConfig m_egl_config;
}

With the above preparations, we will follow the steps of the flowchart step by step.

(2) Initialize EGL

gl_render.cpp

bool GLRender::InitEGL() {
    //创建EglSurface对象
    m_egl_surface = new EglSurface();
    //调用EglSurface的init方法
    return m_egl_surface->Init();
}

egl_surface.cpp

PS We also mentioned above that the initialization of EGL is mainly the operation of EGLDisplay EGLConfig EGLContext, so now it is the operation of EGLCore.

EglSurface::EglSurface() {
    //创建EGLCore
    m_core = new EglCore();
}

bool EglSurface::Init() {
    //调用EGLCore的init方法
    return m_core->Init(NULL);
}

egl_core.cpp

EglCore::EglCore() {
}


bool EglCore::Init(EGLContext share_ctx) {
    if (m_egl_dsp != EGL_NO_DISPLAY) {
        LOGE(TAG, "EGL already set up")
        return true;
     }
    if (share_ctx == NULL) {
            share_ctx = EGL_NO_CONTEXT;
     }
     //获取Dispaly
    m_egl_dsp = eglGetDisplay(EGL_DEFAULT_DISPLAY);
     if (m_egl_dsp == EGL_NO_DISPLAY || eglGetError() != EGL_SUCCESS) {
            LOGE(TAG, "EGL init display fail")
            return false;
     }
        EGLint major_ver, minor_ver;
     //初始化egl
     EGLBoolean success = eglInitialize(m_egl_dsp, &major_ver, &minor_ver);
     if (success != EGL_TRUE || eglGetError() != EGL_SUCCESS) {
            LOGE(TAG, "EGL init fail")
            return false;
     }
        LOGI(TAG, "EGL version: %d.%d", major_ver, minor_ver)
     //获取EGLConfig   
     m_egl_config = GetEGLConfig();
     const EGLint attr[] = {EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE};
     //创建EGLContext
     m_egl_context = eglCreateContext(m_egl_dsp, m_egl_config, share_ctx, attr);
     if (m_egl_context == EGL_NO_CONTEXT) {
            LOGE(TAG, "EGL create fail, error is %x", eglGetError());
     return false; }
        EGLint egl_format;
     success = eglGetConfigAttrib(m_egl_dsp, m_egl_config, EGL_NATIVE_VISUAL_ID, &egl_format);
     if (success != EGL_TRUE || eglGetError() != EGL_SUCCESS) {
            LOGE(TAG, "EGL get config fail, error is %x", eglGetError())
            return false;
     }
    LOGI(TAG, "EGL init success")
    return true;
}

EGLConfig EglCore::GetEGLConfig() {
    EGLint numConfigs;
    EGLConfig config;

  //希望的最小配置，
    static const EGLint CONFIG_ATTRIBS[] = {
            EGL_BUFFER_SIZE, EGL_DONT_CARE,
            EGL_RED_SIZE, 8,//R 位数
            EGL_GREEN_SIZE, 8,//G 位数
            EGL_BLUE_SIZE, 8,//B 位数
            EGL_ALPHA_SIZE, 8,//A 位数
            EGL_DEPTH_SIZE, 16,//深度
            EGL_STENCIL_SIZE, EGL_DONT_CARE,
            EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT,
            EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
            EGL_NONE // the end 结束标志
    };
  //根据你所设定的最小配置系统会选择一个满足你最低要求的配置，这个真正的配置往往要比你期望的属性更多
    EGLBoolean success = eglChooseConfig(m_egl_dsp, CONFIG_ATTRIBS, &config, 1, &numConfigs);
    if (!success || eglGetError() != EGL_SUCCESS) {
        LOGE(TAG, "EGL config fail")
        return NULL;
    }
    return config;
}

(3) Create Window

gl_render.cpp

void GLRender::InitDspWindow(JNIEnv *env) {
  //传进来的Surface对象的引用
    if (m_surface_ref != NULL) {
        // 初始化窗口
        m_native_window = ANativeWindow_fromSurface(env, m_surface_ref);

        // 绘制区域的宽高
        m_window_width = ANativeWindow_getWidth(m_native_window);
        m_window_height = ANativeWindow_getHeight(m_native_window);

        //设置宽高限制缓冲区中的像素数量
        ANativeWindow_setBuffersGeometry(m_native_window, m_window_width,
                                         m_window_height, WINDOW_FORMAT_RGBA_8888);

        LOGD(TAG, "View Port width: %d, height: %d", m_window_width, m_window_height)
    }
}

(4) Create EglSurface and bind it to the thread

gl_render.cpp

void GLRender::CreateSurface() {
    m_egl_surface->CreateEglSurface(m_native_window, m_window_width, m_window_height);
    glViewport(0, 0, m_window_width, m_window_height);
}

egl_surface.cpp

/**
 * 
 * @param native_window 传入上一步创建的ANativeWindow
 * @param width 
 * @param height 
 */
void EglSurface::CreateEglSurface(ANativeWindow *native_window, int width, int height) {
    if (native_window != NULL) {
        this->m_native_window = native_window;
        m_surface = m_core->CreateWindSurface(m_native_window);
    } else {
        m_surface = m_core->CreateOffScreenSurface(width, height);
    }
    if (m_surface == NULL) {
        LOGE(TAG, "EGL create window surface fail")
        Release();
    }
    MakeCurrent();
}

void EglSurface::MakeCurrent() {
    m_core->MakeCurrent(m_surface);
}

egl_core.cpp

EGLSurface EglCore::CreateWindSurface(ANativeWindow *window) {
  //调用EGL Native API创建Window Surface
    EGLSurface surface = eglCreateWindowSurface(m_egl_dsp, m_egl_config, window, 0);
    if (eglGetError() != EGL_SUCCESS) {
        LOGI(TAG, "EGL create window surface fail")
        return NULL;
    }
    return surface;
}

void EglCore::MakeCurrent(EGLSurface egl_surface) {
  //调用EGL Native API 绑定渲染环境到当前线程
    if (!eglMakeCurrent(m_egl_dsp, egl_surface, egl_surface, m_egl_context)) {
        LOGE(TAG, "EGL make current fail");
    }
}

(5) Rendering

gl_render.cpp

void GLRender::Render() {
    if (RENDERING == m_state) {
        pImageRender->DoDraw();//画画画....
        m_egl_surface->SwapBuffers();
    }
}

egl_surface.cpp

void EglSurface::SwapBuffers() {
    m_core->SwapBuffer(m_surface);
}

egl_core.cpp

void EglCore::SwapBuffer(EGLSurface egl_surface) {
  //调用EGL Native API
    eglSwapBuffers(m_egl_dsp, egl_surface);
}

The rest of the suspension and destruction will be left to the readers for their own research.

Code

EGLDemoActivity.java

EGL Native