Webrtc screen sharing

Function introduction

Screen sharing includes two functions: screen capture and video streaming. Unlike remote desktops, screen sharing only shares local desktop content to the network in the form of video streams. The focus of this article is to explain how to use webrtc's screen capture function. For the application of the webrtc video encoding and transmission function, a special article is needed to explain it. I will not expand it here for the time being, but focus on the screen capture. webrtc provides screen sharing functions for multiple platforms. Here we take the windows 10 platform as the development environment to describe how to use webrtc's screen sharing functions on the windows platform.

Screen sharing includes three collection methods, including full-screen collection, window collection, and designated area collection. Each collection method has its own specific usage scenarios. The main focus here is how to use webrtc to implement these three collection methods.

The source code of screen capture in webrtc is in the webrtc/src/modules/desktop_capture/ directory. The DesktopCapturer class is defined in desktop_capturer.h in the desktop_capture directory. The DesktopCapturer class abstracts the interface used for screen capture. The screen capture of the windows platform is implemented in the webrtc/src/modules/desktop_capture/win directory. There are ScreenCapturerWinGdi class, ScreenCapturerWinMagnifier class, DesktopAndCursorComposer class, WindowCapturerWinGdi class, WgcCapturerWin class. These implementation classes respectively implement the screen capture and window capture functions of the Windows platform.

Full screen capture

The ScreenCapturerWinGdi class only implements the simple screen capture function. If you need to filter out the specified window during full screen capture, you need to use the ScreenCapturerWinMagnifier class to set the window to be filtered through the SetExcludedWindow interface. The ScreenCapturerWinMagnifier class only implements the function of filtering the window. If you need to display the mouse position while filtering the window, you must use the DesktopAndCursorComposer class. The DesktopAndCursorComposer class implements the function of merging the mouse position with the screen image.

Window collection

The WindowCapturerWinGdi class was the first to realize the function of capturing the specified window, but for windows with hardware acceleration enabled, the contents of the window cannot be captured, only the frame of the window can be captured. In the latest version of webrtc, the WgcCapturerWin class is provided. WgcCapturerWin implements the capture full screen and capture window functions. Importantly, WgcCapturerWin can capture windows with hardware acceleration enabled, such as chrome browsers.

Collection area

The DesktopCapturer class does not provide an interface to capture the specified area. Therefore, you need to add an impure virtual function to the DesktopCapturer class. The function accepts four parameters, which are the coordinates x and y of the upper left corner of the specified area, as well as the area size width and height. Then create a new class that inherits ScreenCapturerWinGdi, and then overload the CaptureFrame method. You can copy the CaptureFrame implementation in the ScreenCapturerWinGdi class, and then specify the captured area as a custom area (modify the original full-screen area to a custom area). In this way, the designated area is collected.

<u> pits: The width of the designated area should preferably be an integer multiple of 16, and it cannot be an odd number. The height is preferably an integer multiple of 2. </u>

Screen sharing process

1. Create an instance of DesktopCapturer, you can create different DesktopCapturer implementation classes according to your needs, such as ScreenCapturerWinMagnifier class or DesktopAndCursorComposer class or WgcCapturerWin class.

2. Get the screen ID list or window ID list.

3. Specify the screen ID or window ID list to be collected.

4. Register the data callback and start collecting.

5. Encode and transmit the screen image in the callback.

<u> pits: DesktopCapturer instance must be created, initialized and destroyed in the same thread. </u>

Code example

#ifndef __RCRTC_DESKTOP_CAPTURER__
#define __RCRTC_DESKTOP_CAPTURER__

#include "api/video/i420_buffer.h"

#include "libyuv/convert.h"
#include "libyuv/video_common.h"

#include "media/base/video_broadcaster.h"
#include "media/base/video_common.h"

#include "modules/desktop_capture/desktop_and_cursor_composer.h"
#include "rtc_base/thread.h"
#include "system_wrappers/include/sleep.h"
#include "webrtc_desktop_capturer.h"

typedef void (*desktop_capture_frame_callback)(int width,
                                               int height,
                                               int y_stride,
                                               int u_stride,
                                               int v_stride,
                                               const uint8_t* y,
                                               const uint8_t* u,
                                               const uint8_t* v,
                                               void* context);

class RcrtcDesktopCapturer : public webrtc::DesktopCapturer::Callback {
 public:
  RcrtcDesktopCapturer(const std::map<std::string, std::string>& opts) {}
  inline ~RcrtcDesktopCapturer() override {}

  // overide webrtc::DesktopCapturer::Callback
  void OnCaptureResult(webrtc::DesktopCapturer::Result result,
                       std::unique_ptr<webrtc::DesktopFrame> frame) override {
    if (result == webrtc::DesktopCapturer::Result::SUCCESS) {
      int width = frame->size().width();
      int height = frame->size().height();

      rtc::scoped_refptr<webrtc::I420Buffer> I420buffer =
          webrtc::I420Buffer::Create(width, height);

      const int conversionResult = libyuv::ConvertToI420(
          frame->data(), 0, I420buffer->MutableDataY(), I420buffer->StrideY(),
          I420buffer->MutableDataU(), I420buffer->StrideU(),
          I420buffer->MutableDataV(), I420buffer->StrideV(), 0, 0,
          // width, height,
          I420buffer->width(), I420buffer->height(), I420buffer->width(),
          I420buffer->height(), libyuv::kRotate0, ::libyuv::FOURCC_ARGB);

      if (conversionResult >= 0) {
        webrtc::VideoFrame videoFrame(I420buffer,
                                      webrtc::VideoRotation::kVideoRotation_0,
                                      rtc::TimeMicros());

        rtc::scoped_refptr<webrtc::I420BufferInterface> buffer(
            videoFrame.video_frame_buffer()->ToI420());
        _frameCallback(buffer->width(), buffer->height(), buffer->StrideY(),
                       buffer->StrideU(), buffer->StrideV(), buffer->DataY(),
                       buffer->DataU(), buffer->DataV(), this->_userContext);
      }
    }
  }

  void setExccludeWindow(webrtc::DesktopCapturer::Source windowId) {
    _excludeWindowList.push_back(windowId);
  }

  void CaptureThread() {
    webrtc::DesktopCaptureOptions opts =
        webrtc::DesktopCaptureOptions::CreateDefault();
    opts.set_allow_use_magnification_api(true);  //设置过滤窗口选项
    // 使用 DesktopAndCursorComposer 可以采集鼠标
    std::unique_ptr<webrtc::DesktopCapturer> capturer =
        std::unique_ptr<webrtc::DesktopCapturer>(
            new webrtc::DesktopAndCursorComposer(
                webrtc::DesktopCapturer::CreateScreenCapturer(opts), opts));

    // 设置开始采集状态
    capturer->Start(this);

    // 设置要过滤的窗口
    for (auto source : _excludeWindowList) {
      capturer->SetExcludedWindow(source.id);
    }

    while (_isrunning) {
      webrtc::SleepMs(_msPerFrame);
      // 采集桌面图像
      capturer->CaptureFrame();
    }
  }

  bool Start() {
    _isrunning = true;
    _capture_thread = rtc::Thread::Create();
    _capture_thread->Start();
    _capture_thread->PostTask(RTC_FROM_HERE, [&] { CaptureThread(); });
  }

  void Stop() {
    if (_isrunning) {
      _isrunning = false;
      _capture_thread->Stop();
      _capture_thread.reset();
    }
  }

 public:
  int _msPerFrame = 100;  // 100毫秒采集一次，每秒钟采集10帧
  webrtc::DesktopCapturer::SourceList _excludeWindowList;  //需要过滤的窗口列表
  desktop_capture_frame_callback _frameCallback = nullptr;  //视频输出回调
  void* _userContext = nullptr;

 private:
  std::unique_ptr<rtc::Thread> _capture_thread;
  bool _isrunning = false;
};

#endif

to sum up

If you clarify the context of webrtc screen sharing, the whole process is not too complicated. However, if you don't have an overview of webrtc's screen sharing at the beginning, you rush to the pit step by step, and you will get stuck somewhere and you will not be able to move forward. I hope this article is helpful to those who are preparing to apply webrc screen sharing.

Rongyun RCRTC C++ SDK has realized the screen sharing function of Windows, and has dealt with the details of many product requirements. Interested students can follow our SDK and Rongyun.

I wish you all the best in your work!