Kurento combat four: application development guide

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: Classification and summary of all original articles and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

Overview of this article

• This article is the fourth article of "Kurento Actual Combat". In the previous article, we first deploy KMS and then start the official demo. We have also learned all the important concepts of Kurento, and then we will start application development;
• The main content of this article is to analyze the official <font color="blue">kurento-hello-world</font> project to understand the basic process and knowledge points of Kurento application development. The code used in this article is the officially released version 6.15.0 , Address: https://github.com/Kurento/kurento-tutorial-java/archive/6.15.0.zip
• When reading the code, if you can clearly divide the functional modules as a whole, and then target the details one by one, you will learn and understand the source code more efficiently. Next, we will follow the official Kurento standard routine to split and break them one by one. ；

How to divide functional modules

According to different responsibilities, the entire code is split into three parts:

1. WebSocket related: WebSocket related general processing, such as connection establishment, closing, abnormal callback, business logic distribution, etc.;
2. WebRTC signaling related: ICE, SDP related processing;
3. Business logic: If 1 and 2 represent the general processing of WebRTC, then the rest is how to use Kurento to achieve business requirements. The main content of this part is that business applications use Kurento's official client to interact with KMS, and control KMS as the end The side provides services, and the interaction method is as shown in the figure below:
   

• Follow the above method to split the code and delimit the boundary. Whether you are reading the official demo or developing your own application, you can deal with it clearly and clearly. Next, learn the official hello-world source code together to see if a complete Kurento application is How it was developed

WebSocket related

The simplest logic should be the general WebSocket processing. Let's look at this part first, and the complicated ones will be discussed later. The logic related to WebSockert in the Handler class is as follows:

1. Inherited from TextWebSocketHandler (only process text type data, for binary data, close the session directly);
2. Rewrite afterConnectionEstablished: the callback of WebSocket connection establishment, only one line of log is printed;
3. Override handleTransportError: callback when WebSocket exception occurs, only close WebSocketSession;
4. Override afterConnectionClosed: This method will be executed regardless of whether the WebSocket is closed normally or an exception occurs. The logic is also very simple. It is to call the stop method. This method is used to release KMS resources. It will be called in several places. We will save it for later and Talk about other places dealing with KMS;
5. The most important part of the WebSockert code is the handleTextMessage method, which is the processing logic when receiving front-end data: first convert the data to a JsonObject object, the messageId field of this object has four values, each id and its corresponding processing method are as follows The table shows:

messageId	Approach	Description
PROCESS_SDP_OFFER	handleProcessSdpOffer	Processing logic after receiving front-end SDPOffer data
ADD_ICE_CANDIDATE	handleAddIceCandidate	Processing logic after receiving front-end ICE data
STOP	handleStop	HashMap deletes user data, and then calls MediaPipeline.release remotely
ERROR	handleError	HashMap deletes user data, and then calls MediaPipeline.release remotely

6. Not all applications need to rewrite all the code, or decide whether to rewrite based on actual needs. Take the <font color="blue">kurento-one2one-call</font> project as an example, only rewrite handleTextMessage and afterConnectionClosed, others can use the parent class, as shown below:

7. There is also a sendMessage method to send a message to the browser side, and a sendError method to send an error message;

Signaling related

• The function of the <font color="blue">kurento-hello-world</font> application is to realize real-time audio and video communication with KMS. Therefore, the signaling processing of the WebRTC standard is indispensable. Unfortunately, the official Kurento does not Too much encapsulation of the processing (it may be that the signaling and different business processing logic are different, resulting in poor abstraction), and the result is a bunch of signaling processing codes scattered in the business code;
• Even if the business and signaling processing codes appear in the Handler class at the same time, as long as you are familiar with the WebRTC signaling processing flow, it is easy to read the code. The following figure combines the WebRTC standard signaling processing flow, connecting the front-end and server-side codes in series Analyze together, the left side is the js code executed on the browser, and the right side is the server side. These codes are marked with red arrows in the specific position of the WebRTC signaling processing flow. So far, the entire flow is clearly displayed:

• If you look at the above picture on your computer or mobile phone and feel vague, please download the original file and open it with <font color="blue">draw.io</font>. The directory where the file is located is: https://github.com/ zq2599/blog_demos/tree/master/files , the file name is <font color="red">helloworld-flow.drawio</font>

• The above figure lists all the codes related to signaling. After reading these, the rest is the business code, which is the <font color="blue">handleProcessSdpOffer</font> method in the purple part of the figure;

  Business related

• The <font color="blue">kurento-hello-world</font> application transmits the local camera and microphone data to KMS, and then obtains the data from KMS and displays it on the page. First, let’s take a look at how the official KMS pipeline is described:

• It can be seen from the above figure that the pipeline logic is very simple: there is only one WebRtcEndpoint, and you can connect your Src and Sink to complete it. Let's take a look at the corresponding code, in the method handleProcessSdpOffer:

    // 创建pipeline
    final MediaPipeline pipeline = kurento.createMediaPipeline();
    user.setMediaPipeline(pipeline);

    // 创建webRtcEndpoint
    final WebRtcEndpoint webRtcEp =
        new WebRtcEndpoint.Builder(pipeline).build();
    user.setWebRtcEndpoint(webRtcEp);

    // 自己的sink连接上自己的src
    webRtcEp.connect(webRtcEp);

    // ---- Endpoint configuration

    String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
    
    // 注册各类监听，例如媒体资源状态变化、ICE变化等
    // 通过websocket回复SDP Offer
    initWebRtcEndpoint(session, webRtcEp, sdpOffer);

    log.info("[Handler::handleStart] New WebRtcEndpoint: {}",
        webRtcEp.getName());
    
    // ---- Endpoint startup
    // 取得ICE信息
    startWebRtcEndpoint(webRtcEp);

• Let's take a look at the stop method to stop WebRtc. In fact, it sends a release command to KMS:

  private void stop(final WebSocketSession session) {
    // Remove the user session and release all resources
    final UserSession user = users.remove(session.getId());
    if (user != null) {
      MediaPipeline mediaPipeline = user.getMediaPipeline();
      if (mediaPipeline != null) {
        log.info("[Handler::stop] Release the Media Pipeline");
        mediaPipeline.release();
      }
    }
  }

summary

The above is the source code analysis of the entire <font color="blue">kurento-hello-world</font>. When the code of the entire project is split and then analyzed, it becomes very clear and simple:

1. WebSocket is no different from conventional java development, just move closer to the standard;
2. WebRTC related code accounts for a large proportion, but it strictly follows the standard signaling process. As long as you are familiar with WebRTC, it is easy to read and understand;
3. Business logic is actually related to business requirements. Here you need to be familiar with the capabilities provided by KMS in order to give full play to the examples of KMS. The pipeline layout and the use of each element will also be the focus of our subsequent articles. Make good use of these elements and polish Provide more powerful and flexible services;

You are not lonely, Xinchen is with you all the way

1. Java series
2. Spring series
3. Docker series
4. kubernetes series
5. Database + Middleware Series
6. DevOps series

Welcome to pay attention to the public account: programmer Xin Chen

Search for "Programmer Xin Chen" on WeChat, I am Xin Chen, and I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos