Audio and video synchronization! RTCP protocol analysis and code implementation

RTCP is the abbreviation of Real-Time Control Protocol. RTCP is defined by RFC 3550 (replaces the obsolete RFC 1889).

The real-time transport protocol (RTP) and the real-time control protocol (RTCP) are used in combination to monitor the data transmission of large-scale multicast networks. RTP carries media streams, while RTCP is used for monitor transmission statistics and service quality . Monitoring enables the receiver to detect any packet loss and compensate for any delay jitter.

Both protocols work independently of the basic transport layer protocol and network layer protocol. The information in the RTP header tells the receiver how to reconstruct the data and describes how the codec bitstream is packed.

Below we focus on the RTCP function, RTCP packet, etc. Finally, RTCP protocol analysis is realized.

RTCP What are the features

1. The main function of RTCP is to provide quality feedback data distribution. This is an indispensable part of the role of RTP, the transport protocol, and the control function of other transport protocols related to traffic and congestion.

2. RTCP The persistent transport level identifier source with RTP is called the canonical name or CNAME. Since if a conflict is found or the program is restarted, the recipient asks CNAME to track each participant.

The recipient may also ask the CNAME to associate multiple data streams from a given participant to the number of related RTP sessions in the collection, such as synchronized audio and video. The inter-media synchronization also requires the time stamp contained in the RTCP packet by the sender of NTP and RTP data.

3. The first two functions require all participants to send RTCP packets, so the rate must be controlled so that RTP can be expanded to a large number of participants. By having each participant send its control packet to all others, everyone can independently observe the number of participants.

4. This function is very useful for loose conversation process in which participants can enter and leave at will. Participants can enter or leave freely without member control or parameter coordination.

Functions 1-3 should be used in all environments, especially in IP multicast environments. RTP application designers should avoid larger numbers of mechanisms that can only work in unicast mode and cannot be extended to. The transmission of RTCP can be controlled separately for the sender and the receiver, which is suitable for, for example, a one-way link, and the receiver has no possibility of feedback.

RTCP protocol port

RTSP usually uses the RTP protocol to transmit real-time streams, RTP generally uses even-numbered ports, and RTCP uses adjacent odd-numbered ports, that is, RTP port number +1.

RTP port

RTCP port

What are the RTCP packets

In RTCP communication control, the functions of the RTCP protocol are realized through different types of RTCP packets. RTCP is also transmitted based on UDP packets, and there are mainly five types of packets:

1. SR: The sender report, sent by the application program or the middle end that sends the RTP datagram.
2. RR: Receiver report, sent by an application or mid-end that accepts but does not send RTP datagrams.
3. SDES: Source description, a carrier that transmits identification information related to session members, such as user names, emails, and phone numbers.
4. BYE: Notify to leave, notify other members in the reply that they will exit the conversation.
5. APP: Defined by the application itself, as an extension of the RTCP protocol.

#define RTCP_SR      200
#define RTCP_RR      201
#define RTCP_SDES    202
#define RTCP_BYE     203
#define RTCP_APP     204

We can judge the RTCP header based on these five types of packets

static bool dissect_rtcp_heur(u_char *rtcp_info,int PayloadLen)
{
    unsigned int offset = 0;
    unsigned int first_byte = 0;
    unsigned int packet_type = 0;


    /* 查看第一个字节 */
    first_byte = rtcp_info[offset];

    /* 版本位是否设置为2*/
    //printf("version: %d\n",((first_byte & 0xC0) >> 6));
    if (((first_byte & 0xC0) >> 6) != 2)
    {
        return false;
    }

    /* 看包类型 */
    offset += 1;
    packet_type = rtcp_info[offset];
    //printf("packet_type: %d\n",packet_type);
    /* 复合数据包中的第一个数据包应该是发送方或接收者报告 */
    if (!((packet_type == RTCP_SR)  || (packet_type == RTCP_RR) ||
          (packet_type == RTCP_BYE) || (packet_type == RTCP_APP) ||
          (packet_type == RTCP_PSFB)))
    {
        return false;
    }

    /*总长度必须是4个字节的倍数*/
    //printf("PayloadLen: %d\n",PayloadLen);
    if (PayloadLen % 4)
    {
        return false;
    }

    /* OK, dissect as RTCP */
    dissect_rtcp(rtcp_info,packet_type,offset,PayloadLen);
    return true;
}

RTCP protocol message format

SR: sender report

version (V) : same as RTP packet header

padding (P) : same as RTP packet header.

Receive Report Counter (RC) : 5b The number of report blocks received in this SR packet.

packet type (PT) : 8bit SR packet type is 200

length (length) : the length of the SR packet with 32bit as 1 unit minus 1

Synchronization Source (SSRC) : The synchronization source identifier sent by the SR packet. It is the same as the SSRC in the corresponding RTP packet.

NTP timestamp (Network Time Protocol) : The absolute time when the SR packet was sent. Used to synchronize different streams.

RTP timestamp : corresponds to the NTP timestamp, and has the same initial value as the timestamp in the RTP packet.

Send's Packet count : The total number of bytes of valid data sent by the sender during the period from the start of sending the packet to the generation of this SR packet, excluding the header and padding. When the sender changes the SSRC, this field must be cleared.

Synchronization source n's SSRC identifier : This report contains statistics about the packets received from this source.

Loss rate : Indicates the loss rate of the RTP packet sent from the last SR or RR packet according to the synchronization source n.

Cumulative lost data : The total number of lost RTPs sent by SSRC_n during the period from the beginning of receiving SSRC_n packets to sending SR.

Extended maximum sequence number : The maximum sequence number in the RTP packet received from SSRC_n.

jitter (Interarrival jitter) : The statistical variance estimation of the RTP packet receiving time.

Last SR timestamp (Last SR) : Take the middle 32 bits of the NTP timestamp in the SR packet recently received from SSRC_n. If the SR packet has not been received, it is 0.

depends on the SR delay last time (Delay since Last SR) : The delay from the last time SSRC_n received the SR packet to the time it was sent

Wireshark captures packets:

Participants in an active session use SR when sending and receiving RTP packets. SR has three different parts: header information, sender information, and many receiver report blocks. SR can also have an extended field related to the outline.

Wireshark captures packets:

SDES: Source description

SDES provides a carrier that transmits identification information related to session members, such as user names, emails, and phone numbers. There must be an SDES packet in each RTCP mixed packet.

The header contains a length field, a payload type field (PT=202), and a source count (RC) field. The 5 bits in the RC field indicate the number of information blocks in the packet.

Each information block contains an SSRC or CSRC value, followed by one or more identifiers and some information that can be used for SSRC or CSRC.

The SDES packet of the CNAME item must be included in each combined RTCP packet. The SDES package may include other source description items, which should be based on special application requirements and also consider bandwidth limitations.

Wireshark captures packets:

The SDES source description package provides intuitive text information to describe the participants of the session, including source description items such as CNAME, NAME, EMAIL, PHONE, and LOC.

These provide convenience for the receiver to obtain the relevant information of the sender. The SDES packet is composed of a packet header and a data block. There can be no data blocks or multiple data blocks. The packet header is composed of version (V), padding (P), length indication, packet type (PT) and source count (SC).

PT occupies 8 bits and is used to identify the SDES packet of RTCP. SC occupies 5 bits and indicates the number of SSRC/CSRC blocks contained in the SDES packet. The zero value is valid, but has no meaning.

BYE: Notice to leave

BYE grouping is used to indicate that one or more media sources are no longer active.

Wireshark captures packets:

As an option, the BYE packet can include an 8-digit octal count, followed by a text message to indicate the reason for leaving.

Finally, each RTCP packet in the combined packet can be processed independently, and does not need to be processed in the order of packet combination.

There are several mandatory constraints in the package.

1. As long as the bandwidth allows, the reception statistics in the SR packet or the RR packet should be sent frequently, so the combined RTCP packet sent in each cycle should include the report packet.
2. SDES CNAME should be included in each package, because the new recipient needs to identify the source by receiving the CNAME and contact the media for synchronization.
3. In front of the combination package is the number of package types, whose growth should be limited.

RTCP protocol realize the synchronization of media streams

Through packet capture and analysis of the RTCP sender report, the synchronization of RTP actually relies on these three domains:

1. sender SSRC: The synchronization source identifier sent by the SR packet. It is the same as the SSRC in the corresponding RTP packet.
2. NTP timestamp: The absolute time when the SR packet was sent. Used to synchronize different streams.
3. RTP timestamp: Corresponds to the NTP timestamp, and has the same initial value as the timestamp in the RTP packet.

How to calculate NTP time? In RTCP, the NTP time is stored in 8 bytes, divided into: MSW and LSW, each occupying 4 bytes.

const char *tvb_ntp_fmt_ts_sec(u_char *rtcp_info, int offset)
{
    uint32_t tempstmp = 0;
    time_t temptime = 0;
    struct tm *bd;
    char *buff = NULL;
    
    tempstmp = ntohl(*(uint32_t*)(rtcp_info + offset));
    if (tempstmp == 0){
        return "NULL";
    }

    /* We need a temporary variable here so the unsigned math
    * works correctly (for years > 2036 according to RFC 2030
    * chapter 3).
    */
    temptime = (time_t)(tempstmp - NTP_BASETIME);
    bd = gmtime(&temptime);
    if (!bd){
        return "Not representable";
    }

    buff = (char *)malloc(NTP_TS_SIZE);
    snprintf(buff, NTP_TS_SIZE,
        "%s %2d, %d %02d:%02d:%02d UTC",
        mon_names[bd->tm_mon],
        bd->tm_mday,
        bd->tm_year + 1900,
        bd->tm_hour,
        bd->tm_min,
        bd->tm_sec);
    return buff;
}

NTP timestamp

  
    /* NTP timestamp */
    ts_msw = ntohl(*(uint32_t*)(rtcp_info + offset));
    printf("ts_msw: 0x%x\n",ts_msw);
    ts_lsw = ntohl(*(uint32_t*)(rtcp_info + offset + 4));
    printf("ts_lsw: 0x%x\n",ts_lsw);
    printf("MSW: %s\n",tvb_ntp_fmt_ts_sec(rtcp_info,offset));
    offset += 8;

RTCP protocol implementation

Below I give the code that implements the analysis of the RTCP protocol, and parse the field information according to the playback message.

/* 接收者/发送者计数是最后5位   */
#define RTCP_COUNT(octet)   ((octet) & 0x1F)


#define RTCP_PT_MIN  192
/* Supplemental H.261 specific RTCP packet types according to Section C.3.5 */
#define RTCP_FIR     192
#define RTCP_NACK    193
#define RTCP_SMPTETC 194
#define RTCP_IJ      195
/* RTCP packet types according to Section A.11.1 */
/* And https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml */
#define RTCP_SR      200
#define RTCP_RR      201
#define RTCP_SDES    202
#define RTCP_BYE     203
#define RTCP_APP     204
#define RTCP_RTPFB   205
#define RTCP_PSFB    206
#define RTCP_XR      207
#define RTCP_AVB     208
#define RTCP_RSI     209
#define RTCP_TOKEN   210

#define RTCP_PT_MAX  210


static const char mon_names[12][4] = {
    "Jan",
    "Feb",
    "Mar",
    "Apr",
    "May",
    "Jun",
    "Jul",
    "Aug",
    "Sep",
    "Oct",
    "Nov",
    "Dec"
};

/** data structure to hold time values with nanosecond resolution*/
typedef struct {
    time_t    secs;
    int    nsecs;
} nstime_t;


/*
 * 1900-01-01 00:00:00 (proleptic?) UTC.
 * Used by a number of time formats.
 */
#define EPOCH_DELTA_1900_01_01_00_00_00_UTC 2208988800U

/*
 * NTP_BASETIME is in fact epoch - ntp_start_time; ntp_start_time
 * is January 1, 2036, 00:00:00 UTC.
 */
#define NTP_BASETIME EPOCH_DELTA_1900_01_01_00_00_00_UTC
#define NTP_FLOAT_DENOM 4294967296.0
#define NTP_TS_SIZE 100


/* 解剖长度字段。附加到此字段的文字表示转换为的实际字节数 (即 (原始值 + 1) * 4) */
static int dissect_rtcp_length_field(u_char *rtcp_info, int offset)
{

    uint16_t  raw_length = ntohs(*(uint16_t*)(rtcp_info + offset));
    printf("(%u bytes)\n", (raw_length+1)*4);
    offset += 2;
    return offset;
}
static int dissect_rtcp_rr(u_char *rtcp_info, int offset,int count, int packet_length )
{
    int counter = 0;
    uint8_t  rr_flt = 0;
    int    rr_offset = offset;
    
    counter = 1;
    while ( counter <= count ) {
        uint32_t lsr = 0, dlsr = 0;

        /* Create a new subtree for a length of 24 bytes */

        /* SSRC_n source identifier, 32 bits */

        offset += 4;

        /* Fraction lost, 8bits */
        rr_flt = rtcp_info[offset];

        offset++;

        /* Cumulative number of packets lost, 24 bits */
        offset += 3;


        /* Sequence number cycles */

        offset += 2;
        /* highest sequence number received */

        offset += 2;

        /* Interarrival jitter */

        offset += 4;

        /* Last SR timestamp */
        lsr = ntohl(*(uint32_t*)(rtcp_info + offset));
        printf("Last SR timestamp: 0x%x\n",lsr);
        offset += 4;

        /* Delay since last SR timestamp */
        dlsr = ntohl(*(uint32_t*)(rtcp_info + offset));

        printf("(%d milliseconds)\n",(int)(((double)dlsr/(double)65536) * 1000.0));
        offset += 4;

        counter++;
    }

    return offset;
}

const char *tvb_ntp_fmt_ts_sec(u_char *rtcp_info, int offset)
{
    uint32_t tempstmp = 0;
    time_t temptime = 0;
    struct tm *bd;
    char *buff = NULL;
    
    tempstmp = ntohl(*(uint32_t*)(rtcp_info + offset));
    if (tempstmp == 0){
        return "NULL";
    }

    /* We need a temporary variable here so the unsigned math
    * works correctly (for years > 2036 according to RFC 2030
    * chapter 3).
    */
    temptime = (time_t)(tempstmp - NTP_BASETIME);
    bd = gmtime(&temptime);
    if (!bd){
        return "Not representable";
    }

    buff = (char *)malloc(NTP_TS_SIZE);
    snprintf(buff, NTP_TS_SIZE,
        "%s %2d, %d %02d:%02d:%02d UTC",
        mon_names[bd->tm_mon],
        bd->tm_mday,
        bd->tm_year + 1900,
        bd->tm_hour,
        bd->tm_min,
        bd->tm_sec);
    return buff;
}

static int dissect_rtcp_sr(u_char *rtcp_info, int offset,int count, int packet_length)
{

    uint32_t     ts_msw = 0, ts_lsw = 0;
    int         sr_offset = offset;

    /* NTP timestamp */
    ts_msw = ntohl(*(uint32_t*)(rtcp_info + offset));
    printf("ts_msw: 0x%x\n",ts_msw);
    ts_lsw = ntohl(*(uint32_t*)(rtcp_info + offset + 4));
    printf("ts_lsw: 0x%x\n",ts_lsw);

    //printf("offset: 0x%x 0x%x 0x%x 0x%x\n",rtcp_info[offset],rtcp_info[offset + 1],rtcp_info[offset + 2],rtcp_info[offset + 3]);
    printf("MSW: %s\n",tvb_ntp_fmt_ts_sec(rtcp_info,offset));
    offset += 8;

    /* RTP timestamp, 32 bits */
    
    offset += 4;
    /* Sender's packet count, 32 bits */

    offset += 4;
    /* Sender's octet count, 32 bits */

    offset += 4;


    /* The rest of the packet is equal to the RR packet */
    if ( count != 0 )
        offset = dissect_rtcp_rr(rtcp_info, offset, count, packet_length-(offset-sr_offset));
    else
    {
        /* If length remaining, assume profile-specific extension bytes */
        if ((offset-sr_offset) < packet_length)
        {

            offset = sr_offset + packet_length;
        }
    }

    return offset;
}

static int dissect_rtcp_sdes(u_char *rtcp_info, int offset, int count)
{
    int           chunk = 0;

    int           start_offset = 0;
    int           items_start_offset = 0;
    uint32_t       ssrc = 0;
    unsigned int  item_len = 0;
    unsigned int  sdes_type = 0;
    unsigned int  prefix_len = 0;

    chunk = 1;
    while ( chunk <= count ) 
    {
        /* Create a subtree for this chunk; we don't yet know
           the length. */
        start_offset = offset;

        ssrc = ntohl(*(uint32_t*)(rtcp_info + offset));
        printf("Chunk %u, SSRC/CSRC 0x%X\n", chunk, ssrc);

        /* SSRC_n source identifier, 32 bits */
        offset += 4;

        /* Create a subtree for the SDES items; we don't yet know
           the length */


        /*
         * Not every message is ended with "null" bytes, so check for
         * end of frame as well.
         */
         
          /* ID, 8 bits */
        sdes_type = rtcp_info[offset];
        printf("Type: %d\n",sdes_type);
        if (sdes_type == 0)
            break;
        offset++;
        /* Item length, 8 bits */
        item_len = rtcp_info[offset];
        printf("Length: %d\n",item_len);
        offset++;
        
        char *pszText = (char*)malloc(item_len);
        if (pszText != 0)
        {
            memcpy(pszText, rtcp_info + offset,item_len);
            pszText[item_len] = '\0';
            printf("Text = %s\n",pszText);
        }    

        chunk++;
    }

    return offset;
}

static void dissect_rtcp(u_char *rtcp_info,int packet_type, int offset,int PayloadLen)
{
    unsigned int  temp_byte = 0;
    int  elem_count = 0;
    int  packet_length = 0;
    int  total_packet_length = 0;
    int loop = 2;
    bool flag_rtcp = false;

        /*检查是否为有效类型*/
        if ( ( packet_type < RTCP_PT_MIN ) || ( packet_type >  RTCP_PT_MAX ) )
            exit(-1);

        /*
         * 获取完整的RTCP数据包的长度
         */
         
        packet_length = (ntohs(*(uint16_t*)(rtcp_info + offset + 1)) + 1) * 4 ;
        //printf("packet_length: %d\n",packet_length);

        
        temp_byte = rtcp_info[offset-1];
        elem_count = RTCP_COUNT( temp_byte );/* Source count, 5 bits */
        printf("Reception report count: %d\n",elem_count);  

        switch ( packet_type ) 
        {
            
            case RTCP_SR:
            case RTCP_RR:
                /*
                    Real-time Transport Control Protocol (Receiver Report)
                    10.. .... = Version: RFC 1889 Version (2)
                    ..0. .... = Padding: False
                    ...0 0001 = Reception report count: 1
                    Packet type: Receiver Report (201)
                    Length: 7 (32 bytes)
                    Sender SSRC: 0xb584b03e (3045371966)
                    Source 1
                */

                /* Packet type, 8 bits */
                offset++;
                /* Packet length in 32 bit words MINUS one, 16 bits */
                offset = dissect_rtcp_length_field(rtcp_info, offset);
                /* Sender Synchronization source, 32 bits */
                offset += 4;

                if ( packet_type == RTCP_SR )
                {
                    offset = dissect_rtcp_sr(rtcp_info, offset, elem_count, packet_length-8 );
                    printf("dissect_rtcp_sr\n");
                }
                else
                {    
                    offset = dissect_rtcp_rr(rtcp_info, offset, elem_count, packet_length-8 );                            
                }
                
                //uint16_t second_packet_type = ntohs(*(uint16_t*)(rtcp_info + offset));

                //printf("111offset: 0x%x 0x%x 0x%x 0x%x\n",rtcp_info[offset],rtcp_info[offset + 1],rtcp_info[offset + 2],rtcp_info[offset + 3]);
                
                if (rtcp_info[offset + 1] == RTCP_SDES)
                {
                    
                    /* Source count, 5 bits */
                    offset++;
                    /* Packet type, 8 bits */
                    offset++;
                    /* Packet length in 32 bit words MINUS one, 16 bits */
                    offset = dissect_rtcp_length_field(rtcp_info, offset);
                    offset = dissect_rtcp_sdes(rtcp_info,offset,elem_count);

                }

            break;

            default:
                /*
                 * To prevent endless loops in case of an unknown message type
                 * increase offset. Some time the while will end :-)
                 */
                offset++;
                break;

        }
        
}


static bool dissect_rtcp_heur(u_char *rtcp_info,int PayloadLen)
{
    unsigned int offset = 0;
    unsigned int first_byte = 0;
    unsigned int packet_type = 0;


    /* 查看第一个字节 */
    first_byte = rtcp_info[offset];

    /* 版本位是否设置为2*/
    //printf("version: %d\n",((first_byte & 0xC0) >> 6));
    if (((first_byte & 0xC0) >> 6) != 2)
    {
        return false;
    }

    /* 看包类型 */
    offset += 1;
    packet_type = rtcp_info[offset];
    //printf("packet_type: %d\n",packet_type);
    /* 复合数据包中的第一个数据包应该是发送方或接收者报告 */
    if (!((packet_type == RTCP_SR)  || (packet_type == RTCP_RR) ||
          (packet_type == RTCP_BYE) || (packet_type == RTCP_APP) ||
          (packet_type == RTCP_PSFB)))
    {
        return false;
    }

    /*总长度必须是4个字节的倍数*/
    //printf("PayloadLen: %d\n",PayloadLen);
    if (PayloadLen % 4)
    {
        return false;
    }

    /* OK, dissect as RTCP */
    dissect_rtcp(rtcp_info,packet_type,offset,PayloadLen);
    return true;
}


static void confirm_rtcp_packet(struct ip *pIp)
{
    int iIpTotalLen = ntohs(pIp->ip_len);
    int offset = 0;
    int nFragSeq = 0;
    struct udphdr* pUdpHdr = (struct udphdr*)((char*)pIp + (pIp->ip_hl<<2));
    if (pIp->ip_p == IPPROTO_UDP) 
    {
        printf("\n");
        
        int iPayloadLen = iIpTotalLen - (pIp->ip_hl<<2) - 8;
        printf("UDP Payload Len %d\n", iPayloadLen);
        
        u_char *pDnsHdr = (u_char*)(pUdpHdr+1);
        dissect_rtcp_heur(pDnsHdr,iPayloadLen);
        
    }    
}

Compile and run:

summary

The RTCP protocol is the cornerstone of streaming media communication. The RTCP protocol is responsible for service quality assurance such as reliable transmission, flow control, and congestion control. The RTCP function, RTCP packet format and code implementation are explained above. Finally, to learn a new protocol, it is best to study the official documents, because this is the most authoritative information.