New App User Experience-Agora Native SDK 3.4.0

Agora Native SDK 3.4.0 has been officially launched this month. The new version not only adds richer real-time beauty sound effects, screen sharing, virtual metronome and other functions, but also greatly upgrades the stability, compatibility and security compliance of the SDK, hoping to bring smoother, Comfortable interactive new experience.

What is updated in version 3.4.0?

feature added

1. Sound effect file playback progress

In the process of real-time audio and video interaction, when we want to better enhance the atmosphere and add interest, we usually choose to play sound effects (for example, adding fighting sounds in the game, adding accompaniment when singing, etc.) for interaction. But sometimes if you need to adjust the playback position after playing the sound effect file, only play a certain section of a complete audio track alone or want to quickly skip a certain part of the audio content, version 3.4.0 provides control of the playback progress of the sound effect file. The following methods are specifically added:

• playEffect3 : Set the playback position when playing the sound effect file through the startPos
• setEffectPosition : After playing the sound effect file, set the playback position of the sound effect file.
• getEffectDuration : Get the total duration of the local sound effect file.
• getEffectCurrentPosition : Get the playback progress of the sound effect file.

2. Virtual metronome

In order to meet the needs of metronome in online music teaching and other scenes, version 3.4.0 adds the following methods:

• startRhythmPlayer : Turn on the virtual metronome.
• stopRhythmPlayer : Turn off the virtual metronome.
• configRhythmPlayer : After turning on the virtual metronome, reconfigure the virtual metronome.

Function optimization

1. Video fluency and quality balance under weak networks

When the bandwidth is limited, in order to ensure the fluency of the video, usually video coding will choose to reduce the video frame rate first to maintain the video quality, or choose to reduce the video quality to ensure the video frame rate. However, in the scenes of one-to-one calls, one-to-one teaching, or multi-person meetings, it is actually difficult to give up picture quality and fluency. Therefore, in version 3.4.0, we optimized the degradationPreference add support to AgoraDegradationBalanced , which will reduce the video frame rate and video quality under a weak network, in order to strike a balance between fluency and video quality.

2. Original video data (C++)

In order to facilitate developers to obtain the original video data of each stage of transmission and meet the needs of more scenarios, we have supported C++ callbacks getRotationApplied and getMirrorApplied in versions prior to 3.4.0 to rotate and mirror the original video data. In order to improve the user experience, the new version expands the video data format supported by these callback functions from RGBA to RGBA and YUV 420.

3. Screen sharing function optimization

Screen sharing in a real-time interactive scene refers to a technology that shares content on the screen to realize information sharing. Such applications are common in game live broadcasts, video conferences, or online education scenarios (screen sharing of files, data, web pages, courseware, notes, etc.). Users can share their mobile or PC screen content with others for real-time viewing.

The new version of the SDK currently supports screen area sharing, window area sharing, full-screen sharing, sharing front, window minimized callback, and other functions. It also supports the choice of fluency first mode or definition first mode. The usability and experience of screen sharing have been greatly improved.

4. Client recording

To help users more easily and flexibly set the recording while recording the content, the new version startAudioRecordingWithConfig method, by startAudioRecordingWithConfig the config parameters, the user can choose the set recording quality, the content of the path memory, the sampling rate and audio files.

At the same time, this version also adds error code AgoraErrorCodeAlreadyInRecording(160) . startAudioRecordingWithConfig again before the music file is played, the SDK will report the error code.

Why is it currently the most "excellent" version?

Agora Native SDK 3.4.0 is an integrated version with richer functions and higher stability. In order for users to have a better experience in the new version. We made a lot of optimizations from the 3.0 SDK to the 3.4 SDK, and made corresponding improvements in audio processing, video processing, delay optimization, and security compliance. Therefore, the 3.4 version not only adds and optimizes some functions, but also retains and optimizes the excellent functions and features of the previous version.

3A algorithm + AI efficient noise reduction

Whether in a video interview or a video conference scene, we have all encountered situations where we temporarily suspend the interaction between the two parties or rejoin the video room due to noise, echo, etc.

Through the 3A (AEC, ANC, AGC) algorithm of the sound network, it can realize intelligent adaptation to various environments, comprehensively eliminate echo, and provide super-first-class dual-talk performance; it can effectively eliminate all kinds of noise without damaging the voice quality; It can realize automatic audio gain, and users can have a good real-time interactive experience even in a noisy environment.

At the same time, the sound network SDK presets a signal classification module at the front end of the noise reduction module, which can accurately detect the type of signal, and adjust the type and parameters of the noise reduction algorithm according to the type of the signal. Common signal types include general voice, a cappella , Music signal, etc. Through the sound network AI noise reduction algorithm, the use of deep learning technology to process real-time audio through feature extraction, neural network and gain adjustment to suppress noise, breaking through the performance bottleneck of traditional signal processing solutions, thereby providing a clear voice environment for real-time audio and video. It can achieve a good noise reduction effect, and solve the noise reduction problem of real-time separation of human voice and noise while ensuring the fidelity of human voice.

who want to learn more about this part of the content, please refer to " Detailed Explanation of Low Latency and High Sound Quality: Echo Cancellation and Noise Reduction "

2. Richer beautiful sound effects

Real-time sound effect refers to adding a certain style to the sound, enhancing the sense of hierarchy and space of the sound. In order to improve the user’s audio experience, we have moved traditional professional sound cards, tuner, and 10,000-yuan offline plug-ins online through software algorithms. Users can use Bel Canto special effects, virtual stereo, and beautify the sound without having to download plug-ins. Sound effects such as, automatic voice change, etc., while enhancing user experience and scene entertainment, greatly reduce the threshold of use.

At present, it has realized 3 kinds of bel cantos, 6 kinds of bel cantos, 8 kinds of timbre changes, 18 kinds of human voice effects, 7 kinds of variable voice effects, and 4 kinds of basic voice change effects.

Demo audition: https://www.agora.io/cn/audio-demo you want to understand the principle of real-time , you can read our previous share . For the specific usage of the interface, you can visit the document center of Soundnet, search and view the advanced guide "Voice Change and Reverberation".

3. Beauty optimization-both "natural" and "beauty"

In social entertainment or education scenarios, users often want to show the other party a good state and mental outlook when making a video call or live broadcast. By focusing on areas such as portrait skin, hair, and background, developers can easily achieve more realistic and "natural" whitening and rosy effects. Developers can also customize the beauty effects by adjusting parameters.

Even in extreme low light conditions, by applying the self-developed dark light enhancement algorithm of Soundnet, it can adaptively adjust the brightness value of the dark area, restore and highlight the detailed information of the image, improve the visual effect of the video image, and let the video image have More natural high-quality picture quality. This function is suitable for scenes such as low overall brightness, backlight, cloudy sky, uneven lighting, etc. Dim light enhancement currently supports the iOS platform, and balances and tunes the use effect and device performance. It can support 720P@30fps real-time processing on iPhone 6s models and above.

% anti-packet loss ability, SDK crash rate "one in ten thousand"

In the optimization of real-time data transmission quality, based on the transmission of Soundnet SD-RTN™, there is no difference between SD-RTN™ and dedicated lines in terms of packet arrival rate, and the arrival rate of jitter 200ms is 99.9%. Stability has reached the level of dedicated line.

At the same time, in response to possible weak network transmission and packet loss in real-time voice interaction, Agora has a set of professional anti-weak network transmission and anti-packet loss algorithms. In the process of SDK iteration, we have also continuously optimized the anti-weak network indicators behind it. At present, the anti-weak network boundary of video has been increased from 60% to 70%, and audio has been increased to 80%. In such a weak network environment, a smooth audio and video experience can still be guaranteed. At the same time, according to data provided by users in the Americas, Asia, Southeast Asia, the Middle East and other regions, the first frame of the SDK and the sounding time have also been fully optimized, and the customer closed-loop verification results are completely consistent with the laboratory test results.

5. Security compliance

Acoustic Network follows internationally recognized information security and privacy protection standards and industry requirements, and is committed to adopting international best practices to build a privacy and security management system. At present, Shengwang has passed ISO/IEC 27001, ISO/IEC ISO27017, ISO/IEC 27018 system certification, and obtained a SOC2 Type I service certification report issued by a third-party professional organization to prove its privacy compliance and security management ability.

For more features and new functions of this new version, you may not be able to repeat them in this introduction. Click [here] get more SDK related information online.