Develop Android audio and video calls based on Linphone

1. Introduction to Linphone

1.1 Introduction

LinPhone is an open source Internet phone or Voice over IP (VOIP) system that follows the GPL protocol, and its main features are as follows. Using linphone, developers can communicate freely on the Internet, including voice, video, and instant text messages. Using SIP protocol, linphone is a standard open source VoIP system that can connect linphone with any SIP-based VoIP operator, including our own free SIP-based Audio/Video server.

LinPhone is a free software (or open source software), you can freely download and develop on the basis of LinPhone. LinPhone is available for Linux, Windows, MacOSX desktop computers as well as Android, iPhone, Blackberry mobile devices.

To learn the source code of LinPhone, open source starts from the following parts:
The SIP three-layer protocol architecture implemented by the Java layer framework: transport layer, transaction layer, syntax encoding and decoding layer;
SIP functions implemented by linphone dynamic library C source code: registration, request, request timeout, invite session, hang up, invite video, send and receive text messages...
The audio and video encoding and decoding functions implemented by the C source code of the linphone dynamic library;
Audio and video capture and playback functions on the Android platform;

1.2 Basic use

If you are an Android user, you can install it from the Google Play Store or download Linphone from this link. After the installation is complete, click the menu button in the upper left corner and select to enter the assistant interface. In the assistant interface, you can set a SIP account or a Linphone account, as shown below:

For us, to set up a SIP account, we need to fill in several parameters:

• Username: It is the SIP account number or name.
• Password: The password corresponding to the SIP account.
• Domain Name: Fill in the IP address or domain name of the SIP server (IPPBX).
• Display Name: The display name of the SIP account, which is optional.
• Transmission: The SIP server supports the transmission protocol, generally UDP, or TCP or TLS as required.

After the registration is successful, the softphone APP will have a prompt message, and the upper left corner will display the connection status, as shown in the figure below.

Then, enter the SIP account of the other party, and you can call, as shown below.

1.3 Related documents

The following are some materials that may be used in Linphone development:

• Linphone official website: http://www.linphone.org/technical-corner/liblinphone
• Official website documentation: https://wiki.linphone.org/xwiki/wiki/public/view/Lib/Getting%20started/Android/
• Official Android Demo: https://gitlab.linphone.org/BC/public/linphone-android
• Various versions of the aar library: https://linphone.org/releases/maven_repository/org/linphone/linphone-sdk-android/

2. Get started quickly

2.1 Compile the App

First, open the project with Android Studio, and then build/install the application, which may be slow during the compilation process. Of course, you can also use the command to compile:

 ./gradlew assembleDebug
//或者
./gradlew installDebug

2.2 Compile SDK

In the development of Android applications, there are source code dependencies and sdk dependencies in the way of introducing third-party libraries. Of course, we can also download the sdk code and then perform local compilation.

 git clone https://gitlab.linphone.org/BC/public/linphone-sdk.git --recursive

Then install the instructions in the official documentation to compile the sdk.

2.3 Integrate Linphone

First of all, you need to introduce linphone dependencies. You can directly download the aar package and execute it locally, or you can use gradle to import it. Here, we use the sdk that has been compiled by others:

 dependencies {
    //linphone
    debugImplementation "org.linphone:linphone-sdk-android-debug:5.0.0"
    releaseImplementation "org.linphone:linphone-sdk-android:5.0.0"
}

CoreManager

In order to facilitate the call, we need to simply encapsulate Linphone. First of all, according to the introduction of the official document, create a CoreManager class, which is the management class in the sdk, used to control the ringtone of the incoming call and start the CoreService, no need to call if there are no special requirements. It should be noted that the media package needs to be imported to start the ringtone, otherwise there will be no ringtone, as follows:

 implementation 'androidx.media:media:1.2.0'

Then, we create a new LinphoneManager class to manage the Linphone sdk, such as registering Linphone to the server, making voice calls, and so on.

 class LinphoneManager private constructor(private val context: Context) {
    
    ...  //省略其他代码

    /**
     * 注册到服务器
     *
     * @param username     账号名
     * @param password      密码
     * @param domain     IP地址：端口号
     */
    fun createProxyConfig(
        username: String,
        password: String,
        domain: String,
        type: TransportType? = TransportType.Udp
    ) {
        core.clearProxyConfig()
        val accountCreator = core.createAccountCreator(corePreferences.xmlRpcServerUrl)
        accountCreator.language = Locale.getDefault().language
        accountCreator.reset()
        accountCreator.username = username
        accountCreator.password = password
        accountCreator.domain = domain
        accountCreator.displayName = username
        accountCreator.transport = type
        accountCreator.createProxyConfig()
    }

    /**
     * 取消注册
     */
    fun removeInvalidProxyConfig() {
        core.clearProxyConfig()
    }

    /**
     * 拨打电话
     * @param to String
     * @param isVideoCall Boolean
     */
    fun startCall(to: String, isVideoCall: Boolean) {
        try {
            val addressToCall = core.interpretUrl(to)
            addressToCall?.displayName = to
            val params = core.createCallParams(null)
            //启用通话录音
//            params?.recordFile = LinphoneUtils.getRecordingFilePathForAddress(context, addressToCall!!)
            //启动低宽带模式
            if (LinphoneUtils.checkIfNetworkHasLowBandwidth(context)) {
                Log.w(TAG, "[Context] Enabling low bandwidth mode!")
                params?.enableLowBandwidth(true)
            }
            if (isVideoCall) {
                params?.enableVideo(true)
                core.enableVideoCapture(true)
                core.enableVideoDisplay(true)
            } else {
                params?.enableVideo(false)
            }
            if (params != null) {
                core.inviteAddressWithParams(addressToCall!!, params)
            } else {
                core.inviteAddress(addressToCall!!)
            }
        } catch (e: Exception) {
            e.printStackTrace()
        }
    }

    ... //省略其他代码
}

CoreService

Next is the CoreService class. The function of this class is a keep alive service. It will call the vibration method and start the notification when there is an incoming call, so it must be registered in AndroidManifest.xml.

 <service
   android:name="org.linphone.core.tools.service.CoreService"
   android:foregroundServiceType="phoneCall|camera|microphone"
   android:label="@string/app_name"
   android:stopWithTask="false" />

Inherit CoreService like the official Demo and implement it yourself.

 class CoreService : CoreService() {

    override fun onCreate() {
        super.onCreate()
        Log.i("[Service] Created")
    }
    
    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        Log.i("[Service] Ensuring Core exists")
        if (corePreferences.keepServiceAlive) {
            Log.i("[Service] Starting as foreground to keep app alive in background")
            if (!ensureCoreExists(applicationContext, pushReceived = false, service = this, useAutoStartDescription = false)) {
                coreContext.notificationsManager.startForeground(this, false)
            }
        } else if (intent?.extras?.get("StartForeground") == true) {
            Log.i("[Service] Starting as foreground due to device boot or app update")
            if (!ensureCoreExists(applicationContext, pushReceived = false, service = this, useAutoStartDescription = true)) {
                coreContext.notificationsManager.startForeground(this, true)
            }
            coreContext.checkIfForegroundServiceNotificationCanBeRemovedAfterDelay(5000)
        }
        return super.onStartCommand(intent, flags, startId)
    }
    
    override fun createServiceNotificationChannel() {
        // Done elsewhere
    }
    
    override fun showForegroundServiceNotification() {
        Log.i("[Service] Starting service as foreground")
        coreContext.notificationsManager.startCallForeground(this)
    }
    
    override fun hideForegroundServiceNotification() {
        Log.i("[Service] Stopping service as foreground")
        coreContext.notificationsManager.stopCallForeground()
    }
    
    override fun onTaskRemoved(rootIntent: Intent?) {
        if (!corePreferences.keepServiceAlive) {
            if (coreContext.core.isInBackground) {
                Log.i("[Service] Task removed, stopping Core")
                coreContext.stop()
            } else {
                Log.w("[Service] Task removed but Core in not in background, skipping")
            }
        } else {
            Log.i("[Service] Task removed but we were asked to keep the service alive, so doing nothing")
        }
        super.onTaskRemoved(rootIntent)
    }
    
    override fun onDestroy() {
        if (LinphoneApplication.contextExists()) {
            Log.i("[Service] Stopping")
            coreContext.notificationsManager.serviceDestroyed()
        }
        super.onDestroy()
    }
}

3. Other optimizations

For some devices, there may be whistling and noise problems. You can modify the voice parameters in the assets/linphone_factory file. Some are already configured by default. If your requirements cannot be met, you can add the following parameters.

echo cancellation

• echocancellation=1: echo cancellation must be 1, otherwise you will hear your own voice
• ec_tail_len= 100: The tail length indicates the echo duration. The longer the tail length, the stronger the CPU processing power is.
• ec_delay=0: Delay, which means the time from the microphone to the speaker of the echo, not written by default
• ec_framesize=128: The number of samples, it must be exactly one sampling period is the best, it is not written by default

echo suppression

• echolimiter=0: When it is equal to 0, there will be a hollow sound when it is not open. It is recommended not to open it.
• el_type=mic: This option full and mic indicate which device to suppress
• eq_location=hp: This indicates which device the equalizer is used on
• speaker_agc_enabled=0: This indicates whether speaker gain is enabled
• el_thres=0.001: The threshold of the system response means that the system has response processing above which threshold
• el_force=600 : The larger the value of the control radio range, the wider the radio, meaning whether it can receive far background sound
• el_sustain=50: Control the time from sound to silence, which is used to control whether the sound is elongated, which means whether the word is elongated after speaking.

Noise reduction

• noisegate=1 : This means that noise reduction is turned on, and there will be background sound when it is not turned on
• ng_thres=0.03: This means that the sound above the threshold can pass, which is used to judge which is noise
• ng_floorgain=0.03: This indicates that the sound below the threshold is used for gain to compensate for the sound that is too small to be eaten

Network jitter delay packet loss

• audio_jitt_comp=160: This parameter is used for jitter processing. The larger the value, the better the jitter processing, but the theoretical value of the sound delay is 80. Adjust 160 according to the actual situation.
• nortp_timeout=20: This parameter is used for packet loss processing. The smaller the value is, the faster the packet loss will be. The sound will not be interrupted for a long time. At the same time, the sound must be combined with el_sustain to sound good.

Source code reference:
https://github.com/MattLjp/LinphoneCall