Device hot-plug processing in the scenario of web-interactive microphone connection

The background is this~

Our company has a new web terminal product, which is compatible with both PC and mobile terminals. It is mainly used for audio and video call scenarios such as online seminars, education and training. When it is still in the testing stage, our team will use this product to make microphone calls in various scenarios that require online communication. Real user scenarios have helped us to discover many test cases that are usually overlooked, such as talking in the elevator, in the garage, on the subway, etc., and constantly switching scenarios. What we need to solve today is one of the problems that our product brother found on the subway.

A bug in a specific combination: Android Chrome + Bluetooth headset + WebRTC

Encounter problems

One day there was a meeting within the team, and when it was time to get off work, everyone was still in high spirits (bushi), and there was no intention to call off the meeting. But at a critical moment, the product brother suddenly received a message from his sister-in-law, ordering him to go home immediately. The meeting can't be terminated suddenly, and the sister-in-law's orders can't be ignored. It's unfortunate that our new product can come in handy in this scene: both the people who stayed and the product brother joined the online meeting room and made a microphone call , the meeting materials are shared to the online meeting through screen sharing, PPT, whiteboard, etc. In this way, the product brother will not be delayed when he goes home and holds meetings.

In this way, the product brother went home while turning on his mobile phone and participating in the meeting. But after getting to the subway, the product guy felt that it was unethical to put it out, so he took out the Bluetooth headset, put on the headset skillfully and successfully connected to the mobile phone. At this time, something unexpected happened. The sound was not played from the earphone, but from the speaker of the mobile phone! what happened? Did the earphones break today? The product guy can't believe how the newly bought earphones "end of life" so quickly! Impossible, obviously I still use these headphones to listen to music today. By the way, first try to see if you can still listen to the music. (Switch to the music app to listen to the song) No problem! (Cut back again) Why is there still no sound from the earphones? After several attempts, it was finally confirmed that the problem was not with the earphones, but with our new product.

confirm question

In order to exclude the impact of business code, we test it through a simple demo . This demo is implemented by the WebRTC team . It creates a local audio and video stream through the basic WebRTC API and uses the video tag to play it.

First, let's take a look at the WebRTC API used in the demo. Subsequent solutions also have certain dependencies on these two APIs:

WebRTC API	effect
navigator.mediaDevices.enumerateDevices()	Get a list of media input and output devices. Device types include audio input devices, audio output devices, and video input devices.
navigator.mediaDevices.getUserMedia()	Capture a media stream with default parameters or specified device information and other parameters, and can use the video element to play.

Considering that the WebRTC API has high requirements on browser versions, and considering the browser market share of Chinese people, I only tested Chrome browsers under Android, and Safari browsers and Chrome browsers under iOS.

Test cases and results:

After some testing, it is basically confirmed that the problems encountered can be found on Android Chrome. Later, in Chromium Bug 1285166 , it was found that the Chromium team affirmed the existence of this problem, but the list status was marked as "WontFix". It seems that in the short term, I can only fill in the hole and make compatibility with Android Chrome.

As suggested in Chromium Bug 1285166 , we should have listened for the devicechange event and then switched the audio output device with setSinkId() . However, after learning about setSinkId() compatibility I gave up on this idea.

Combined with a performance found during the test - through getUserMedia() after using the specified audio input device to create a video stream, the audio output channel will be automatically changed to the corresponding device - suddenly came up with an idea: by switching the audio Input device to achieve the effect of switching audio output device. Looking back, our original expectation is to switch the audio input and output devices at the same time. Isn't this the best of both worlds?

Solve the problem

Analysis of alternatives and their advantages and disadvantages

Based on the above test results, two solutions were tentatively finalized:

Option 1: The user chooses the audio input and output device independently

Ideas:

1. Use enumerateDevices() to get the device list, and provide a drop-down list for the user to choose the audio input device;
2. After the user actively chooses to switch the device, use getUserMedia() to recreate the local audio and video stream;
3. Monitor device changes through the devicechange event and update the drop-down list.

advantage:

It is uncertain under what circumstances the actual device being used does not match the expected device, so whenever a problem occurs, it is up to the user to choose the device.

shortcoming:

It needs to be manually switched by the user, and the cost is high. If there is no audio output in the first place (for example, even other people in the mic room turn off the mic or only pull the stream but not push the stream), the user cannot perceive that the device he is using is not as expected, so he may not switch the device manually.

There are two points to note in the implementation process:

1. You need to use getUserMedia() to get the media device permission to get a valid device list.
2. devicechange Event compatibility is poor, not supported at all on Android Chrome.

For point 2, compatibility is achieved by polling the device list with a timer:

 // 监听设备变更事件
function initDeviceChangeListener() {
  const isSupportDeviceChange = 'ondevicechange' in navigator.mediaDevices;
  log(`[support] 是否支持 devicechange 事件:${isSupportDeviceChange}`);
  if (isSupportDeviceChange) {
    navigator.mediaDevices.addEventListener('devicechange', checkDevicesUpdate);
  } else {
    setInterval(checkDevicesUpdate, 1000);
  }
}
​
const prevDevices = await getDevices();
async function checkDevicesUpdate() {
  // 获取变更后的设备列表，用于和 prevDevices 比对
  const devices = await getDevices();
  // 新增的设备列表
  const devicesAdded = devices.filter(device =>
    prevDevices.findIndex(({ deviceId, kind }) =>device.kind === kind && device.deviceId === deviceId) < 0
  );
  // 移除的设备列表
  const devicesRemoved = prevDevices.filter(prevDevice => 
    devices.findIndex(({ deviceId, kind }) => prevDevice.kind === kind && prevDevice.deviceId === deviceId) < 0
  );
  // 设备发生变化
  if (devicesAdded.length > 0 || devicesRemoved.length > 0) {
    // TODO
  }
  prevDevices = devices;
}

This implementation is based on the experience found during testing: "By getUserMedia() after creating a video stream with the specified audio input device, the audio output channel will automatically change to the corresponding device". But is this true and reliable?

After briefly verifying the feasibility of the code in the local development environment, I put the relevant files on the server and continue to use the real test machine for verification ( MDN document here briefly explains why the IP address in the LAN cannot be used to access the test code).

Unexpectedly, I stepped on another pit: after the microphone was switched from the default device to the Bluetooth headset, the sound was output from the earpiece. Problematic test models and browsers: Mi 11 + WeChat, HUAWEI Mate 20 Pro + Chrome 94.0.4606.85.

Cause of the problem: Choose to use an audio input device other than deviceId: 'default', and the automatically switched audio output device does not meet expectations. See Chromium Bug 1277467 for details. How to understand the "non-deviceId: 'default' device" mentioned here? We explain in option two.

As a result, option 1 is not only unreliable, but also introduces new problems.

Option 2: Automatically switch devices for users through code

Ideas:

1. Monitor device changes through devicechange events;
2. When an audio input device is added or removed, use the system's default device to recreate the local audio and video stream.

advantage:

Reduce the burden on users, meet user expectations, and have no perception of the automatic switching process.

However, how does the system's default device know? Let's first take a look at what the device list data obtained under most Android Chrome looks like. The screenshot below was taken on Chrome on a Redmi phone with a Bluetooth headset connected.

As you can see in the screenshot, the browser returns 4 device information ( MediaDeviceInfo ), and the label in the device information represents the description of the device. The first device in the screenshot represents the system default device, and the following 2, 3, and 4 represent hands-free, earpiece, and Bluetooth headset (that is, the "non-deviceId: 'default' device" mentioned in Scheme 1). label各Android 设备下系统默认设备的---59713c3dc61c915d3e53c15e27500f73---值可能存在差异，但它deviceId 'default' ，所以我们deviceId === 'default' As the judgment basis for the default device of the system.

final compatible solution

So far, the solution to the problem has come to the fore. Here first put the GitHub link of the solution implementation code.

Briefly outline the steps:

1. If it is not Android, there is no need to deal with it and end directly.
2. Save a list of devices, denoted as prevDevices .
3. Monitor device change behavior.
4. When a device is changed, get a new list of devices, record it as curDevices .

5. Compare prevDevices and curDevices to get a list of added and removed devices.

   1. In the case of the "host" role:

      1. If the added or removed device list contains a device of type audioinput , use the system default audioinput device as the audio source to recreate a local audio and video stream.
      2. Replace the local audio and video stream (or just replace the audio track of the local audio and video stream, use RTCRtpSender.replaceTrack() ).

   2. In the case of the "spectator" role:

      1. If the newly added device includes a Bluetooth headset, the user may be prompted that there may be no sound, which can be resolved by refreshing the page.

Device hot-plug processing on the PC side

Why do you need to deal with hot plugging?

Compared with the hot-plugging of mobile devices, the browser implementation on the PC side has basically no problems, including devicechange event compatibility, the audio input device is not automatically switched to the corresponding audio output channel and other issues on the PC None exist. Then what is the problem that needs to be dealt with on the PC side?

The thing is, in a certain live broadcast, the streaming picture was stuck in the last frame. After a round of investigation, it was finally confirmed that the problem was that the data line of the external camera was in poor contact.

In these cases, what we can do is to optimize the interaction, such as reminding the user or providing a shortcut entry for switching devices. The following is the user prompt rendering of the newly added device on the PC side.

Our solution

We divide the hot-plugging of devices on the PC side into two cases, adding a device and removing a device .

When adding a new device, it is generally not necessary to automatically switch to the new device, but to remind the user and provide a shortcut entry for switching to the new device. But there is an exception, when the newly added device is the only device in the list of devices of this type, the automatic switching is implemented inside the code. The overall flow chart is as follows:

When removing a device, only the removal of the currently used device needs to be done: automatically switch to another available device. The overall process is as follows:

How to judge the "device in use" mentioned in both processes?

Very simple, first get the local media stream object in use, through MediaStream.getVideoTracks() and MediaStream.getAudioTracks() you can get the MediaStreamTrack instances corresponding to the video track and audio track respectively; Then through mediaStreamTrack.getSettings().deviceId , you can know the currently used device deviceId , so as to determine whether it is "the device in use".

Implementation code:

 /**
 * 从给定的设备列表找当前在使用的设备
 * @param {MediaStream} localStream - 在使用的本地音视频流
 * @param {MediaDeviceInfo[]} devices - 给定的设备列表
 * @returns MediaDeviceInfo[]
 */
function getInUseDevices(localStream, devices) {
  // localStream 为本地音视频流
  if (!localStream) return [];
  const audioTrack = localStream.getAudioTrack();
  const videoTrack = localStream.getVideoTrack();
  const inUseMic = audioTrack ? audioTrack.getSettings().deviceId : '';
  const inUseWebcam = videoTrack ? videoTrack.getSettings().deviceId : '';
  const inUseDevices = [];
  for (let i = 0; i < devices.length; i++) {
    const device = devices[i];
    if ((device.kind === 'audioinput' && device.deviceId === inUseMic) ||
      (device.kind === 'videoinput' && device.deviceId === inUseWebcam)) {
      inUseDevices.push(device);
    }
  }
  return inUseDevices;
}

In fact, there are also "pits" on the PC side

Although there is no compatibility issue with the devicechange event on the PC side, the interface for obtaining the device list navigator.mediaDevices.enumerateDevices() will also "fail" in some specific cases. One of the cases found so far is that when an external camera is used as a video source under Windows, the device is removed after closing the page, and the removed external device will be returned in the device list obtained when the page is reopened. In this case, you need to restart the browser to return to normal. If you want to follow the issue details and progress, you can check Chromium Bug 1336115 .

summary

• Android Chrome + Bluetooth headset + WebRTC has the problem that the audio output channel is not switched to the Bluetooth headset, and you need to perform device compatibility processing yourself.
• There is no compatibility problem on the PC side, and more is to optimize the interaction, and the two situations of device addition and removal need to be considered separately.
• Known issues for Chromium can be viewed here . Accordingly, WebKit also has its corresponding bug list .