The practice of instant delivery of rich media in customer service IM message communication

introduction

Rich media refers to the display method of media media such as pictures, voices, videos, files, etc. transmitted in the instant messaging process.

1. Background

The customer service one-stop platform aims to provide a one-stop service office platform for service personnel in the customer service domain within the Dewu Ecosystem. We have multiple lines of business. In the process of chatting with users, the customer service needs to send rich media in many scenarios. Compared with ordinary text transmission, rich media can intuitively let users know the content of the message, but it also faces problems such as large files, large memory consumption, and long transmission process during the transmission process.

2. Challenges faced

The general process for customer service to send messages such as large files (videos, pictures) to users is as follows:

First upload to CDN through the file upload service, and return the corresponding CDN address link; secondly, obtain the CDN address link, and return the link to the user interface for rendering through the IM gateway.

During the entire transmission process, the front-end must wait for the file to be uploaded successfully and get the link before rendering. If the transmitted file is large, the customer service needs to wait for a long time, which has a great impact on the wiring efficiency of the customer service. The ideal way is that when the customer service sends the file, the file is immediately rendered in the chat window. At this time, it is not the complete file, but the image of the file, such as the file name and cover image, which is uploaded through the status of the message. control.

Taking video transmission as an example, if the video is directly displayed in the cache and displayed in the customer service chat content area, the huge cache will cause the user's browser to crash in minutes. For example, for a video larger than 70M, when the network, computer hardware and other environments are good, the process from reading the file to obtaining the first frame picture transmission takes about 2~3s. If the network is normal, there are many people in the same environment. It will take longer in the case of sending video files, or hardware devices in general.

How to make the transmission of large files silky smooth without affecting the efficiency of customer service wiring?

3. Solutions and Results

1. Use fileReader.target.result as the url of the video to render on the page

The initial method used is to intercept the first frame of the video when the video is uploaded to the CDN, and then upload the first frame of the intercepted video to the CDN, and then send it to the client through a long chain (wss), because intercepting the first frame is a synchronous process. , you need to get the url of the screenshot before rendering to the page. As a result, the customer service cannot see the sent video in the chat interface as soon as you click send. As shown in the video above, the customer service cannot perceive the progress of the video sending.

Read file information through FileReader:

 export function getFileInfo(file: File): Promise<any> {
  return new Promise((resolve, reject) => {
    try {
      const reader = new FileReader()
      reader.readAsDataURL(file) 
      reader.onload = (event: ProgressEvent<FileReader>) => {
        resolve(event)
      }
    } catch (e) {
      reject(e)
    }
  })
}

Set properties through the returned file information:

 export function getVideoInfo(file) {
  return new Promise((resolve, reject) => {
    getFileInfo(file)
      .then(fileReader => {
        const target = fileReader.target.result
        if (/video/g.test(file.type)) {
          const video = document.createElement('video')
          video.muted = true
          video.setAttribute('autoplay', 'autoplay')
          video.setAttribute('src', target)
          video.addEventListener('loadeddata', () => {
            // ...
          })
          video.onerror = e => reject(e)
        }
      })
      .catch(e => reject(e))
  })
}

As shown in the above code video.setAttribute('src', target), if the target is used as the url of the video to render on the page, the page will crash in minutes. You can take a look at a 1M video file, read the file content through readAsDataURL(file), and get a base64 string of data:url. Using this string for rendering is equivalent to adding a 1.4M string content to the page, as shown below. As shown, the consequences of doing so are unimaginable, and if the file is slightly larger, there will be more obvious freezes.

So this plan was rejected at the beginning of development.

2. Use URL.createObjectURL(file) to get the URL

After the first solution was rejected, the implementation of URL.createObjectURL was investigated. Use URL.createObjectURL(file) to get the URL (this URL object represents the specified File object or Blob object), and then put it in the cache of chat data, so that it can be quickly sent to the customer service chat window page. Its main implementation code is as follows:

 if (/*******/) {
    // ...
    //. blob作为预览视频的url
    state.previewVideoSrc = URL.createObjectURL(file)
    state.previewVideo = true
    state.cachePreviewVideoFile = file
    nextTick(() => {
      focus()
    })
  } else {
    // ...
  }

After this transformation, it is obvious that after the video is sent, it can be quickly displayed on the page, so that the customer service can perceive the status and progress of the video sending. Compared with the first solution, the process of video sending has been significantly improved. The rendered code effect is shown in the following figure:

but!

When sending video information to the client, it is necessary to carry the first frame and video duration as the display cover. The historical practice is:
First, the front-end obtains the file information, converts it into an image through canvas, and uploads it to the CDN; after obtaining the first frame and file information, upload it to the CDN, return the URL, and then send it to the user through the long chain, and update the URL address of the page to CDN The real address returned.

The file needs to be read when the first frame is taken. Since it is to read the file, there is still a certain time-consuming task. As shown in the following code snippet, this time-consuming task will also affect the customer service experience.

 export function getVideoInfo(file, msgid?: string) {
  return new Promise((resolve, reject) => {
    getFileInfo(file, msgid)
      .then(fileReader => {
        const target = fileReader.target.result
        if (/video/g.test(file.type)) {
          const video = document.createElement('video')
          video.muted = true
          video.setAttribute('autoplay', 'autoplay')
          // target只作为url创建视频用于获取视频大小、播放时长等基本信息，不用于页面渲染
          video.setAttribute('src', target)
          video.addEventListener('loadeddata', () => {
            const canvas = document.createElement('canvas')
            canvas.width = video.videoWidth
            canvas.height = video.videoHeight
            const width = video.videoWidth
            const height = video.videoHeight
            canvas.getContext('2d')!.drawImage(video, 0, 0, width, height)
            const src = canvas.toDataURL('image/jpg')
            const imgFile = dataURLtoFile(src, `视频_${Math.random()}.png`)
            return getImgInfo(imgFile, fileReader.msgid).then(
              ({ width: imgWidth, height: imgHeight, file: imgFile, size: imgSize, src: imgSrc, msgid }) => {
                resolve({
                //  ...
                })
              }
            )
          })
          video.onerror = e => {
            // ...
            reject(e)
          }
        }
      })
      .catch(e => {
        reject(e)
      })
  })
}

When uploading a video, the file server provides a way to obtain the first frame of the picture, and then splices the corresponding parameters on the link address, as shown below:

 // 拼接的获取图片首帧的URL地址
export const thumbSuffix = `?x-oss-process=video/snapshot,****`
export function addOssImageParams(url, isThumb = false) {
  const suffix = isThumb ? thumbSuffix : urlSuffix
  if (!url) return ''
  // ...
  return url
}

However, in the actual usage scenario, it is not enough to only obtain the information of the first frame of the video, but also to obtain information such as the width and height of the video, and the playback duration, and send it to the gateway through a network request, and finally display it on the client. The process of reading files cannot be avoided, and the time-consuming problem still needs to be solved.

3. Web Worker reads file information asynchronously

Although the fast rendering of the file is achieved through the second solution, if the reading of the file information is done in the main thread of the browser, it will take a long time and will hinder the operation of the customer service. It would be perfect if this process could be implemented asynchronously. Although JS is single-threaded, the browser provides the ability of Web Worker, so that JS can also communicate with the main thread in an asynchronous manner. First, compare the difference between the browser's main thread execution and the main sub-thread execution, as shown in the following figure:

When the main thread of the browser is executing the sending of files, if the task of sending files is not over, other tasks will be blocked, which is equivalent to that during the sending period, the customer service cannot do anything; The thread reads the file. During the reading of the file, the main thread can continue to perform other tasks. After the child thread finishes reading the file, it sends relevant information through postMessage to inform the main thread that the file is read, and the main thread starts rendering. The whole process does not block any customer service.

The process implemented by the main sub-thread of the Web Worker is as follows:

First create a sub-thread task in the thread subscription center, as follows:

 // 子线程任务
export function subWork() {
  self.onmessage = ({ data: { file } }) => {
    try {
      // 读取文件信息
      // ...
      // 发送对应信息
      self.postMessage({ fileReader: **** })
    } catch (e) {
      self.postMessage({ fileReader: undefined })
    }
  }
}

Then initialize the Worker in the thread subscription center, as follows:

 export const createWorker = (subWorker, file, resolve, reject) => {
  const worker = new Worker(URL.createObjectURL(new Blob([`(${subWorker.toString()})()`])))
  // 发到子线程
  worker.postMessage({
    file
  })
  // 监听子线程返回数据
  worker.onmessage = ({ data: { fileReader } }) => {
    resolve(fileReader)
    // 获取到结果后关闭线程
    worker.terminate()
  }
  // 监听异常
  worker.onmessageerror = function () {
    worker.terminate()
  }
}

Finally, call the Worker in the main thread to obtain the file information, as follows:

 // 创建主线程任务
export const getFileInfoFromSubWorker = files => {
  return new Promise((resolve, reject) => {
    createWorker(subWork, files, resolve, reject)
  })
}

Through the above three steps, you can basically get the file information without affecting the customer service operation. After obtaining the video information object, you can obtain the video-related attribute information through URL.createObjectURL(file), as follows:

 export function getVideoInfo(file, blob, msgid?: string) {
  return new Promise((resolve, reject) => {
    if (/video/g.test(file.type)) {
      const video = document.createElement('video')
      video.muted = true
      video.setAttribute('autoplay', 'autoplay')
      // blob作为url: URL.createObjectURL(file)
      video.setAttribute('src', blob)
      video.addEventListener('loadeddata', () => {
        const width = video.videoWidth
        const height = video.videoHeight
        resolve({
          videoWidth: width,
          videoHeight: height,
          videoDuration: video.duration * 1000,
          videoFile: file,
          videoSize: file.size,
          videoSrc: blob,
          msgid
        })
      })
      video.onerror = e => {
        reject(e)
      }
    }
  })
}

As mentioned above, after obtaining the file object information, the width and height of the video are directly obtained by blob as the width and height of the first frame picture. The combination of the two achieves that without affecting the customer service operation, the video transmission can be done Arrived as silky smooth.

Through the method of Web Worker+URL.createObjectURL(file) , it is solved that when the rich media file is sent, regardless of whether it is sent successfully or not, the effect of sending it in seconds can be achieved, that is, the video information can be displayed in the chat box first, and then the sending status can be sent to the chat box. Identifies the current sending progress.

4. Summary

Rich media sending is involved in many IM scenarios. What kind of technology is used to make communication and communication between customer service and users more convenient is the focus of this article. Through the practice in the actual customer service business scenario, the technical solution in this article has solved the problems in the business very well, and the actual online has been running relatively stably. Finding problems from business, solving problems with technical means, improving the efficiency of customer service, and bringing good experience to users are our constant pursuit goals. If you have better suggestions after reading this article, you can leave us a message. In addition, the technical points in the field of customer service are far more than these, down-to-earth, step by step, I believe that the precipitation of instant messaging in the field of customer service will get better and better.

5. Knowledge expansion

1. Implementation differences of file reading

Both URL.createObjectURL() and FileReader.readAsDataURL(file) can get file information, why do we choose to use the former instead of the latter?

The main differences between the two are:

• What is obtained through FileReader.readAsDataURL(file) is a string of data:base64, and the string of base64 bits is larger
• Create a DOMString through URL.createObjectURL(blob), which contains the URL containing the file information (the specified File object or Blob object)

The timing of execution is different:

• createObjectURL is immediate execution
• FileReader.readAsDataURL is (after some time) executed asynchronously

Memory usage is different:

• createObjectURL returns a url with hash, which is always stored in memory. When the document is triggered to unload or execute revokeObjectURL to release memory;
• FileReader.readAsDataURL returns a base64 string, which consumes more memory than blob url, but this data will be automatically cleared by garbage collection mechanism.

Use options:

• Using createObjectURL can save performance and get faster;
• If the device performance is good enough and you want to get the base64 of the image, you can use FileReader.readAsDataURL.

2. Concepts of streaming media, rich media, multimedia

What is the difference between streaming, rich media, and multimedia?

Streaming media: While using it, download things that may be used later in the background.
Rich media: page content in which text, pictures, video, and audio are mixed.
Multimedia: pictures, text, audio, video and other materials. Among them, streaming media is a transmission method, rich media is a display method different from plain text, and multimedia is a means of displaying content.

Recommended reading:
Dewu Customer Service IM Message Communication SDK Self-Research Road Micro Front-end Practice in Customer Service Domain

*Text/Jun
@德物科技public account