Summary on iOS

Broadcast summary below (1)

Overview

Recently modified the video playback function in the project, from the previous full downloading and replaying, to the side-down mode. Since the videos in our project are encrypted when they are sent out, the whole process is actually downloading and decrypting. While playing.

The technical solution broadcasted below is easy to search on blogs on the Internet. There are no more than two ways, built-in local proxy server and AVAssetResourceLoader . We adopted the AVAssetResourceLoader solution provided by the system.

Scheme principle

The specific AVAssetResourceLoader can be found in many logic diagrams on the Internet, as shown in the following figure (from the network).

Here is a brief introduction to this picture in combination with our actual code.

When we usually use AVPlayer to play url, we will create a player like this (abbreviated)

let videoAsset = AVURLAsset(url: "http://resource_url/xxxxx")
let item = AVPlayerItem(asset: videoAsset)
let player = AVPlayer(playerItem: item)

If we set up playback in this way, the entire internal flow of playback is actually invisible to us, video downloading and caching, etc., we can only control the playback pause of the player through some known methods.

If we want to achieve the desired effect in our project, download and play at the same time, we may need to take over the video caching module, so we must be able to enter the entire playback process, AVAssetResourceLoader is actually given to us by Apple Leave a small hole, and then set up AVAssetResourceLoaderDelegate protocol to take over the process of data processing (including obtaining data and filling data into the player).

videoAsset.resourceLoader.setDelegate(self, queue: queue)

Note

1. To enter AVAssetResourceLoader , in addition to setting the delegate for videoAsset.resourceLoader, we also need to change our url to an unrecognizable scheme. Our resource paths are all http or https, we need to change the scheme of the url Unrecognized (private), for example, http://resource/xxx/xxx.mp4 changed to http-prefix://reource/xxxx/xxx.mp4
2. There must be a video suffix at the end of the url path, similar to .mp4, the resource path I used before has no suffix, which caused the player to fail to start broadcasting.

AVAssetResourceLoaderDelegate

AVAssetResourceLoaderDelegate has two commonly used callback methods as follows

// MARK: - AVAssetResourceLoaderDelegate
func resourceLoader(_ resourceLoader: AVAssetResourceLoader, shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest) -> Bool {}

func resourceLoader(_ resourceLoader: AVAssetResourceLoader, didCancel loadingRequest: AVAssetResourceLoadingRequest) {}

When the player starts playing, it will shouldWaitForLoadingOfRequestedResource , and the details of the specific data required are encapsulated in loadingRequest .
Because this callback will go many times, the above figure shows the loadingRequest that needs to be saved every time, but in the actual project, I use a different strategy. I deal with each loadingRequest corresponding to a worker object. So every time we ask for data, there is a separate worker to handle the corresponding network request (not considering caching), which is more organized. At the same time, we also need to save our workers, because if the player needs to support the progress bar drag At this time, you need to manually seek to a certain position, which will trigger didCancel , so we also need to stop our corresponding worker internally.

Callback processing

When we receive a callback, we mainly focus on this loadingRequest of type AVAssetResourceLoadingRequest.

It has a dataRequest attribute inside, and dataRequest has some useful information such as requestedOffset, requestedLength and so on. We construct our Range through requestedOffset and requestedLength, stuff it into the request header, and get the data of the corresponding range.

When our player starts playing, the first callback received, requestedOffset=0, requestedLength=2, that is, the two bytes of 0-1 are requested. This request can actually be understood as a sniffing request, the purpose is In order to get the relevant information of the video, file size, type, etc.

guard request.contentInformationRequest == nil else {
    if request.dataRequest?.requestsAllDataToEndOfResource == false {
        request.contentInformationRequest?.contentLength = totalLen
    } else {
        request.contentInformationRequest?.contentLength = Int64(data.count)
   }
   request.contentInformationRequest?.isByteRangeAccessSupported = true
   request.contentInformationRequest?.contentType = "video/mp4"
   request.finishLoading()
   return
}

The above code is the processing method of the first sniffing request. Through request.contentInformationRequest==nil , it is judged that it is the first sniffing request, and then we need to fill the contentInformationRequest of the request, and then call finishLoading() after filling the information, and the current loadingRequest is over.

After the first sniffing request is over, if we return no problem, the player will immediately perform the next callback to start the required video data. When I test in the project, the second request is generally 0-xxx (file size -1), ask for the entire file, then we dataTask types of requests, waiting for the server to return a piece of data, not part of the data received after the call dataRequest.respond(with: data) , called after completion of all charges request.finishLoading() .

In fact, this is the most basic data filling logic. Except for the special processing of the first sniffing request, the latter is to receive the data and fill it back into the dataRequest. After all the requested data is filled, call finishLoading.

In the process of requesting the entire file, we sometimes find a phenomenon, that is, after responding part of the data, the loadingRequest is canceled, and then we start to ask for the data of the later range. In fact, this can be understood as a moov looking for the file. In the process, the moov of the file may be in the file header or at the end of the file. The time scale, duration, display characteristics and information of each track of the video are defined in the moov. This part can be understood by understanding the format of the mp4 file header.

We don't care what part of the data he asks for, as long as we request the corresponding data, there is no problem responding back.

supplement

So what is the difficulty of such a simple logic for our own project, here is a brief description.

As mentioned earlier, the resources in our project are encrypted, using the AES encryption algorithm, so that after we receive the data, we cannot directly return it to dataRequest . We need to decrypt it first, and then simply say what we use The encryption strategy is that every 16 bytes is an encrypted fragment, but the data returned by the request cannot be guaranteed to be a multiple of 16 each time, so we can deal with the multiple of 16 to decrypt this problem, and then there is a range correction problem, hit For example, we need 10 bytes of data 1-10, but the range of my request header cannot write 1-10 directly, because according to our 16 bytes is an encrypted fragment, we need 1-10, in In the 0-15 segment, so we must first request the 0-15 segment, then decrypt it, then take out 1-10 from it, and fill it back. Of course, there are still some details that will not be described, wait for the opportunity Let's talk about the decryption method of AES separately in combination with the project.

Summarize

The above are some of the small points summarized in the process of implementing the side-by-side broadcast. Of course, everyone may encounter different problems in the actual project. At the same time, this article does not involve data caching. There are also many good caching solutions on github, everyone You can take a look.

Thanks for reading.