From 350ms to 80ms, create the ultimate silky experience of iOS short videos in the new retail scene

As a new promotion point for App products, content has received more and more attention and investment. Short videos are a powerful tool to increase user stickiness and increase user stay time. The content and experience of short videos are directly related to whether users are willing to stay for a long time. Hema also proposes a plan for full-link content video to realize the improvement of product power. Currently there are short video scenes including: homepage, search, product details, talent show, immersive video, real fragrance video, box area homepage feeds stream, topic, UGC content, topic collection landing page, community, recipe, box shoot one Key cutting, live playback, weex, etc.

Author｜God catch

Review｜Taiyi

The goal of this optimization is to align the Hema App with the experience of mainstream short video apps, such as Douyin and Handtao. The specific hard indicators for optimization include the playback success rate, the freeze rate, and the second opening rate. In addition, in order to reflect the real experience of users watching short videos, Hema also added a somatosensory indicator: the rendering time of the first frame.

Optimization effect comparison

https://www.youku.com/video/XNTgwMTI2MzgxNg==

The above video test is based on iPhone 6S, you can see that in most cases, Douyin can start playing immediately after sliding to the next video; and before the optimization of Hema, after sliding to the next video, the cover image will be displayed first. Continue to play again, there is a flashing process. The optimized Hema has a similar effect to that of Douyin.

In order to measure the comparison of the experience before and after optimization with Douyin, the current method of recording a few frames on the screen is used to calculate the time from the full display of the video page to the rendering time of the first frame. The somatosensory data is as follows:

In addition, there are some optimizations of hard indicators, and the results are as follows:

Optimization

In the early stage of this optimization, I investigated many excellent solutions in Alibaba Group, most of which are connected to the mobile player, and the kernel is based on the open source ijkPlayer. However, the player level itself has a high threshold, and the mobile shopping has been optimized well, so the optimization direction this time is mainly focused on the preloading scheme of the upper-level business. Specifically start from the following aspects:

Unified video playback proxy and cache

The loading speed of a video depends largely on the time-consuming downloading from the Internet. Increasing the video cache can effectively increase the speed of the secondary playback of the video. In order to implement the caching mechanism, a proxy server needs to be introduced to take over the video data download process, as follows:

A. Play process before optimization:

B. Play process after optimization:

Before the business layer sets the videoUrl to the player, it encrypts the original videoUrl and replaces it with the local proxyUrl of 127.0.0.1, and directs the request to the proxy webServer. At this time, the proxy module is called to analyze the original video url, read the cache or remotely Request, and finally return data to the player through the server.

Adding intermediate agents for video playback is also a common method in the industry. Hema’s reliance on the mobile player also has ready-made agent services, but its agent function is placed in another independent DW library, which is redundant for Hema, and the SDK is currently temporarily available. The independent pre-download interface is not supported, and the upper layer cannot be optimized for the first broadcast. Therefore, Hema currently has an independent agent layer to support flexible customization at the upper level.

Another advantage of self-built agents is that some business scenarios that do not use the unified mobile player can also enjoy the cache service at the same time, such as the system player used by some flutter pages. At least the management of the cache, the protection of the maximum value of the cache is currently set, and the video cache is cleaned every time the App returns to the foreground.

Proxy and cache for m3u8

In addition to common mp4 videos, m3u8 videos are also encountered daily, such as live playback videos in Hema. (Video link)

This type of video is different from mp4. It does not directly return the video stream when requesting the url, but first returns the playlist text. The playlist contains the various video clips that can be played, as follows:

This kind of video caching process is implemented by modifying the url in the m3u8 playlist and replacing it with the proxy url, and then the proxy can be used. Previously, there was a problem with the caching support of m3u8 on the iOS side and it would crash. The reason was that after modifying the url of the first video of m3u8’s Playlist to proxy proxyUrl, the playback of the first fragment was normal, but the url of the subsequent fragments was still the original url. When the player loads this original relative URL path, the first segment of the domain name and path will be spliced internally, which causes problems with the URL after the second segment and crashes directly. The current processing method is to change all urls in the playlist to the fullpath of the proxy url.

After having two kinds of videos, mp4 and m3u8, the complete process is as follows:

Independent preloading capability

The above-mentioned proxy cache can increase the speed of secondary playback, but there is still no cache available for the video played for the first time, and the download process is still very time-consuming. Therefore, an independent pre-loading capability is required to cooperate with the business layer to download the video data in advance (without rendering) at the right time.

At present, the bottom layer provides the [HMVideoLoader preLoadUrls:URLS] method, which internally caches the video according to the url, and the download size is limited to 1M. When multiple videos are pre-downloaded at the same time, they are executed serially to ensure that they do not occupy too much bandwidth and affect business processing. When the user flicks to the video position, the playback can be started directly to achieve the first opening speed optimization.

What needs to be mentioned is that the preloading here reuses the above-mentioned proxy class, and also uses url as the key for data caching, so that the same cache can be read in subsequent secondary playbacks. If during the preloading process, you slide to the video and start playing, stop the preloading task first to avoid repeated downloads of the same video causing cache conflicts.

Video bit rate and resolution optimization

Video preloading and proxy caching are all based on the perspective of preparing video data in advance. This has the premise that the preparation time is short and the service can be used in time. If the video is large, the network is poor, and the service needs to be consumed immediately, You may not be able to enjoy the optimization effect, so you need to further optimize the video bit rate and resolution.

In the early days, Hema all broadcasted H264 videos, and they were all high-definition videos. This is actually not so big on many feeds streams, which affects the loading speed and wastes traffic. At present, H265 transcoding has been applied for configuration on cloudVideo. After Hema video is uploaded, two channels of 265 and 264 videos can be obtained at the same time, and there are 3 resolutions of HD, SD, and general definition. Here comes the degrees of freedom. Let's take a look at the comparison of the same video size after switching:

A. H264 cut to H265 (all HD): the original H264 size is 10.6M, after switching it becomes 7.1M

B. Switch to H265 and modify the resolution: the original H264 is 21M, after switching it becomes 8.3M

From these two examples, we can see that under the premise that the same video is HD, the size is reduced by about 30% after switching to H265 video. If the resolution is reduced to SD at the same time, the video size is reduced very significantly, which means that the video The bit rate has dropped, and users can download the first frame of data faster.

At present, the Hema server interface has been modified to support the direct return of the H265 video address. The strategy on iOS is to use h265 first, and request different resolutions according to the current environment:

A. Below iOS11, use h264; iOS11 and above, use h265 (The hardware solution is turned on by default in the mobile player)

B. Resolution, define different resolution request order according to current model (high, medium, low), network type (wifi/4g), current network situation (strong, weak), as follows, the final returned array is spelled in order The priority of the resolution parameter, such as hd#sd#ld, which means that HD is prioritized.

static NSString * const VIDEO_HD = @"hd";
static NSString * const VIDEO_SD = @"sd";
static NSString * const VIDEO_LD = @"ld";
static NSString * const VIDEO_HD_H265 = @"hd_265";
static NSString * const VIDEO_SD_H265 = @"sd_265";
static NSString * const VIDEO_LD_H265 = @"ld_265";

+ (NSArray*) getExpectedVideoDefinition {
    NSArray *VIDEO_PRIORITY_GOOD_ENV = nil;
    NSArray *VIDEO_PRIORITY_NORMAL_ENV = nil;
    NSArray *VIDEO_PRIORITY_BAD_ENV = nil;

    if ([[[UIDevice currentDevice] systemVersion] compare:@"11.0" options:NSNumericSearch] == NSOrderedAscending) {
        VIDEO_PRIORITY_GOOD_ENV = @[VIDEO_HD, VIDEO_SD, VIDEO_LD];
        VIDEO_PRIORITY_NORMAL_ENV = @[VIDEO_SD, VIDEO_LD, VIDEO_HD];
        VIDEO_PRIORITY_BAD_ENV = @[VIDEO_LD, VIDEO_SD, VIDEO_HD];
    }
    else{
        VIDEO_PRIORITY_GOOD_ENV = @[VIDEO_HD_H265, VIDEO_SD_H265, VIDEO_LD_H265];
        VIDEO_PRIORITY_NORMAL_ENV = @[VIDEO_SD_H265, VIDEO_LD_H265, VIDEO_HD_H265];
        VIDEO_PRIORITY_BAD_ENV = @[VIDEO_LD_H265, VIDEO_SD_H265, VIDEO_HD_H265];
    }

    AliHADeviceEvaluationLevel deviceLevel = [AliHADeviceEvaluation evaluationForDeviceLevel];
    NetworkQualityStatus networkQualityStatus = [[NWNetworkQualityMonitor shareInstance] currentNetworkQualityStatus];
    NetworkStatus nwStatus = [[NWReachabilityManager shareInstance] currentNetworkStatus];
        
    NSArray *videoPriority = VIDEO_PRIORITY_NORMAL_ENV;
    if (networkQualityStatus == SEMP_StrongSemaphore) {
        if (deviceLevel == HIGH_END_DEVICE) {
            videoPriority = VIDEO_PRIORITY_GOOD_ENV;
        } else {
            if (nwStatus == ReachableViaWiFi) {
                videoPriority = VIDEO_PRIORITY_NORMAL_ENV;
            } else {
                videoPriority = VIDEO_PRIORITY_BAD_ENV;
            }
        }
    } else {
        if (deviceLevel == HIGH_END_DEVICE || deviceLevel == MEDIUM_DEVICE) {
            videoPriority = VIDEO_PRIORITY_NORMAL_ENV;
        } else {
            videoPriority = VIDEO_PRIORITY_BAD_ENV;
        }
    }
    
    return videoPriority;
}

Immersive video page turning motion optimization

After the above solution went online, looking back at the data, the average loading speed has improved, but there is still a loading time of nearly 200ms, which includes the process of player initialization, downloading or loading cached data, and rendering the first frame. The reason is that In a complex network environment for users, it is difficult to ensure that everyone has the best experience. 200ms in the full-screen immersive video scene, although it is much faster than before, it still makes users feel that they are not smooth, that is, after turning to the next page, the user still stays for a short time before playing the first frame. What’s worse is that, for the Hemao video, the cover art of many videos is uploaded by the expert himself, and it is very likely that it is different from the first frame. This makes the pause when jumping from the cover image to the first frame more obvious.

In order to achieve the silky feeling of Douyin, in addition to the above measures, another layer of pretreatment is needed on the upper body feel. Here, a dual player strategy is adopted, as follows:

The basic process is to instantiate the second player in advance while playing the current video, load the video url and pause after playing to the first frame, and perform serial pre-downloading for the 3rd and 4th videos (pre-downloading is a pure downloading process. No rendering logic). After adding the "pre-play" mechanism for the next video, when the user slides to the next video, the user can immediately resume playback from the paused state of the first frame. It is no longer necessary to display the cover image in advance, and the playback speed is also improved. . The rendering of business data other than video can be performed during the process of user swiping and turning pages.

Load optimization of the first video

The above optimizes the user's page turning experience, but the loading experience of the first video of this immersive page still needs to be optimized separately, because when entering the page, it does not leave it with a pre-loading opportunity. as follows:

As shown in the figure, when entering the immersive page, you always need to request the videoList data of the page first, and then request the data of the first video serially. Even if the cover image is added, the user will feel slow. For this reason, now modify the strategy to the right picture. When you jump to the immersive page, you need to pass in the videoUrl from the previous page in advance to play it in advance, and at the same time make an mtop request to render the business data. This ensures that the loading of video and business data can be performed asynchronously. Since the user's main focus is on the video, from the user's point of view, the page loading speed becomes faster.

Audio experience optimization

In the early days, Hema did not pay attention to audio optimization, and received a lot of feedback. Currently, the optimization strategy is as follows:

1. App starts without interrupting music.
2. When entering the audio exclusive page (such as true fragrance video, immersive video), interrupt the music.
3. When you exit the App or return to the background, the music will be restored.
4. Audio playback is not controlled by the mute button (similar to Douyin).

Follow-up optimization direction

1. The player layer provides further encapsulation: encapsulate all boundary logics such as video loading, preloading, dual players, first video judgment, exit, and pause in the screen. At present, each business needs to consider more such boundary conditions. You can consider the encapsulation layer Take it away.
2. Seamless switching of playback progress between pages: Click to switch from a small-size video to an immersive full-screen process to achieve seamless switching. The playback progress continues from the previous page without interrupting the audio. This can further optimize the experience of the first video on the immersive page, and completely realize the "zero time-consuming" body feel.
3. Performance optimization for simultaneous playback of multiple videos: In most scenarios, Hema will only play 1 video at the same time, but some services require multiple videos to be played at the same time, which poses a high challenge to memory and scrolling performance.
4. Video to Gif: In some scenes, the screen is full of videos and needs to be played at the same time. If N players are instantiated at the same time, the effect can be imagined. Consider trying to synchronously produce gif source in the video content production stage. In certain scenarios, APP can use gif to replace the player to achieve preview.
5. Video editing-voice to subtitles: The video editing function has been established in Box immediately based on the ability of Taopai to provide common and easy-to-use editing capabilities for content producers. Consider adding a voice-to-subtitle module to enhance the expression of box horse merchandise in the video.

In the next issue, we will continue to share the experience optimization practices of Hema iOS / Android short videos.

"Video Cloud Technology" Your most noteworthy audio and video technology public account, pushes practical technical articles from the front line of Alibaba Cloud every week, and exchanges and exchanges with first-class engineers in the audio and video field. The official account backstage reply [Technology] You can join the Alibaba Cloud Video Cloud Product Technology Exchange Group, discuss audio and video technologies with industry leaders, and get more industry latest information.