Cloud Music iOS Network Image Download Optimization Practice

Image from: https://unsplash.com
Author of this article: lgq

background

Picture display is indispensable in major APPs. As we all know, Cloud Music is a music software with social attributes, so in any social scene, there will be demands to display pictures, and there are often heavy picture scenes, such as a cloud The feed stream scenes of Mlog in music are all pictures, or the atlas in Mlog, which requires a large number of pictures to be displayed. If the pictures cannot be displayed in time and cannot be consumed by users in a timely manner, it will cause users to browse information not smoothly. Lead to the loss of users, so optimizing image download is imminent.

Existing image download technology

Here is a brief introduction to the image resource service connected to the Cloud Music APP. It can be cut and compressed at the remote end through splicing parameters to download different images. For more information reference

Factors Affecting Image Downloads

1. size of picture
2. Network situation
3. local cache
4. cdn cache

To sum up, how to improve the download speed of pictures can be optimized from the above points.

Optimization method

Network Optimization

• Under the traditional HTTP1.0 architecture, multiplexing is impossible. Using the HTTP2.0 method, requesting resources of the same IP domain name can save a lot of connection establishment and transmission time.
• In addition, when I was working on a page with heavy audio and video scenes, I found that the data of audio and video streaming media would sometimes occupy a lot of bandwidth, resulting in very slow download of pictures. At this time, it is necessary to properly control the download of audio and video scene resources. , such as current limiting and other operations, depending on the business priority. For example, when using socket to download audio and video scenarios, you can adjust the recv buffer size appropriately.

Image size optimization

• Format optimization is the easiest to think of and the most effective. If you use regular pictures such as jpg and png, the size of the picture will be relatively large. At present, our nos service supports the specified type and converts the picture into a specific type. Format, so we use webp here to reduce the size of the image. (Just need to splicing type as webp in the request parameter)

So what else can we do besides that?

• For example, a 100 100 control can be cropped on demand. In the case of 3 times the screen, we only need to download 300 300 pictures. If the picture exceeds the size, it is meaningless for us to download such a large one. Therefore, according to the size of the control, we can determine the size of the picture we download, thereby reducing the picture we need to download.
• For scenes that do not require such high compression quality, we only need images with a quality of 80.

Thinking <br>After completing the above items, we can find that the speed is increased by at least 30%, but can we do more, or is there any flaw in this scheme?

Forensics <br>For this purpose, we simply pulled the background data. The following problems were found:

1. The parameters of URL splicing are different, which makes it impossible to hit the local cache, so there will be a problem of repeated downloads, such as user avatars, user avatars appear repeatedly in each scene, and the sizes are different, which will be downloaded multiple times, which will lead to a certain waste of resources. At the same time, due to the different link parameters, the cdn hit rate is not high
2. The UI size of different models may not be the same, resulting in different sizes of downloaded files. The more models, the more spliced sizes, and the server needs to repeat cutting.
3. The quality parameters are determined by the upper-layer business, which will lead to the failure of agreement between different ends, and various pictures will be downloaded.

solution

1. URL parameter standardization The so-called standardization is to standardize the parameter splicing used by the front-end, which is divided into sequence standardization and parameter value fitting.
   We know a URL link to download an image http://path?imageView=1&enlarge=1&quality=80&thumbnail=80x80&type=webp .
2. The parameters are sorted by the first letter, so that if the parameter requirements are the same, there will be no repeated requests.
3. The thumbnail parameter actually corresponds to the size of the image that needs to be downloaded. We fit it (according to the data obtained from the back-end statistics), divide it into multiple gears (the gears can be configured), and scale them in equal proportions according to the wide side, so as to make it as possible as possible Less avoidance of the screen of the model is a little bit, and there are cases of other sizes.
4. The quality is also graded and divided into multiple gears (the gears can be configured).
5. Deduplication, parameters may be multi-spliced, and redundant parameters are deduplicated
6. The simple understanding of local size image reuse is that there is a large image locally. When taking a small image, no additional network request is required, and it is directly cropped locally.
   We have optimized the logic of reading the local cache. When fetching the cache, we will perform an associated search, find an available image for cropping, and return it directly.
   The specific rules are as follows:
7. Different cropping parameters can be converted, x, z cropping parameters can be converted to y, y can not be converted to x, z. can be converted to the same cropping parameters. The meanings of x (inner abbreviation), y (cropping abbreviation), and z (outer abbreviation) are in this document , representing different filling modes.
8. High-quality pictures can be reused as low-quality pictures, and low-quality pictures cannot be reused as high-quality pictures

iOS code implementation

After talking about the plan, we can go to the code. Here is the implementation plan for iOS:

First of all, we do a certain encapsulation based on SDWebImage, first briefly understand the general process in SDWebImage.

As we can see from the figure, imageLoader is mainly used for downloading pictures, and imageCache is used for searching cache, both of which are managed in manager

Retrofit process

We only need to fix the URL at the beginning of the data flow, and at the same time add an additional lookup to the image when looking up the cache.

URL FIX

We add a classification to the URL and perform a fix operation on the URL. The solution is to use the system provided NSURLComponts to align the operation, extract its parameters, deduplicate and standardize, and we have some historical reasons. Some old parameters are converted to the correct format, the final step is sorting, and the fix process is complete.

 - (NSURL *)demo_fixImageURL {

    NSURLComponts *componts = [NSURLComponts compontsWithURL:self
                                             resolvingAgainstBaseURL:YES];
    NSMutableArray<NSURLQueryItem *> *queryItems = componts.queryItems.mutableCopy;

    ... 从URL取出 NSURLQueryItem 省略一些代码

    if (qualityItem) {
        //quality 拟合， 将质量参数分为几档
        NSString *defaultQualityStr = @"39,69,89";

        //这里是伪代码，就是为了获取配置信息
        NSArray<NSString *> *qualityLevel = CustomConfigQualityLevels;

        //固定 4档
        if (qualityLevel.count == 3) {
            NSInteger quality = [qualityItem.value intValue];
            NSString *fixQuality = @"";
            if (quality <= [[qualityLevel _objectAtIndex:0] intValue]) {
                fixQuality = [@(ImageQualityLevelLow) stringValue];
            } else if (quality <= [[qualityLevel _objectAtIndex:1] intValue]) {
                fixQuality = [@(ImageQualityLevelMed) stringValue];
            } else if (quality <= [[qualityLevel _objectAtIndex:2] intValue]) {
                fixQuality = [@(ImageQualityLevelHigh) stringValue];
            } else {
                fixQuality = [@(ImageQualityLevelOrigin) stringValue];
            }
            NSURLQueryItem *fixQualityItem = [[NSURLQueryItem alloc] initWithName:@"quality" value:fixQuality];
            [queryItems removeObject:qualityItem];
            [queryItems addObject:fixQualityItem];
        }
    }

    if (sizeItem) {
        //size 按照宽边拟合 分为几档且 等比缩放
        NSString *defaultSizeStr = @"30,60,90,120,180,256,315,512,720,1024";

        //这里是伪代码 就是为了获取配置信息
        NSArray<NSString *> *sizeLevels = CustomConfigSizeLevels;
        NSString *originSizeStr = sizeItem.value;

        CGSize originSize = CGSizeZero;

        NSString *separatedStr = nil;
        for (NSString *separated in @[@"x", @"z", @"y"]) {
            NSArray *sizeList = [originSizeStr compontsSeparatedByString:separated];
            if (sizeList.count == 2) {
                originSize = CGSizeMake([sizeList[0] intValue], [sizeList[1] intValue]);
                separatedStr = separated;
                break;
            }
        }

        CGSize finalSize = CGSizeZero;
        if (!CGSizeEqualToSize(originSize, CGSizeZero)) {
            BOOL isW = originSize.width > originSize.height;
            NSInteger len = isW ? originSize.width : originSize.height;
            NSInteger requestSize = 0;
            for (NSString *sizeLevel in sizeLevels) {
                NSInteger sizeNumber = [sizeLevel integerValue];
                if (sizeNumber >= len) {
                    if (requestSize == 0) {
                        requestSize = sizeNumber;
                    } else {
                        requestSize = MIN(requestSize, sizeNumber);
                    }
                }
            }
            if (isW) {
                if (originSize.width != 0) {
                    NSInteger h = (requestSize / (originSize.width * 1.f)) * originSize.height;
                    finalSize = CGSizeMake(requestSize, floor(h));
                }

            } else {
                if (originSize.height != 0) {
                    NSInteger w = (requestSize / (originSize.height * 1.f)) * originSize.width;
                    finalSize = CGSizeMake(w, floor(requestSize));
                }
            }
        }

        if (!CGSizeEqualToSize(finalSize, CGSizeZero)) {
            NSString *fixSize = [NSString stringWithFormat:@"%ld%@%ld",(NSInteger)finalSize.width, separatedStr, (NSInteger)finalSize.height];
            NSURLQueryItem *fixSizeItem = [[NSURLQueryItem alloc] initWithName:@"thumbnail" value:fixSize];
            [queryItems removeObject:sizeItem];
            [queryItems addObject:fixSizeItem];
        }

    }

    //去重复
    NSMutableArray<NSString *> *keys = @[].mutableCopy;
    queryItems = [queryItems bk_select:^BOOL(NSURLQueryItem *obj) {
        BOOL containsObject = [keys containsObject:obj.name];
        [keys addObject:obj.name];
        return !containsObject;
    }].mutableCopy;

    //首字母排序
    queryItems = [queryItems sortedArrayUsingComparator:^NSComparisonResult(NSURLQueryItem *obj1, NSURLQueryItem *obj2) {
        return [obj1.name compare:obj2.name options:NSCaseInsensitiveSearch];
    }].mutableCopy;

    //最终组合
    componts.queryItems = queryItems.copy;
    NSURL *finalURL = componts.URL;

    return finalURL;
}

SDWebImageManager

After fixing the URL, what is the next step and how to pass the fixed URL? It can also be seen from the above SDWebImage process that all image download processes are inseparable from SDWebImageManager, so we inherit SDWebImageManager and rewrite the following methods

 - (SDWebImageCombidOperation *)loadImageWithURL:(nullable NSURL *)url
                                          options:(SDWebImageOptions)options
                                          context:(nullable SDWebImageContext *)context
                                         progress:(nullable SDImageLoaderProgressBlock)progressBlock
                                        completed:(nonnull SDInternalCompletionBlock)completedBlock

If you want to go through the repair process in the future, you only need to use our packaged manager to achieve the following

 - (SDWebImageCombidOperation *)loadImageWithURL:(nullable NSURL *)url
                                          options:(SDWebImageOptions)options
                                          context:(nullable SDWebImageContext *)context
                                         progress:(nullable SDImageLoaderProgressBlock)progressBlock
                                        completed:(nonnull SDInternalCompletionBlock)completedBlock
                                         corp:(BOOL)corp {

    NSURL *fixURL = [self.class fixURLWithUrl:url];
    SDInternalCompletionBlock fixBlock = completedBlock;
    if (![fixURL.absoluteString isEqualToString:url.absoluteString] && corp) {
        fixBlock = [self.class fixcompletedBlockWithOriginCompletedBlock:completedBlock url:url];
    }
    return [super loadImageWithURL:fixURL options:options context:context progress:progressBlock completed:^void(UIImage * _Nullable image, NSData * _Nullable data, NSError * _Nullable error, SDImageCacheType cacheType, BOOL finished, NSURL * _Nullable imageURL) {

        if (fixBlock) {
            fixBlock(image,data,error,cacheType,finished,imageURL);
        }
    }];

}

Careful students can find that we have added a parameter corp . If the upper-level business needs to follow the size he passed in, we will do a layer of cropping and scaling operations. The specific operation is placed in fixBlock . The default is not to fix, because the picture sent by the nos server itself is not necessarily the desired size of the business incoming.

The core code of fixblock uses the cropping that comes with sd_webimage

 cutImage = [image sd_resizedImageWithSize:requestSize scaleMode:[urlInfo.cropStr isEqualToString:@"x"] ? SDImageScaleModeAspectFit : SDImageScaleModeAspectFill];

The code is basically here fixURL The operation is basically completed, but if you need to be compatible with the old cache (already locally and permanently cached (special case), but the resource pictures that have been removed from the Internet), in fixblock In the case of loading failure, we use the old URL to fetch the local cache once.

 [[self sharedManager] loadImageWithURL:url options:options | SDWebImageFromCacheOnly context:mutContext.copy progress:nil completed:completedBlock];

Note: URLs that have been fixed will not be fixed again. Whether it is a permanent cache or not is distinguished by imageCache

SDWebImageFromCacheOnly means only read from the cache, avoiding the problem of repeated requests.

imageCache

As mentioned above, to achieve reuse, imageCache needs to be modified. Here I have to mention the following process of SDWebImage finding cache

As can be seen from the figure, the URL needs to be converted to a cacheKey, and then the cache is retrieved from memory or disk. So how do we transform it, because we need to find locally reusable images through URLs

The cacheKey needs to retain certain rules, and some things of the original URL can be seen through the cache. So our cachekey is generated like this

 + (NSString *)cacheKeyForURL:(NSURL *)url  {

    NSURL *wUrl = url;
    NSString *host = wUrl.host;
    NSString *absoluteString = wUrl.absoluteString;
    if (!host)
    {
       return absoluteString;
    }

    NSRange hostRange = [absoluteString rangeOfString:host];
    if (hostRange.location + hostRange.length < absoluteString.length)
    {
       NSString *subString = [absoluteString substringFromIndex:hostRange.location + hostRange.length];
       if (subString.length != 0)
       {
           return subString;
       }
    }
    return absoluteString;
}

In short, remove the host and keep the remaining parameters. ps: Because fixURL has repeated request parameters, cacheKey can also be guaranteed to be unique for the same image.

Then how to find other local pictures through the URL, and how to associate them?

You can find the associated cachekey through the path, and then find the corresponding image

After finding the picture, select one that can be used and crop it. The process is as follows:

Here we encapsulate an object for the cached image information. Note that it will be persisted in the database ImageCacheKeyAndURLObject array, and its key is the request URL link path , note that the database has an upper limit, At the same time, it will be cleaned up at the appropriate time (such as image cache expiration, etc.)

The following is the object that encapsulates the persistence

 @interface WebImageCacheImageInfo : NSObject

@property (nonatomic) BOOL isAnimation;
@property (nonatomic) CGFloat sizeW;
@property (nonatomic) CGFloat sizeH;

- (CGSize)size;

@end

@interface WebImageURLInfo : NSObject

@property (nonatomic) CGSize requestSize;
@property (nonatomic) NSString *cropStr;
@property (nonatomic) NSInteger quality;
@property (nonatomic) NSInteger enlarge;

@end

@interface WebImageCacheKeyAndURLObject : NSObject<NMModel>


@property (nonatomic, readonly) NSString *path;
@property (nonatomic) NSString *cacheKey;
@property (nonatomic, nullable) NSURL *url;
@property (nonatomic, nullable) WebImageCacheImageInfo *imageInfo;

- (NSArray<WebImageCacheKeyAndURLObject *> *)relationObjects;
- (nullable WebImageCacheKeyAndURLObject *)canReuseObject;
- (WebImageURLInfo *)urlInfo;
- (void)storeImage:(UIImage *)image;
- (void)remove;
@end

How to store image information

 - (void)storeImage:(UIImage *)image {
    if (self.path.length == 0) {
        return;
    }
    BOOL isAniamtion = image.sd_isAnimated;
    CGSize size = image.size;
    if (image) {
        _imageInfo = [WebImageCacheImageInfo new];
        _imageInfo.sizeH = size.height;
        _imageInfo.sizeW = size.width;
        _imageInfo.isAnimation = isAniamtion;
    }

    NSMutableArray<WebImageCacheKeyAndURLObject *> *items = [[self searchfromDBUsePath:self.path] mutableCopy];
    if (items.count == 0) {
        items = @[].mutableCopy;
    }
    if ([items containsObject:self]) {
        [items removeObject:self];
    }
    [items addObject:self];
    [self saveDBForPath:self.path item:items];
}

How to judge whether the picture can be reused?

 - (WebImageCacheKeyAndURLObject *)canReuseObject {

    WebImageURLInfo *info = self.urlInfo;
    if (CGSizeEqualToSize(CGSizeZero, info.requestSize)) {
        return nil;
    }
    NSArray<WebImageCacheKeyAndURLObject *> *relationObjects = [self relationObjects];

    // 非动图 尺寸非0

    relationObjects = [relationObjects bk_select:^BOOL(WebImageCacheKeyAndURLObject *obj) {
        return !obj.imageInfo.isAnimation && obj.imageInfo.size.width > 0 && obj.imageInfo.size.height > 0;
    }];

    @weakify(self)
    relationObjects = [relationObjects bk_select:^BOOL(WebImageCacheKeyAndURLObject *obj) {
        @strongify(self)
        return ![obj.cacheKey isEqualToString:self.cacheKey];
    }];

    // 质量大于请求的图
    relationObjects = [relationObjects bk_select:^BOOL(WebImageCacheKeyAndURLObject *obj) {

        WebImageURLInfo *objInfo = obj.urlInfo;

        NSInteger quality = objInfo.quality == 0 ? 75 : objInfo.quality;
        NSInteger requestQuality = info.quality == 0 ? 75 : info.quality;
        return quality >= requestQuality;
    }];

    //缩放能支持的
    NSArray<WebImageCacheKeyAndURLObject *> *canUses = nil;
    if ([info.cropStr isEqualToString:@"x"] || [info.cropStr isEqualToString:@"z"]) {
        canUses = [relationObjects bk_select:^BOOL(WebImageCacheKeyAndURLObject *obj) {
            WebImageURLInfo *objInfo = obj.urlInfo;
            if ([objInfo.cropStr isEqualToString:@"x"] || [objInfo.cropStr isEqualToString:@"z"]) {
                CGSize displaySize = WebImageDisplaySizeForImageSizeContentSizeContentMode(obj.imageInfo.size, info.requestSize, [info.cropStr isEqualToString:@"x"] ? UIViewContentModeScaleAspectFit :  UIViewContentModeScaleAspectFill);

                CGFloat p = 0;
                if (info.requestSize.width > 0) {
                    p = displaySize.width / obj.imageInfo.size.width;
                } else {
                    p = displaySize.height / obj.imageInfo.size.height;
                }
                return p <= 1;
            } else {
                // y 不可以转z/x
                return NO;
            }
        }];
    } else if ([info.cropStr isEqualToString:@"y"]) {
        canUses = [relationObjects bk_select:^BOOL(WebImageCacheKeyAndURLObject *obj) {
            WebImageURLInfo *objInfo = obj.urlInfo;
            if ([objInfo.cropStr isEqualToString:@"x"] || [objInfo.cropStr isEqualToString:@"z"]) {
                CGSize displaySize = WebImageDisplaySizeForImageSizeContentSizeContentMode(obj.imageInfo.size, info.requestSize, UIViewContentModeScaleAspectFill);
                CGFloat p = 0;
                if (info.requestSize.width > 0) {
                    p = displaySize.width / obj.imageInfo.size.width;
                } else {
                    p = displaySize.height / obj.imageInfo.size.height;
                }
                return p <= 1;
            } else  if ([objInfo.cropStr isEqualToString:@"y"]) {
                return (obj.imageInfo.size.width >= info.requestSize.width && obj.imageInfo.size.height >= info.requestSize.height);
            }
            return NO;
        }];
    }
    return canUses.firstObject;
}

To filter the animation, because the local clipping of the animation is more difficult to handle, and the proportion is not high, so ignore him here first, WebImageCacheKeyAndURLObject records cacheKey and other related information, the core also records some related information the actual cached image size. Easy to query. WebImageDisplaySizeForImageSizeContentSizeContentMode is the incoming image size, container size, and fill mode to calculate the scaled image size.

Once the relationship is established, when is the time to look for it?
We inherit SDImageCache and rewrite it

 - (nullable NSData *)diskImageDataBySearchingAllPathsForKey:(nullable NSString *)key;

This method searches further if data is not found. Crop the associated image found and use the same correction method as above

 if ([objInfo.cropStr isEqualToString:@"x"] || [objInfo.cropStr isEqualToString:@"z"]) {
                result = [result fixResizedImageWithSize:requestSize scaleMode:[objInfo.cropStr isEqualToString:@"x"] ? UIViewContentModeScaleAspectFit : UIViewContentModeScaleAspectFill needCorp:NO];
            } else if ([objInfo.cropStr isEqualToString:@"y"]) {
                result = [result fixResizedImageWithSize:requestSize scaleMode:UIViewContentModeScaleAspectFill needCorp:YES];
            }

Add the fixsize method here

 - (UIImage *)fixResizedImageWithSize:(CGSize)size scaleMode:(UIViewContentMode)scaleMode needCorp:(BOOL)needCorp {

    if (scaleMode != UIViewContentModeScaleAspectFit && scaleMode!= UIViewContentModeScaleAspectFill) {
        return self;
    }

    // 如果是fill模式，实际size会大于容器size 如果需要裁剪为容器大小就不走这一步了
    if (scaleMode == UIViewContentModeScaleAspectFill && !needCorp) {
        size = WebImageDisplaySizeForImageSizeContentSizeContentMode(self.size, size, scaleMode);
    }

    UIImage *fixImage = [self sd_resizedImageWithSize:size scaleMode:scaleMode == UIViewContentModeScaleAspectFit ? SDImageScaleModeAspectFit : SDImageScaleModeAspectFill];
    return fixImage;

}

In this way, we can get the repaired image, and the process is complete.

Categories such as UIImageView and UIButton

We wrap a layer of our own downloads, and then pass in our manager.

 context = @{
           SDWebImageContextCustomManager:[WebImageManager sharedManager]
       };

say something extra

The CDN hit rate is related to whether the resource has ever been requested, and the key to hit the CDN is the requested URL, so it is very important to keep the same rules for large front-end requests! In this way, each end can rub against the preheated image resources of the other end.

Summarize

我们核心点就修正了URL改造SDWebImageManager , SDImageCache ,并且建立了CacheKey关联关系， 兼容一些老逻辑 In this way, the local process can be considered to go through. In addition to the conventional idea of optimizing images, this article provides a new idea, using the downloaded size images locally to make a fuss, thus accelerating and throttling, and achieving certain benefits, if readers also use similar stitching urls If the way of downloading pictures, this optimization method can be tried. It is inconvenient to display the specific values of the results obtained after all are completed. It is probably to increase the download speed by 50%, and at the same time, it can save a certain amount of CDN bandwidth, with an average daily saving of at least 10%.

This article is published from the NetEase Cloud Music technical team, and any form of reprinting of the article is prohibited without authorization. We recruit various technical positions all year round. If you are ready to change jobs and happen to like cloud music, then join us at grp.music-fe(at)corp.netease.com!