How to reduce cloud music iOS package size by 87MB

The author of this article: Dapeng

Cloud Music iOS client is an old project that started in 2013. After nearly ten years of rolling business development, from the development of a single music APP to the present, it has become a giant APP similar to the platform level with the support of various services. With the development of the business, it becomes more and more bloated, which affects the actual experience of users, and even the reputation of the brand. Before the author began to optimize, the package size displayed by Cloud Music in the AppStore had reached as much as 420MB. In this case, the team Opened the special package size optimization.

Package size optimization is an old proposition of client-side development. Basically, as an iOS development student, you know more or less how to do it. However, with the development of Apple, some of the original feasible measures are no longer applicable in the new version, so this article The article focuses on some of the latest practical experience in the optimization process, and how it is implemented in large projects, so let's start without further ado.

caliber

Before starting to optimize, we first need to find out the various calibers of the package volume and the relationship between them, because some follow-up optimization measures will lead to the change of different calibers, so we must first determine what the final target caliber is. .
First of all, we can see the specific installation size and download size of our APP in the Apple background, and also include different model versions.

So how is the download size and installation size of Apple's background generated? Please see the figure below. After uploading, Apple officially unpacks the IPA package we uploaded, and then performs DRM encryption on the binary (this item will also cause the package Volume growth) and AppThinning, AppThinning will tailor the resources and code of the original package to different degrees according to different models, so as to generate versions suitable for specific models. In addition, Apple will generate a generic version that contains the complete set, but it is of no practical use. About DRM and AppThinning are not expanded here, there are links at the end of the article.

As shown above, we have three metrics before and after upload:

• APP original package size: The actual size of the APP after the IPA is unpacked before uploading
• Download size: The size of the prompt box when downloading traffic in the AppStore
• Installation volume: The size displayed in the APP details in the AppStore

After figuring out the relationship between various indicators, we finally chose the installation volume with the strongest user perception as the core indicator, and optimized it as the final goal.

analyze

Although the goal has been determined, before optimization, it is necessary to analyze the status quo and find the biggest deterioration point, so as to carry out targeted optimization and obtain the maximum profit ratio. Then the following figure is the basic situation of the cloud music iOS package. It can be seen that the red resources account for more than half of the volume, and the binary accounts for more than a quarter, so the focus of subsequent optimization can be placed on resources and binary source code.

resource

In fact, there are several conventional ways to deal with resources: resource cleaning, resource sorting, resource compression, resource cloud migration, resource consolidation, etc. In short, we are trying every means to reduce the local space occupied by resources. Let’s briefly introduce us. Work done in cloud music.

Resource cleanup

Before starting the overall resource optimization, the first step is to clean up the resources that are no longer in use, including pictures, configuration files, audio and video, etc. The main idea of detecting useless resources is to judge whether the resources are referenced through static detection, for example Use ImageName to determine whether a picture is used. Of course, the online detection method is more accurate, but it is not necessary for resources. However, as an old business, cloud music uses pictures in various poses. For example, the referenced file name is not Specifications, failure to use AssetCatelog, manual stitching of image names 2x3x, etc., this requires a slightly customized way to search, to exclude abnormal situations, other APPs can be adjusted according to their actual conditions, the ideas are the same, and there are ready-made tools on the Internet.

After several rounds of cleaning, Cloud Music has successively cleaned up 1200+ files of types such as pictures, and the original package volume of about 12+MB has been reduced, which is quite impressive.

Resource sorting

Resource sorting is actually to manage the right resources in the right way, here mainly refers to the Asset.car file. As we all know, Apple has launched the AssetCatelog file since iOS7 to help developers manage resources, the most important of which is the image resource. After compiling, the Asset.car file will be generated and put into the IPA package. As mentioned earlier, cloud music is an old project, so there are some resource images that are not managed by Asset, and using Asset will bring benefits to the package size, so It is necessary to migrate existing resources and use Assets for management; but here is a question, why does using Assets bring package volume benefits?

To answer the above questions, we must first start with the principle of Asset. In the compilation process of AssetCatelog, taking ImageSet type as an example, firstly, the ImageSet type images in Asset will be compressed losslessly, and multiple ImageSet images will be synthesized into one A big picture, so after compiling, it is impossible to read the picture by means of bundle path, you must use Apple's ImageName API, because it obtains specific picture information from the synthesized big picture by means of coordinates; The advantage of doing this is that there will be a gain in image volume during the process of compression and synthesis; but after our research, we found that not all images will have this benefit. Some large-volume images will be generated after lossless compression and synthesis. The volume is larger, we guess that this may be related to the synthesis of large images, and the smaller the image, the higher the possibility of profit.

In addition, it is best not to use the ImageSet type for animations, because there will be problems with animations during the process of compression and synthesis, resulting in incorrect data read through ImageName, which will cause problems such as unplayability; however, the type of DataSet can be used, DataSet does not participate in synthesis and compression, so it will not affect it. For other resource types, the way of DataSet can generally be used, and NSDataAsset can be used when reading. So how do you know the status of the resources processed by the Asset? You can use the following command to parse the compiled Asset.car to obtain the compiled information of the resources.

xcrun --sdk iphoneos assetutil --info Assets.car

As can be seen from the above figure, there is no compression for the resources of the Data type; for the Image type, the specific compression algorithm and some image information are marked. In addition, in the process, we also found that the compression algorithms used by Apple are different for different pictures, and they will be compressed into multiple copies, which is why after we moved some resources from the bundle to the AssetCatalog, the IPA volume still changed. The reason is big, but it doesn’t matter, the installation volume will decrease, because the biggest benefit of using Asset actually comes from Apple’s AppThinning mentioned above. Apple’s slimming mechanism will adjust Asset.car according to different models. For distribution, for example, 1x2x3x has different device models, so although it will be compressed into multiple copies, each machine actually uses only one copy, which is why even if the IPA becomes larger, the installation volume will actually change. little.

The last thing I want to say is that because there is no way to compile Assets one by one to compare the size before and after compilation, from the perspective of probability, it is more recommended that small images (within 5k) and images with multiple versions (2x3x) be put into AssetCatalog management, other In fact, it is more free to store resources separately, rather than using DataSet, because it is more convenient to use various compression methods without worrying about being affected by Apple's processing, which will be discussed in detail in the subsequent resource compression.

After this optimization, Cloud Music iOS client has migrated 2400+ image resources of various sizes, achieving an installation volume gain of 22+MB.

resource compression

Resource compression is well understood. As the name implies, it is to compress resources in various ways. The most important resource in cloud music is pictures, and other types account for a small proportion. Common picture resource formats are mainly png, apng, webp, etc. Most of the pictures in the music package are also in the above formats; because after the previous work, almost all pictures are managed in AssetCatalog, and as mentioned above, Apple will compress the picture resources of AssetCatalog without loss, so if we ourselves The lossless compression applied to the image resource has no effect, because Apple will press it again, and the final result will be subject to him. Therefore, in order to get the optimization result in compression, it is necessary to substantially reduce the size of the image, so lossy compression must be done. For conventional image formats, we use pngquant, tinypng and other algorithms and tools for compression. When using pngquant, after testing with large data samples, we finally choose a lossy ratio of 80%, because at this time the ratio is the highest in the yield curve and relatively When the image quality is less affected, the curve may be different for different projects, because the actual resource situation of each project is different, so you need to obtain the data of the project yourself. The specific method can be scripted. To try different compression ratios and record the compression results to form a graph. In addition, there are many leftover large webp animations in our package, which cannot be compressed in general methods. After a certain research, we finally found that Google officially provides Webpmux to disassemble and compress webp animations frame by frame and Synthesis, based on this, we wrote a script that can compress webp animations, and realized the compression of webp animations. Finally, we integrated the image compression capabilities of all common formats into a large script to compress all the image resources in the package. This script is also useful for subsequent anti-degradation.

After this, the overall compression of each size is 5000+ png images, 100+ apng animations, 100+ webp animations, and the overall revenue is 42+MB (original package volume).

Resource cloud migration

After cleaning, sorting, and compressing, the resource part still occupies a lot of package volume, so we started a special project for cloud migration of large resources. The reason for large resources is that the benefits brought by large resources are the highest. After discussion, combined with The actual situation of cloud music has finally set a baseline of 50kb, and if it is larger than 50kb, it will be defined as a large resource. It is not that we have not considered the solution of unified resource migration and unified download, but considering the experience and cost of cloud music, we finally choose to deal with the high ROI part in the traditional way. After screening, there are 150+ cases in the cloud music package that meet the situation of large resources, and more than 85% of them can be migrated to the cloud. As for whether resources should be placed locally or in the cloud, we and design students have jointly formulated specifications for the use of relevant resource pictures/animations, and technical students will judge purely technical resources.

After the migration project is completed, 100+ large resources have been migrated, and the revenue is about 31+MB (original package volume).

Resource Consolidation

There are actually two main points for resource merging. One is the deduplication of a single similar image. We have spent a certain amount of effort using the analysis algorithm of similar images to detect all the resource images of Cloud Music. The results did not match our expectations. There are not many similar pictures, including icons, this part is not profitable. The other is the merger of AssetCatalog. Combined with the actual situation of cloud music, there is no benefit from this, mainly because the resources of cloud music are currently managed in a centralized manner.

binary

Each APP program will eventually be compiled into a main binary file, and all static library dependencies will be linked in. The size of this part is mainly affected by the amount of code and compilation parameters. The following optimization ideas also focus on reducing the amount of code and optimizing compilation. parameter.

Dead code detection

If you want to reduce the amount of code, the first thing that comes to your mind is to clean up useless code, so which code is useless? This has useless code detection. The general detection methods are divided into online dynamic detection and offline static detection. The accuracy of dynamic detection is much higher than that of static detection, and the static code compiler already supports some cutting methods, such as DeadCode. Optimization; then based on this, we have adopted a more accurate online big data dynamic detection, the only disadvantage is that the data acquisition cycle is long, and it needs to be run online.

Initially, our idea was to use the hook class initialization method + initialize to determine whether a class is used, but this scheme has several problems: the first is the problem of startup timing, because we use AB sampling, then it must be initialized in AB If it is turned on at a later point in time, the class before AB initialization cannot be recorded unless all users record it, only sampling when uploading, but this will affect users who are not grayscale; the second is +initialize itself The timing of calling The problem is that not all classes' +initialize will be called. After that, we adopted another scheme. In OBJC, each class has its own metadata, and a flag bit in the metadata stores whether it is initialized or not. This flag bit is not affected by any factors, as long as it is The initialization will be marked, and the way to obtain the mark bit in the source code of objc is as follows:

struct objc_class : objc_object {
    bool isInitialized() {
        return getMeta()->data()->flags & RW_INITIALIZED;
    }
}

But this method APP cannot be called directly, it is a method of objc; but it does not mean that the data of the flag bit RW_INITIALIZED does not exist, the data is still there, so we can simulate it through the existing interface and the source code information that can be read The above code, to obtain the flag bit data to determine whether a class is initialized, the code is as follows:

#define FAST_DATA_MASK  0x00007ffffffffff8UL
#define RW_INITIALIZED  (1<<29)
- (BOOL)isUsedClass:(NSString *)cls {
    Class metaCls = objc_getMetaClass(cls.UTF8String);
    if (metaCls) {
        uint64_t *bits = (__bridge void *)metaCls + 32;
        uint32_t *data = (uint32_t *)(*bits & FAST_DATA_MASK);
        if ((*data & RW_INITIALIZED) > 0) {
            return YES;
        }
    }
    return NO;
}

Through the above simulation code, I can get whether a certain class is used, and after reporting the information, I can analyze which classes can be cleaned based on big data. The classes used, but these do not mean that they can actually be cleaned up. For example, some are doing AB, some are pre-embedding business, etc., so the data results need to be filtered again on the business side. In the end, we processed 1200+ classes and succeeded After cleaning 300+, the income is about 2+MB (all calibers in the binary chapter are the original package caliber), and the remaining unprocessed ones are still being processed and optimized as a long line.

Second and third party library offline

Based on the unused class data above, business components or second- or third-party libraries that are no longer in use can be obtained through cluster analysis. During the optimization process, we have identified several second- and third-party libraries that can be offline, and the income is 4+ MB.

Dynamic library dependency clipping

In addition to the processing of business codes, Cloud Music itself also relies on some dynamic libraries, and some static dependencies of these dynamic libraries are repeated due to diachronic reasons, as shown in the following figure:

This is an extreme case. There is an OpenSSL symbol in the main program, in the dynamic library A, and in the dynamic library B, respectively, so this will cause repetition and occupy the binary volume; then the best solution to this problem is It is to move to static, convert the dynamic library into a static library, link it in the main program, remove the original dependency, and use the Symbol in the main binary, which can also improve the startup speed to a certain extent, because it reduces the number of dynamic libraries. . By solving a problem like this, the overall gain is 3+mb.

Compile optimization

After optimizing the tailoring code in various ways, it is time to start optimizing another factor that affects the size of the binary, which is the compilation parameters. There are many compilation parameters, which can be divided into compile-time parameters and link-time parameters. Next, I will sort out the basic All parameters that will affect the binary volume are for the reader's reference

Asset Catalog Compiler Optimization

Asset compilation optimization can reduce the volume of Asset.car products. Before this cloud music, only the main project was opened, and the compilation parameters of components were not opened. After optimization, the income was less than 2.1MB.

EXPORTED_SYMBOLS_FILE

For the APP, it can be regarded as a large "dynamic library". When the user clicks to open the APP, the system starts to load the dynamic library. Then the dynamic library will always have externally exposed symbols, namely Exported Symbols, but for the APP Generally speaking, it will not be called elsewhere in the iOS system. It is more that the APP calls the system's services, so we can drop the Exported Symbols to Trim. Fortunately, the compiler provides EXPORTED_SYMBOLS_FILE that allows us to limit the symbols that are exported. , so as to reduce the size of the binary; the specific method is to create a new txt file, put it in the project directory (only the project directory, no need to add it to the xcode project, it will become a resource and affect the size of the package), and point EXPORTED_SYMBOLS_FILE to this file, then if it is If the file is empty, all the exported symbol segments will be cut off. By specifying the specific symbols to be left in the txt file, the compiler will cut off the undeclared part.

The picture below shows the symbol segment being cut out after it is turned on

It is worth noting that if the APP uses Firebase, it cannot be completely cropped, which will cause Firebase to fail to start successfully, and thus unable to obtain Crash information. The reason is that Firebase relies on the __mh_execute_header symbol in the Export Info above, so it can be listed above. If __mh_execute_header is added to the txt file mentioned in the article, the compiler will retain the part of __mh_execute_header when cutting.

This item is optimized during the linking period, as long as the main project is turned on, after Cloud Music is turned on, this income is 2.4MB.

Link-Time Optimization

The optimization of LTO is mainly reflected in the optimization of discarded code cutting across files, the optimization of empty logic that will never be executed, and the optimization of inline, which means directly copying functions, reducing inline levels, and improving the execution efficiency and space utilization of function stacks. For details, please refer to LLVM's official document , which will not be repeated here.

In addition, it has been tested and verified that LTO is only valid for static languages. OC is a dynamic language. All functions and methods may be dynamically called at runtime, so it is impossible to tailor. This is why when linking a static library, if it is a C library, So it seems that the original binary is very large. In fact, only a small part of the actual use is actually linked in, but if it is an OC library, basically all of it will be linked. So if you have more C or C++ code in your APP source code, you may get more benefit in this item.

Although the name of LTO seems to be link-time optimization, it actually needs to be involved at compile-time, otherwise it will have no effect. This is related to cross-file optimization, and some information must be generated during compile-time to provide link-time optimization.

After LTO optimization, the revenue obtained by Cloud Music is 1MB.

GCC_OPTIMIZATION_LEVEL

This is intended to generate lower binary products through more aggressive GCC compilation optimization. Xcode defaults to Debug setting O0 and Release setting to Os, but in fact, Oz mode can also be used to achieve a smaller size.

In fact, the principle of Oz is just the opposite of the above LTO idea of inline or outline. Oz wants to reduce the inline level of the function through more outlines, but this will be the function of the function. The call stack becomes very deep, which will reduce the execution efficiency of the function. As shown in the above figure, it will become relatively "slow". In fact, it is also a game of time and space in essence; in addition, if you want to enable this item, you can refer to the Douyin article. , they have encountered some problems with objc_retainAutoreleaseReturnValue, but so far, we have not found it in the process of actual practice, but based on stability considerations, this item has not yet been launched in cloud music, but only in the debug environment for testing. , is still under observation.
If this option is enabled, the estimated revenue after testing is around 10+MB.

Other compilation optimizations

• Enable C++ Exceptions and Enable Objective-C Exceptions, closing this option can bring benefits in binary volume, but it will affect TryCatch, use it as appropriate, cloud music is not enabled
• Architectures, architectural instruction set, this part needs to pay attention to whether some two or three-party Frameworks contain unnecessary instruction sets

• Strip Symbols, related to clipping symbols, not expanded here, and related settings below

  • Strip Linked Product = YES
  • Strip Style = All Symbols, Note: This setting does not take effect when Strip Linked Product is not enabled
  • Deployment Postprocessing Note: No matter how this item is set in packaging, Apple will set it to YES by default
• Symbols Hidden by Default = YES, set symbol visibility
• Make Strings Read-Only = YES
• Dead Code Stripping = YES, compile-time detection determines that no code is used for clipping
• Optimization Level, generally debug is set to None, Release is set to Os

binary summary

In addition to the above measures to optimize binary, there are actually many other measures in the industry, but cloud music has not adopted it for various reasons, such as renaming the _Text code segment, thereby bypassing Apple's DRM encryption to reduce Binary size, but after iOS13, Apple has realized this problem and solved it to a certain extent, so this optimization method has basically failed; there is also binary segment compression, from the perspective of risk and benefit, it is not yet available. Use; there is also attribute dynamic, mainly for dynamic optimization of model attributes with a large number of attributes, dynamically adding get/set methods, so as to obtain the benefit of omitting this part of the method, the income estimate is also very small, so it is not used. . In fact, in summary, there are many optimization methods, but for a specific APP, you can choose the most appropriate measures according to the actual situation.

Anti-deterioration

In the process of optimization, we found that the actual deterioration rate of the project is also very fast, even reaching 40%~50% of the optimization amount of each iteration. That is to say, we assume that an iteration optimizes 10MB, but the deterioration of this iteration reaches 4~5MB, so we have to start anti-deterioration work at the same time as governance. We have formulated some anti-deterioration measures, some of which have been launched, and the rest are still under development. So far, good results have been achieved. The deterioration has been effectively curbed, and the optimization results have been preserved. The specific measures are as follows:

• Large resource bayonet: perform resource detection when code is merged, and force bayonet
• Second-party library Third-party library bayonet: check the third-party library and third-party library when code is merged, including adding and upgrading
• Automatic compression: Automatic compression is performed for resource integration, but the first recommendation is to place it at the remote end, and then place it locally if it is very necessary.
• Regular resource situation detection: Regularly automate the resource survey of the whole APP, and trace the problem
• Regular code inspection: Regular and automated code inspection of the entire APP, and useless code is offline
• Cooperate with UED to launch a specification for the use of picture animation animation resources, which stipulates which can be local and which must be remote, and the optimization scheme of animation effect

result

After a period of various optimizations, the installation volume of Cloud Music has dropped by 87MB, from the original 420MB+ to 330MB+ now. The overall sense is still different. The download volume has dropped by 65MB, breaking the Apple OTA limit of 200MB, reaching 160+MB.

Relevant information

• What is app thinning?
• Asset Catalog Format Reference
• pngquant
• webpmux
• Code Size Performance Guidelines
• From Exported Symbols applied to package size optimization to symbol binding
• LLVM Link Time Optimization
• Reducing Code Size Using Outlining
• Interprocedural MIR-level outlining pass

This article is published from the NetEase Cloud Music technical team, and any form of reprinting of the article is prohibited without authorization. We recruit various technical positions all year round. If you are ready to change jobs and happen to like cloud music, then join us at grp.music-fe(at)corp.netease.com!