Youdao Dictionary Android Client Package Volume Optimization Road

1 Background

From the beginning of the mobile Internet, Youdao Dictionary has made users fall in love with translation, word search and learning with its small, fast and powerful image. With the continuous iteration of business and the continuous improvement of functions, Youdao Dictionary is no longer a simple word search software, but a comprehensive learning platform for users. We have explored services such as community, Q&A, live broadcast, and information flow, and currently also carry functions such as audio, video, courses, memorizing words, writing correction, and so on. The dictionary has developed into a comprehensive learning platform, and the original intention of being small and fast still guides us to continuously optimize the startup speed and package size.

After continuous performance optimization, our cold start time has been maintained within 3s of the industry standard level. Our main performance optimization work in the past quarter has focused on the optimization of the installation package size. After a series of efforts, our package size has been reduced by 23.7%, and the installation package size has been reduced from 177MB to 135MB, which is 42MB less overall.

Our analysis and implementation details are detailed below.

2 Analysis

Introduce the contents of the package volume and an overview of the optimization method

A general APK installation package contains the following directories and resources:

META-INF/ signature file

assets/ auxiliary resource files used by the program

res/ is not compiled into the resources.arsc resource file, usually a picture

lib/ depends on the library files of different native platforms

resource.arsc compiled copy, color value, size, theme and other resource indexes

classes.dex compiled code

AndroidMenifest.xml app's name, version, access rights, and referenced library file information

It can be seen that the major parts are assets/, lib/, res/, classes.dex and resources.arsc, which roughly correspond to resources, library files, code and resource indexes. Our main optimization ideas are as follows (the blue box part is the part that has been processed so far):

3 Technical Implementation Details

3.1 Image compression

In the process of APK packaging, the aapt tool will compress the image losslessly by default, but the default compression cannot achieve a good compression effect. After comparing the compression effect of webp and tinypng, we finally chose to use tinypng to compress the image. compression. And we wrote a compilation tool to automatically compress images.

lossy webp > tinypng > lossless webp

For example, this startup image, the original size is 724KB, and the quality is only 23.7KB when it is compressed to about 75%. There is a little difference in the effect, but it is acceptable. So can we compress all png images into lossy webp? The answer is no, take a look at the following example:

Before compression:

After compression:

It can be seen that under the same compression quality (75%), the image becomes very blurred. Even if the compression quality is selected to 99%, there will still be some stripes without natural transitions in the gradient area.

For the above situation, it is better to use the tinypng scheme

Original image: 643KB,

tinypng: 152KB,

webp: 339KB

To sum up, for lossy webp, it is impossible to find a fixed compression quality to suit all scenarios. Lossy webp is sometimes bigger than tinypng, but the display quality is worse.

We originally used Douyin's McImage plugin to process images, but there are some obvious problems with this solution:

1. The scheme adopts lossy webp, and lossy webp cannot determine a general compression quality suitable for all scenarios.
2. All graphs must be compressed each time they are packaged, which seriously affects the iterative efficiency. The baler takes 40 minutes and often OOM.
3. The images in the assets directory are not processed.

In response to the above problems, we have developed a set of automatic image compression tools using tinypng, and made the following adjustments:

1. For the large image png, manually press it into a lossy webp. High returns and manageable risks.
2. For non-large images, an image-optimization plugin was developed for compression. The plugin scheme is:

· png to tinypng. Although it is lossy, from the sampling point of view, there is no obvious change at all to the naked eye.

· Handling assets. There are front-end png images in assets. The advantage of converting tinypng instead of webp is that you do not need to change html, js and other files separately, and it is more compatible with low-version systems. The flutter_assets images of flutter-related projects are relatively large and do not pay attention to compression. The unified processing of plugins can be optimized and repackaged separately without opening the flutter project.

· For compressed images, do cache processing, do not need to recompress, and replace them dynamically when packaging. The compression cache is submitted to gitlab for unified management along with the dictionary project.

The following is a flow chart of our image automatic compression plugin processing:

The judgment of whether the compressed image is available here is mainly based on the size judgment. If the compressed image is larger than the original image, it will be discarded. This is the case with crunchPng compression, for example

Attachment 1: Because tinypng has been used for unified compression, it is recommended to close the crunchPng that comes with google, otherwise the packaging speed will be slower, and the optimized image may also become larger. Just add this line:

 buildTypes.all {\
  isCrunchPngs = false\
}

Additional 2: Comparison of lossless webp and tinypng

As shown in the figure, the full replacement of tinypng is 7.7MB less than the full replacement of webp (including assets). If you consider that the 14.7MB in assets cannot be simply changed to webp, the gap will be even greater.

Additional 3: Is tinypng already the best solution?

Refer to another ImageOptim tool, which combines several tools like OptiPNG, PNGCrush, AdvanceComp, PNGOUT, Jpegoptim + Jpegtran, and Gifsicle to provide the best optimization results, and is almost lossless. For a small part of the picture ImageOptim presses out small, it seems that there is no difference. However, the compression speed is very slow.

Therefore, if you do it to the extreme, you can perform a variety of compression schemes and choose the best image as a replacement. And our image-optimization plugin was designed with this extensibility in mind from the start.

Additional 4: AndResGuard optimization comparison

I tried it and the effect was not obvious, and some resources were lost and crashed. The reason why the effect is not obvious is that currently R8 also has obfuscated compression for resource names (proguard did not before), so the role of AndResGuard is relatively weak now. As for the compression of 7zip is not turned on, in theory, the startup speed will be slower, and I feel that the gain is not worth the loss (in addition, it will cause the Patch optimization algorithm of Google Pay to fail).

3.2 resources.arsc optimization

• Language pack optimization

Open the string of resources.arsc, we can see the following table, we will find a lot of empty places (as shown above). These blank places are actually occupied by FF FF FF... characters, which take up a lot of space (as shown in the figure below). Since Youdao Dictionary has no international translation (there is an international version called U-Dictionary, support is welcome), deleting unnecessary language versions will help reduce the volume.

 android {\
    defaultConfig {\
        resConfigs "zh"\
    }\
}

• As shown above, add a line and keep Chinese. The harvest is bigger than expected, directly reducing 3MB.
• The dimens optimization looked at the size of arsc in recent versions and found that one version increased by 5MB.
• In this version, we have made the tablet adaptation function. Since we use the SmallestWith qualifier adaptation scheme (you can learn about this screen adaptation scheme first), a large number of size resources are generated.

There are more than 3000 resources in total, and each resource has a total of 90 versions from "values-sw300dp" to "values-sw1200dp", which has a large room for optimization.

sqb_px_xx" is used for font adaptation, but the largest font used in the dictionary is "sqb_px_144", so the generation rules are optimized to reduce this type of resources.

After optimization, the number of resources has changed from 3012 to 1662, which is reduced by nearly half. Directly reduced by 2.5MB.

3.3 Deletion of business code

Since tools such as Proguard and lint analyze and cut code from the perspective of code reference, if some abandoned code is not deleted first, it will affect the effect of subsequent work. For some services that have been abandoned and have no entry, if they are not processed, the code and resources will only increase or decrease. Business deletion should be the first step in all package volume processes, otherwise the subsequent removal of useless resources, image compression, confusion and other effects will be discounted. If the time is limited, then deleting the most recent demand will be more profitable than deleting the demand in the ancient times. The reason is that the closer to the current project, the larger the picture resources, font resources, and the so library used (especially audio and video) .

This part of the work is mainly to sort out the business functions and communicate whether some outdated businesses can be deleted. In addition, detailed reference analysis is required to strip out and delete the code related to the abandoned business.

A good project structure is of great benefit to the stripping of business code in the future. At present, the newly developed functions adopt a hierarchical and sub-module organizational structure. There is no interdependence between functional modules, so it will be more convenient to separate or delete services in the future.

3.4 Delete useless resources

For the deletion of useless resources, we mainly use two methods. One is to use the lint tool to find the resources that may not be used in the application and to determine one by one to confirm that they are not used before deleting them. The second is to add shrinkResources to the build.gradle file to compile. Stages are removed using the R8 tool

 buildTypes {\
        release {\
            // Zipalign优化\
            zipAlignEnabled true\
            // 移除无用的resource文件\
            shrinkResources true\
            // 移除没用的代码\
            minifyEnabled true\
        }\
}

When using the lint tool, you need to pay attention to re-judging and confirming the following scenarios

1. For reflective reference resources, it may be identified as useless resources, such as the notification bar icon used by push
2. The layout resources used by DataBinding will be identified as useless resources

3.5 Compression Obfuscation

Use the R8 tool to compress and obfuscate the code during the compilation phase, so as to achieve the effect of compressing the installation package volume. Mainly divided into the following 4 steps:

1. Shrink removes unused classes, methods, fields, etc.;
2. Optimize (optimize) optimize bytecode, simplify code, etc.;
3. obfuscate renaming class names, method names, fields, etc. with short, meaningless names;
4. Preverify (preverify) Add preverification information to the class.

We introduced Proguard two years ago, but used the -dontobfuscate configuration to deobfuscate because of the problems caused by obfuscation. We found that the configuration of -dontoptimize inherited from the dependency library in the previous rule caused the optimization to not take effect. In this optimization, we comprehensively solved many problems caused by confusion, and fully opened optimization and confusion.

Since we have turned on compression before, the classes we need to use have been reserved in proguard. After the obfuscation is turned on, the following problems need to be dealt with:

• getIdentifier Gets the resource issue by name. If it is normal mode, the related resources will not be removed automatically:

• Check Resources.getValue related logic
• Check AssetManager.open related logic
• Reflect, search the reflection package globally, and modify the relevant location java.lang.reflect
• Handle the Retrofit error report ( https://github.com/square/retrofit/issues/3588 ), which is currently solved by upgrading the Gradle plugin version

 Caused by: java.lang.IllegalArgumentException: Method return type must not include a type variable or wildcard: ho8<su3<?>>\
    for method CheckInApi.popupConfig\
    at retrofit2.Utils.methodError(SourceFile:5)\
    at retrofit2.Utils.methodError(SourceFile:1)\
    at retrofit2.ServiceMethod.parseAnnotations(SourceFile:7)\
    at retrofit2.Retrofit.loadServiceMethod(SourceFile:4)\
    at retrofit2.Retrofit$1.invoke(SourceFile:6)\
    at java.lang.reflect.Proxy.invoke(Proxy.java:1006)\
    at $Proxy23.popupConfig(Unknown Source)\
    at com.youdao.dict.checkin.CheckInPopupManager.requestPopupConfig(SourceFile:3)\
    at java.lang.reflect.Method.invoke(Native Method)

Proguard's rules will greatly affect the effect of R8 on code compression and obfuscation, so reviewing and sorting out compression rules can help further volume compression.

3.6 Font optimization

This part of font optimization has been implemented in the previous version, and the effect obtained is quite obvious. Here is a supplementary explanation.

- Font cropping

The size of the general font library will be ten or twenty megabytes. However, only a small number of characters are actually used, so it is very beneficial to properly tailor the font library for the actual usage scenario.

List of common words: https://github.com/DavidSheh/CommonChineseCharacter

Font Compression Tool: https://github.com/forJrking/FontZip

- Font merge

Generally speaking, our development will be modularized, and different teams may use the same fonts when developing different functions. If you are not careful, it will be copied into two or three copies, which greatly increases the number of documents. The solution on the dictionary side is to sink the common fonts into the underlying core base library for reference by each module.

4 Outlook

After the above work, the volume of the current dictionary installation package has been optimized by 23.7%, and the overall volume has been reduced by 42MB. In the next Q2, we will prepare to do two things.

4.1 Packet volume monitoring

In the process of optimizing the package size, after we cut off a little bit of the volume, we turned around and found that other students casually threw in a large picture of a few MB. Therefore, how to stick to the fruits of victory and keep the bag volume in the best state has become the top priority.

The packaging task adds an option to check the package size limit (by default); after the merge request, the packaging task of the dictionary will trigger an automatic build;

After the packaging task is completed, if the package size needs to be checked, the apkcheck step is triggered; the details are as follows:

1. After the packaging task is completed, add a script operation to write the data of this build (such as apk file address, mapping file address, R text address, etc.) into a temporary file;
2. After the package task is built, add Trigger parameterized build on other projects to trigger the apk size check task;
3. Start the inspection process, the inspection process performs inspection tasks on the apk according to the parameters, and generates html for the task results;

4.2 Dynamic Distribution

- Overall business distribution

You can use technologies such as plug-in and dynamic loading, but these may not be the most difficult. The most difficult thing is how to extract some ancestral, low-frequency, and interdependent codes to form independent modules for distribution and dynamic loading. .

- Business sub-function distribution (expected to optimize 39.6MB)

1. Database (Word Lock Screen 8MB)

The word lock screen can retain hundreds of kilobytes of data locally for users to use, and download the complete thesaurus at the same time.

2. OCR Engine Data (22.5MB)

Users should be able to download the training model on demand instead of directly building it; when there is no training model, they can directly request the network.

3. Fonts (9.1MB)

In addition to high-frequency services such as word search, the fonts of low-frequency services can be dynamically distributed, and if there is, it can be displayed, and if not, the system can be used. However, the compatibility library of emoji is special, mainly used for posts in the homepage information flow, UGC posts, etc. If there is no compatible library, the user may display "tofu block" when encountering a special emoji. At this time, if the emoji font library has not been downloaded, it needs to be replaced. In addition, even if the user's system already has this emoji built-in font, the display effect may be different for each mobile phone. You need to confirm with the UI whether to replace it or temporarily display it like this.

Keeping the original intention of keeping the dictionary small, fast and powerful is the driving force for our continuous performance optimization. In the next work, we will continue to optimize and improve the startup speed, installation package size and memory usage. Welcome everyone Keep following and supporting!

5 References

• Reduce your app size
• Shrink, obfuscate, and optimize your app
• Douyin image compression plugin McImage
• Tencent package volume monitoring ApkChecker