Android agp inline support for R files

Author of this article: Zheng Chao

background

Recently, the team upgraded the static code detection capabilities, and the related compilation detection capabilities that they depend on need to use a newer agp, and the current cloud music agp version uses 3.5.0, which has a big gap compared with the current 4.2.0, so we focus on agp There was an upgrade. Through the official documents before the upgrade, it was found that the processing methods of R files were upgraded in agp3.6.0 and 4.1.0 respectively. The specific upgrades are as follows.

Agp 3.6.0 changes

Simplified R class generation

The Android Gradle plugin simplifies the compile classpath by generating only one R class for each library module in your project and sharing those R classes with other module dependencies. This optimization should result in faster builds, but it requires that you keep the following in mind:

• Because the compiler shares R classes with upstream module dependencies, it’s important that each module in your project uses a unique package name.
• The visibility of a library's R class to other project dependencies is determined by the configuration used to include the library as a dependency. For example, if Library A includes Library B as an 'api' dependency, Library A and other libraries that depend on Library A have access to Library B's R class. However, other libraries might not have access to Library B's R class If Library A uses the implementation dependency configuration. To learn more, read about dependency configurations.

Understanding agp3.6.0 literally simplifies the generation process of R. Each module directly generates R.class (before 3.6.0, the process of R.class generation was to generate R.java for each module -> then generate R.class through javac, Now it is unnecessary to generate R.java and generate R.class through javac)

Now let's verify this result, build a project, the android library module will be built in the project. Compile with agp3.5.0 and agp3.6.0 respectively, and then look at the build product.

The build products of agp 3.5.0 are as follows:

The agp 3.6.0 build products are as follows:

From the perspective of the build product, this conclusion is also verified. Agp 3.5.0 to 3.6.0 improves the generation efficiency of R by reducing the intermediate process of R generation (first generate R.java and then generate R.class through javac to directly generate R.class);

The agp 4.1.0 upgrade is as follows:

App size significantly reduced for apps using code shrinking

Starting with this release, fields from R classes are no longer kept by default, which may result in significant APK size savings for apps that enable code shrinking. This should not result in a behavior change unless you are accessing R classes by reflection, in which case it is necessary to add keep rules for those R classes.

From the title, the apk package has been significantly reduced (this is too attractive). Through the following description, it roughly means that the keep rule of R is no longer retained, that is, the R file is no longer included in the app? (How about reducing the size of the package)

Before analyzing this result, let me introduce the problem of R file redundancy in the apk;

R file redundancy problem

Android started from ADT 14 in order to resolve id conflicts in R files in multiple libraries, so the R in the library was changed to a static non-constant attribute.

In the process of apk packaging, the R file in the module is generated by accumulatively superimposing the R of the dependent library. If our app structure is as follows:

The R files generated by each module when compiling and packaging are as follows:

1. R_lib1 = R_lib1;
2. R_lib2 = R_lib2;
3. R_lib3 = R_lib3;
4. R_biz1 = R_lib1 + R_lib2 + R_lib3 + R_biz1 (R of biz1 itself)
5. R_biz2 = R_lib2 + R_lib3 + R_biz2 (R in biz2 itself)
6. R_app = R_lib1 + R_lib2 + R_lib3 + R_biz1 + R_biz2 + R_app (app itself R)

In the final apk, except for R_app (because the R in the app is a constant, the R reference in the javac phase will be replaced with a constant, so when the release is confused, the R file in the app will be shrinked), the rest of the R All files will be put into the apk package. This is the origin of R file redundancy in apk. And if the project depends on more levels, the more upper-level business components will lead to the rapid expansion of the R file in the apk.

R file inline (to solve the redundancy problem)

The redundancy problem caused by the system will not confuse smart programmers. There are already some R file inline solutions in the industry. The general idea is as follows:

Since R_app includes all dependent R, you can customize a transform to change all the R references in the library module to the attribute references in R_app, and then delete the R files in all dependent libraries. In this way, there is only one top-level R file in the app. (This approach is not very thorough. There is still a top-level R in the apk. More thoroughly, you can replace all references to R in the code with constants, and delete the top-level R in the apk)

agp 4.1.0 R file inline

First of all, we used agp 4.1.0 and agp 3.6.0 to build apk for a comparison, and from the final product to confirm whether the R file inlining has been done.
The test project has made some configurations that are easy to analyze, and the configurations are as follows:

1. Turn on proguard

   buildTypes {
    release {
        minifyEnabled true // 打开
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
    }
   }

2. Turn off obfuscation, only keep compression and optimization (to avoid the identification problems caused by opening obfuscation)

   // proguard-rules.pro中配置
   -dontobfuscate

   Build the release package.
First look at the apk generated by agp 3.6.0:

From the figure, you can see that there will be an R file in the bizlib SecondActivity , you will find that there is a reference to the R file inside.

Then look at the apk generated by agp 4.1.0:

As you can see, there is no R file in the bizlib SecondActivity , you will find that the internal reference has become a constant.

From this, it can be determined that agp 4.1.0 did inline R files and did it very thoroughly. It not only deleted redundant R files, but also changed all references to R files to constants.

detailed analysis

Now let's analyze in detail how agp 4.1.0 achieves R inlining. First, let's roughly analyze the inlining of R. Basically, we can guess that it is done in the process of class to dex. After determining the general stage, then we can see if we can narrow down the corresponding scope from the construction product, and it is best to be accurate to the specific task. (Digression: Analyzing and compiling-related issues are generally four-fold: 1. First analyze the corresponding results from the app's build product; 2. When it comes to dependency analysis, you can print out all the input and output of all tasks; 3. 1, 2 When it is not enough, we will consider looking at the corresponding source code; 4. The final big trick is to debug the compilation process;)

First, let's look at the dex in the build product, as shown below:

Next, add all task input and output printing gradle scripts in the app module to assist in the analysis. The relevant scripts are as follows:

gradle.taskGraph.afterTask { task ->
    try {
        println("---- task name:" + task.name)
        println("-------- inputs:")
        task.inputs.files.each { it ->
            println(it.absolutePath)
        }
        println("-------- outputs:")
        task.outputs.files.each { it ->
            println(it.absolutePath)
        }
    } catch (Exception e) {

    }
}

minifyReleaseWithR8 corresponding input and output of 060f69eb120693 are as follows:

It can be seen from the figure that the input is a collection of R files (R.jar) of the entire app, so it is basically clear that R inlining is minifyReleaseWithR8 task.

Next, we will analyze this task in detail.
The specific logic is in R8Task.kt .

Create the minifyReleaseWithR8 task code as follows:

class CreationAction(
        creationConfig: BaseCreationConfig,
        isTestApplication: Boolean = false
    ) : ProguardConfigurableTask.CreationAction<R8Task, BaseCreationConfig>(creationConfig, isTestApplication) {
        override val type = R8Task::class.java
        // 创建 minifyReleaseWithR8 task
        override val name =  computeTaskName("minify", "WithR8")
    .....
}

The task execution process is as follows (due to too much code, only some key nodes are posted below):

    // 1. 第一步，task 具体执行
    override fun doTaskAction() {
    ......
        // 执行 shrink 操作
        shrink(
            bootClasspath = bootClasspath.toList(),
            minSdkVersion = minSdkVersion.get(),
            ......
        )
    }
    
    // 2. 第二步，调用 shrink 方法，主要做一些输入参数和配置项目的准备
    companion object {
        fun shrink(
            bootClasspath: List<File>,
            ......
        ) {
            ......
            // 调用 r8Tool.kt 中的顶层方法，runR8
            runR8(
                filterMissingFiles(classes, logger),
                output.toPath(),
                ......
            )
        }
    // 3. 第三步,调用 R8 工具类，执行混淆、优化、脱糖、class to dex 等一系列操作
    fun runR8(
        inputClasses: Collection<Path>,
        ......
    ) {
        ......
        ClassFileProviderFactory(libraries).use { libraryClasses ->
            ClassFileProviderFactory(classpath).use { classpathClasses ->
                r8CommandBuilder.addLibraryResourceProvider(libraryClasses.orderedProvider)
                r8CommandBuilder.addClasspathResourceProvider(classpathClasses.orderedProvider)
                // 调用 R8 工具类中的run方法
                R8.run(r8CommandBuilder.build())
            }
        }
    }

So far we can know that in fact, in agp 4.1.0, the R file is inlined through R8. How does R8 do it? Here is a brief description, no more specific code analysis:

In terms of capabilities, R8 includes Proguard and D8 (java desugar, dx, multidex), that is, the process from class to dex, and in this process, it does things such as desugar, Proguard and multidex. When R8 shrinks and optimizes the code, the reference to the constant in the code is replaced with a constant value. In this way, there will be no reference to the R file in the code, so the R file will be deleted when shrinking.
Of course, to achieve this effect, agp also needs to make some adjustments to the default keep rules in version 4.1.0. In 4.1.0, the default keep rules for R . The corresponding rules are as follows:

-keepclassmembers class **.R$* {
   public static <fields>;
}

to sum up

1. Judging from the history of agp's processing of R files, the android compilation team has been continuously optimizing the generation process of R files, and has completely solved the problem of R file redundancy in the agp 4.1.0 version.

2. Compile-related problems analysis ideas:

   1. First analyze the corresponding results from the build product of the app;
   2. When it comes to dependency analysis, you can print out all the input and output of all tasks;
   3. If 1, 2 cannot be satisfied, we will consider looking at the corresponding source code;
   4. The last big trick is to debug the compilation process;
3. Judging from the effect of this agp upgrade of the cloud music app, the size of the app has been reduced by nearly 7M, and the compilation speed has also been greatly improved, especially the release speed is 10 minutes + (task merger), and the overall benefits are quite considerable.

test project used in the article;

Reference

1. Shrink, obfuscate, and optimize your app
2. r8
3. Android Gradle plugin release notes

This article was published from big front-end team of NetEase Cloud Music . Any form of reprinting of the article is prohibited without authorization. We recruit front-end, iOS, and Android all year round. If you are ready to change jobs and you happen to like cloud music, then join us at grp.music-fe(at)corp.netease.com!