AGP upgrade journey

The author of this article: Lvkan (xjy2061)

origin

Recently, due to the introduction of new tools that rely on Android Gradle Plugin (hereinafter abbreviated as AGP) version 4.1 or above, and the AGP version currently used by the project is 3.5.0, it needs to be upgraded. Considering that some third-party libraries have not yet provided support for the latest AGP 4.2 version, it was decided to upgrade AGP to 4.1, the highest version 4.1.3, which embarked on this AGP upgrade journey.

Adapt according to official documents

The first step of the upgrade is of course to read the official Android Gradle plug-in version description document, and adapt according to the version changes listed in the document.

AGP 3.6 adaptation

AGP 3.6 introduces the following behavior changes:

By default, native libraries are packaged in uncompressed form

This change makes the native library (native library) packaged in an uncompressed manner, which will increase the size of the APK and bring limited benefits, and some of the benefits rely on Google Play. If it is deemed that the AndroidManifest.xml outweigh the benefits after the evaluation, the following configuration can be added to 0612efad644417. Compress native library:

<application
    android:extractNativeLibs="true"
    ... >
</application>

AGP 4.0 adaptation

AGP 4.0 introduces the following new features:

Dependency metadata

This change will compress and encrypt the metadata of application dependencies and store them in the APK signature block. Google Play will use these dependencies to remind problems. The revenue is limited, but the APK size will increase. If the app is not available on Google Play, it can be Add the following configuration in build.gradle to turn off this feature:

android {
    dependenciesInfo {
        // Disables dependency metadata when building APKs.
        includeInApk = false
        // Disables dependency metadata when building Android App Bundles.
        includeInBundle = false
    }
}

AGP 4.1 adaptation

AGP 4.1 introduces the following behavior changes:

Removed the version attribute from the BuildConfig class in the library project

This change VERSION_NAME and VERSION_CODE fields BuildConfig category of the library module. Generally speaking, to obtain the version number in the library module is to obtain the version number of the App, and BuildConfig.VERSION_NAME and BuildConfig.VERSION_CODE in the library module are the version numbers of the library module itself. At this time, these two fields in the library module should not be used. You can use the following Code to get the version number of the App in the library module:

private var appVersionName: String = ""
private var appVersionCode: Int = 0

fun getAppVersionName(context: Context): String {
  if (appVersionName.isNotEmpty()) return appVersionName
  return runCatching {
    context.packageManager.getPackageInfo(context.packageName, 0).versionName.also {
      appVersionName = it
    }
  }.getOrDefault("")
}

fun getAppVersionCode(context: Context): Int {
  if (appVersionCode > 0) return appVersionCode
  return runCatching {
    PackageInfoCompat.getLongVersionCode(
      context.packageManager.getPackageInfo(context.packageName, 0)
    ).toInt().also { appVersionCode = it }
  }.getOrDefault(0)
}

Problems encountered

After adapting according to the official documents, unexpectedly, many problems were encountered. These problems were partly caused by behavior changes that were not clearly indicated in the official documents, and partly caused by irregular practices that hit the stricter restrictions of the new version of AGP. Introduce the performance, cause analysis and solutions of these problems.

BuildConfig.APPLICATION_ID not found

BuildConfig.APPLICATION_ID field is used in some of our component library modules, and the Unresolved reference error appears when compiling.

The reason is that the BuildConfig.APPLICATION_ID field name in the library module is ambiguous. Its value is the package name of the library module, not the package name of the application. Therefore, this field has been obsoleted since AGP 3.5 and replaced with the LIBRARY_PACKAGE_NAME field, and it has been completely removed since AGP 4.0 delete.

Our original part of the code in the App module uses APPLICATION_ID obtain the App package name. When extracting the code in the App module to the component library in the subsequent componentization and splitting process, in order to avoid incorrectly using the package name of the library module as the App package Name, the method of obtaining the App package name should be modified synchronously, but it was omitted and not modified, which caused the compilation to fail after this AGP upgrade.

To solve this problem, change the method of obtaining the App package name in the library module to the Context.getPackageName() method.

R and ProGuard mapping file cannot be found

We will back up the R and ProGuard mapping files generated when building the release package for later use. The backup fails after the upgrade.

This is because starting from AGP 3.6, the paths of these two files in the build product will change:

• R.txt：build/intermediates/symbols/${variant.dirName}/R.txt -> build/intermediates/runtime_symbol_list/${variant.name}/R.txt
• mapping.txt：build/outputs/mapping/${variant.dirName}/mapping.txt -> build/outputs/mapping/${variant.name}/mapping.txt

Where ${variant.dirName} is $flavor/$buildType (for example, full/release), and ${variant.name} is $flavor${buildType.capitalize()} (for example, fullRelease).

This problem can be solved by modifying the file path in the backup logic to the above new path as follows:

afterEvaluate {
    android.applicationVariants.all { variant ->
        def variantName = variant.name
        def variantCapName = variant.name.capitalize()

        def assembleTask = tasks.findByName("assemble${variantCapName}")
        assembleTask.doLast {
            copy {
                from "${buildDir}/outputs/mapping/${variantName}/mapping.txt"
                from "${buildDir}/intermediates/runtime_symbol_list/${variantName}/R.txt"
                into backPath
            }
        }
    }
}

Fixed resource id is invalid

In order to avoid the possibility of inflate notification after App upgrade overwriting installation, such as RemoteView , when the wrong resource file is found through the resource id, which causes the problem of a crash, we fixed the resource id during the construction, so that the id of some resource files is multiple times It is always the same between builds, and this part of the resource id has changed after the upgrade.

The original fixed resource id was implemented after afterEvaluate , using the tasks.findByName method to obtain the process${variant.name.capitalize()}Resouces (such as processFullReleaseResources) task object, and then before AGP 3.5, using the getAaptOptions method, using reflection in AGP 3.5 to obtain the aaptOptions attribute object in the task object , And then add the --stable-ids parameter and the corresponding resource id configuration file path value additionalParameters But in AGP 4.1, the processing resource task class no longer has the aaptOptions attribute, resulting in a fixed failure.

For AGP 4.1, you can change to the following android.aaptOptions.additionalParameters to fix the resource id:

afterEvaluate {
    def additionalParams = android.aaptOptions.additionalParameters
    if (additionalParams == null) {
        additionalParams = new ArrayList<>()
        android.aaptOptions.additionalParameters = additionalParams
    }
    def index = additionalParams.indexOf("--stable-ids")
    if (index > -1) {
        additionalParams.removeAt(index)
        additionalParams.removeAt(index)
    }
    additionalParams.add("--stable-ids")
    additionalParams.add("${your stable ids file path}")
}

Manifest file modification failed

We will modify the AndroidManifest.xml file to add additional information during the build process, and the modification fails after the upgrade.

After analyzing the build logs of each version of AGP included in this upgrade, it was found that AGP 4.1 added process${variant.name.capitalize()}ManifestForPackage (for example, processFullReleaseManifestForPackage) task process${variant.name.capitalize()}Manifest (for example, processFullReleaseManifest), and its product was different from the original mission. The original way to add additional information to the Manifest is to execute a custom Manifest processing task cmProcess${variant.name.capitalize()}Manifest (such as cmProcessFullReleaseManifest) after the original Manifest processing task is executed, and write information to the product of the original Manifest processing task ² After the upgrade, if both of the two Manifest tasks hit the cache (the execution status is FROM-CACHE ), then the extra information in the Manifest file in the final APK will be the old information written in the previous compilation.

Therefore, the method of writing information should be as shown in the figure below, instead of writing information to its product file after the newly added Manifest processing task is executed.

Transform plugin execution failed

We added some Transform plug-ins during the build process. After the upgrade, one of the plug-ins that used ASM for code instrumentation encountered the following error during execution:

Execution failed for task ':app:transformClassesWithxxx'.
> java.lang.ArrayIndexOutOfBoundsException (no error message)

The exception in the above error message may also be java.lang.IllegalArgumentException: Invalid opcode 169 .

In order to find the specific source of the exception, we added the --stacktrace parameter to rebuild. The location of the exception is triggered by the third-party library Hunter introduced in the plug-in. The ASM used by this plug-in is built-in with AGP, AGP 3.5 uses ASM 6, and since AGP 3.6, ASM 7 is used. It should be that the introduced Hunter has defects on ASM 7, causing abnormalities after the upgrade.

Considering that Hunter only encapsulated the Transform using ASM, and the functions implemented by this plug-in are relatively simple, the problem was solved by removing Hunter and re-implementing it.

Cannot change dependencies of dependency configuration

We used resolutionStrategy.dependencySubstitution to switch the component library source code. After the upgrade, if the component library is cut into the source code, the error like the title will appear when you click the Run button to build in Android Studio.

In the process of troubleshooting, it was found that executing ./gradlew assembleRelease on the command line can be successfully built, and the difference between building through Android Studio Run and the above command line building is that the module prefix ( :app:assembleRelease ) is added before the task executed. Starting from this difference, the cause of the problem was finally found to be gradle.properties was turned on in org.gradle.configureondemand , so that gradle only configures the project related to the requested task. As a result, when the task is executed in the specified module mode, the project that is cut as the source code is not Configuration.

This problem can be solved by turning off the org.gradle.configureondemand

Entry name 'xxx' collided

After the upgrade is built, the package${variant.name.capitalize()} (such as packageFullRelease).

According to the official documentation, AGP 3.6 introduces the following new features:

New default packaging tool

This feature will use the new packaging tool zipflinger to build the APK when building the debug version, and starting from AGP 4.1, this new packaging tool will also be used when building the release version.

The error occurred in the task of packaging and generating APK, it is easy to associate with the above new features. Use the method provided by the official document to add the android.useNewApkCreator=false gradle.properties file. After using the old packaging tool, it can be successfully built. However, the missing Java resource files in the generated APK caused various problems during runtime (for example, OkHttp lacks the publicsuffixes.gz file, which caused the request to never return).

Now there are two directions to solve the problem: solve the problem of missing Java resource files and solve the problem of building errors. In order to solve these problems, it is necessary to analyze the cause of the problem first. Through debugging the AGP construction process and analyzing the AGP source code, it is found that the implementation class corresponding to the packaging task is PackageApplication . The main implementation logic is in its parent class PackageAndroidArtifact . Write Android and APK files to the APK file. The calling process of Java resource files is shown in the following figure:

updateSingleEntryJars method writes the asset file, and the addFiles method writes other Android resource files and Java resource files. Tune writeZip will be based on prior android.useNewApkCreator use the configuration tool to determine which packages, value true with ApkFlinger , otherwise use ApkZFileCreator , android.useNewApkCreator default is true .

If you configure and use the old packaging tool ApkZFileCreator , it will use ZFile read the files generated after resource reduction ³ , and the obfuscated files and 1612efad6449aa 4 1612ef Java resources to the Android Java resources of ad6449ab 4 1612ef File.

The following source code snippet shows the main logic of writing, which is divided into the following 3 steps:

1. Create the ZFile object, read the zip file and add each item in the central directory to entries
2. Traverse ZFile in entries and merge the compressed resource files into the APK file

3. Traverse the ZFile in entries , and write the uncompressed resource file to the APK file

   // ApkZFileCreator.java
   public void writeZip(
    File zip, @Nullable Function<String, String> transform, @Nullable Predicate<String> isIgnored)
    throws IOException {
     // ...
     try {
    ZFile toMerge = closer.register(ZFile.openReadWrite(zip));
    // ...
    Predicate<String> noMergePredicate =
        v -> ignorePredicate.apply(v) || noCompressPredicate.apply(v);
   
    this.zip.mergeFrom(toMerge, noMergePredicate);
   
    for (StoredEntry toMergeEntry : toMerge.entries()) {
      String path = toMergeEntry.getCentralDirectoryHeader().getName();
      if (noCompressPredicate.apply(path) && !ignorePredicate.apply(path)) {
        // ...
        try (InputStream ignoredData = toMergeEntry.open()) {
          this.zip.add(path, ignoredData, false);
        }
      }
    }
     } catch (Throwable t) {
    throw closer.rethrow(t);
     } finally {
    closer.close();
     }
   }
   
   // ZFile.java
   private void readData() throws IOException {
     // ...
     readEocd();
     readCentralDirectory();
     // ...
     if (directoryEntry != null) {
    // ...
    for (StoredEntry entry : directory.getEntries().values()) {
      // ...
      entries.put(entry.getCentralDirectoryHeader().getName(), mapEntry);
      //...
    }
   
    directoryStartOffset = directoryEntry.getStart();
     } else {
    // ...
     }
     // ...
   }
   
   public void mergeFrom(ZFile src, Predicate<String> ignoreFilter) throws IOException {
     // ...
     for (StoredEntry fromEntry : src.entries()) {
    if (ignoreFilter.apply(fromEntry.getCentralDirectoryHeader().getName())) {
      continue;
    }
    // ...
     }
   }

Found reading during commissioning minified.jar files created ZFile in entries no Java resource files, and in front of IncrementalSplitterRunnable.execute tune in PackageAndroidArtifact.getChangedJavaResources getting changed Java resource files, use ZipCentralDirectory normally read Java resource files, indicating ZFile exist defect.

The above-mentioned problem of missing Java resource files occurs when R8 is closed. Later, R8 is turned on and the test is normal, and the new demo project is tested, regardless of whether R8 is turned on or not. Therefore, the following conclusions can be drawn:

• As ZFile comment of 0612efad644bf2, it is not a general-purpose zip tool, and has strict requirements on the zip format and unsupported features; it has restrictions under certain special conditions, and problems such as missing files may occur.
• Because the old packaging tool uses ZFile may cause problems such as missing Java resource files in the generated APk, and it has been officially abandoned and should not be used anymore

Now the direction of solving the problem is back to solving the problem of the build error. ApkFlinger into the Android or Java resource file is shown in the following figure:

Can be seen from the following source code fragment, in ZipArchive.writeSource will tune validateName validity checks written in the name of the entry, if the current central directory zip file contents of the same name already exists, throws IllegalStateException abnormalities, suggesting such as the title wrong.

// ZipArchive.java
private void writeSource(@NonNull Source source) throws IOException {
    // ...
    validateName(source);
    // ...
}

private void validateName(@NonNull Source source) {
    byte[] nameBytes = source.getNameBytes();
    String name = source.getName();
    if (nameBytes.length > Ints.USHRT_MAX) {
        throw new IllegalStateException(
                String.format("Name '%s' is more than %d bytes", name, Ints.USHRT_MAX));
    }

    if (cd.contains(name)) {
        throw new IllegalStateException(String.format("Entry name '%s' collided", name));
    }
}

Judging from the source code and debugging results, the reason for the error like the question is generally that some irregular practices make the Android resource file with the same name exist in the jar file. The 2 examples we encountered are:

• An asset file exists in the aar of a third-party library, and the same asset file also exists in its classes.jar
• A third-party library regards the aar file of another third-party library as an ordinary jar file dependency, resulting in the existence of the AndroidManifest.xml file in its classes.jar

After you know the cause of the problem, you can shrunkJavaRes.jar or minified.jar according to the prompt file name, and then locate the specific location in the project according to the information in the file (such as AndroidManifest.xml ), and then make the corresponding modification. Can.

so file has no strip

After the upgrade, the so file in the generated APK does not have a strip. Use the nm tool ⁵ in the ndk (in macOS, you can also use the system's own nm) to check and find that the symbol table and debugging information still exist.

After analyzing the build log, it is found that the strip${variant.name.capitalize()}Symbols (such as stripFullReleaseSymbols) task is executed, and then the AGP source code is analyzed, and the build process is debugged. It is found that the task StripDebugSymbolsRunnable The main logic can be seen from the following source code snippets:

1. Adjust SymbolStripExecutableFinder.stripToolExecutableFile get the strip tool path in ndk
2. If the tool is not found, copy so directly to the target location and return

3. Call this tool to strip so and output to the target location

   private class StripDebugSymbolsRunnable @Inject constructor(val params: Params): Runnable {
   
    override fun run() {
        // ...
        val exe =
            params.stripToolFinder.stripToolExecutableFile(params.input, params.abi) {
                UnstrippedLibs.add(params.input.name)
                logger.verbose("$it Packaging it as is.")
                return@stripToolExecutableFile null
            }
   
        if (exe == null || params.justCopyInput) {
            // ...
            FileUtils.copyFile(params.input, params.output)
            return
        }
   
        val builder = ProcessInfoBuilder()
        builder.setExecutable(exe)
        // ...
        val result =
            params.processExecutor.execute(
                builder.createProcess(), LoggedProcessOutputHandler(logger)
            )
        // ...
    }
    // ...
   }

Therefore, the reason why so is not stripped should be that the strip tool in ndk was not found. Further analysis of the source code shows that SymbolStripExecutableFinder finds the strip tool path through NdkHandler NdkHandler finds the ndk path through NdkLocator.findNdkPathImpl , so whether so can be stripped ultimately depends on whether the ndk path can be found. The main logic for finding ndk is as follows:

const val ANDROID_GRADLE_PLUGIN_FIXED_DEFAULT_NDK_VERSION = "21.1.6352462"

private fun findNdkPathImpl(
    userSettings: NdkLocatorKey,
    getNdkSourceProperties: (File) -> SdkSourceProperties?,
    sdkHandler: SdkHandler?
): NdkLocatorRecord? {
    with(userSettings) {
        // ...
        val revisionFromNdkVersion =
            parseRevision(getNdkVersionOrDefault(ndkVersionFromDsl)) ?: return null
        
        // If android.ndkPath value is present then use it.
        if (!ndkPathFromDsl.isNullOrBlank()) {
            // ...
        }

        // If ndk.dir value is present then use it.
        if (!ndkDirProperty.isNullOrBlank()) {
            // ...
        }
        // ...
        if (sdkFolder != null) {
            // If a folder exists under $SDK/ndk/$ndkVersion then use it.
            val versionedNdkPath = File(File(sdkFolder, FD_NDK_SIDE_BY_SIDE), "$revisionFromNdkVersion")
            val sideBySideRevision = getNdkFolderRevision(versionedNdkPath)
            if (sideBySideRevision != null) {
                return NdkLocatorRecord(versionedNdkPath, sideBySideRevision)
            }

            // If $SDK/ndk-bundle exists and matches the requested version then use it.
            val ndkBundlePath = File(sdkFolder, FD_NDK)
            val bundleRevision = getNdkFolderRevision(ndkBundlePath)
            if (bundleRevision != null && bundleRevision == revisionFromNdkVersion) {
                return NdkLocatorRecord(ndkBundlePath, bundleRevision)
            }
        }
        // ...
    }
}

private fun getNdkVersionOrDefault(ndkVersionFromDsl : String?) =
    if (ndkVersionFromDsl.isNullOrBlank()) {
        // ...
        ANDROID_GRADLE_PLUGIN_FIXED_DEFAULT_NDK_VERSION
    } else {
        ndkVersionFromDsl
    }

The main search process corresponding to the above source code snippet is shown in the following figure:

According to the above ndk search logic, we can know that the root cause of so is not build.gradle is that we did not configure android.ndkPath android.ndkVersion in 0612efad644fd5, and the local.properties file does not exist when packaging on the packaging machine, so there is no ndk.dir attribute, the ndk version installed on the packaging machine It is not the default version 21.1.6352462 specified by AGP, resulting in the ndk path not being found.

Although I found the reason, there is still a question: why can I strip normally before the upgrade? In order to find the answer, let's take a look at the way AGP 3.5 finds ndk. The main logic is as follows:

private fun findNdkPathImpl(
    ndkDirProperty: String?,
    androidNdkHomeEnvironmentVariable: String?,
    sdkFolder: File?,
    ndkVersionFromDsl: String?,
    getNdkVersionedFolderNames: (File) -> List<String>,
    getNdkSourceProperties: (File) -> SdkSourceProperties?
): File? {
    // ...
    val foundLocations = mutableListOf<Location>()
    if (ndkDirProperty != null) {
        foundLocations += Location(NDK_DIR_LOCATION, File(ndkDirProperty))
    }
    if (androidNdkHomeEnvironmentVariable != null) {
        foundLocations += Location(
            ANDROID_NDK_HOME_LOCATION,
            File(androidNdkHomeEnvironmentVariable)
        )
    }
    if (sdkFolder != null) {
        foundLocations += Location(NDK_BUNDLE_FOLDER_LOCATION, File(sdkFolder, FD_NDK))
    }
    // ...
    if (sdkFolder != null) {
        val versionRoot = File(sdkFolder, FD_NDK_SIDE_BY_SIDE)
        foundLocations += getNdkVersionedFolderNames(versionRoot)
            .map { version ->
                Location(
                    NDK_VERSIONED_FOLDER_LOCATION,
                    File(versionRoot, version)
                )
            }
    }
    // ...
    val versionedLocations = foundLocations
        .mapNotNull { location ->
            // ...
        }
        .sortedWith(compareBy({ -it.first.type.ordinal }, { it.second.revision }))
        .asReversed()
    // ...
    val highest = versionedLocations.firstOrNull()

    if (highest == null) {
        // ...
        return null
    }
    // ...
    if (ndkVersionFromDslRevision != null) {
        // If the user specified ndk.dir then it must be used. It must also match the version
        // supplied in build.gradle.
        if (ndkDirProperty != null) {
            val ndkDirLocation = versionedLocations.find { (location, _) ->
                location.type == NDK_DIR_LOCATION
            }
            if (ndkDirLocation == null) {
                // ...
            } else {
                val (location, version) = ndkDirLocation
                // ...
                return location.ndkRoot
            }
        }

        // If not ndk.dir then take the version that matches the requested NDK version
        val matchingLocations = versionedLocations
            .filter { (_, sourceProperties) ->
                isAcceptableNdkVersion(sourceProperties.revision, ndkVersionFromDslRevision)
            }
            .toList()
        
        if (matchingLocations.isEmpty()) {
            // ...
            return highest.first.ndkRoot
        }
        // ...
        val foundNdkRoot = matchingLocations.first().first.ndkRoot
        // ...
        return foundNdkRoot
    } else {
        // If the user specified ndk.dir then it must be used.
        if (ndkDirProperty != null) {
            val ndkDirLocation =
                versionedLocations.find { (location, _) ->
                    location.type == NDK_DIR_LOCATION
                }
            // ...
            val (location, version) = ndkDirLocation
            // ...
            return location.ndkRoot
        }
        // ...
        return highest.first.ndkRoot
    }
}

The corresponding general process is shown in the following figure:

It can be seen that in AGP 3.5, if the ndk path and version are not configured, the highest version in the ndk directory will be taken. As long as there is a version in the ndk directory, it can be found, so there is no problem before the upgrade. The search logic of AGP 3.6 and 4.0 is similar to that of AGP 3.5, except android.ndkVersion not configured is added. The default version of AGP 3.6 is 20.0.5594570 , and the default version of AGP 4.0 is 21.0.6113669 .

After finding the cause of the problem through the above analysis, the solution is ready to come out. In order to have a wider range of adaptability, the configuration android.ndkVersion can be used to set the ndk version to be consistent with the packager to solve the problem.

summary

This article introduces the AGP upgrade (3.5 to 4.1) process, and provides cause analysis and solutions to the problems encountered. Although the original intention of this upgrade was not to optimize the build, after the upgrade, our build speed increased by about 36%, and the package size was reduced by about 5M. I hope this article can help readers who need to upgrade successfully complete the upgrade and enjoy the official results of continuous optimization of the construction tools.

At this point, this AGP upgrade journey has come to an end, and our development journey will continue.

Reference

• Android Gradle plugin release notes
• Configuration on demand
• AGP source code

This article was published from big front-end team of NetEase Cloud Music . Any form of reprinting of the article is prohibited without authorization. We recruit front-end, iOS, and Android all year round. If you are ready to change jobs and you happen to like cloud music, then join us at grp.music-fe(at)corp.netease.com!

---

1. The product of the processFullReleaseManifestForPackage task is build/intermediates/packaged_manifests/fullRelease/AndroidManifest.xml ↩
2. The product of the processFullReleaseManifest task is build/intermediates/merged_manifests/fullRelease/AndroidManifest.xml ↩
3. build/intermediates/shrunk_processed_res/${varient.name}/resources-$flavor-$buildType-stripped.ap_ (eg build/intermediates/shrunk_processed_res/fullRelease/resources-full-release-stripped.ap_) ↩
4. build/intermediates/shrunk_java_res/${varient.name}/shrunkJavaRes.jar (for example, build/intermediates/shrunk_java_res/fullRelease/shrunkJavaRes.jar), if R8 is closed, it is build/intermediates/shrunk_jar/${varient.name}/minified.jar (for example, build/intermediates/shrunk_jar/fullRelease/minified.jar) ↩
5. toolchains/aarch64-linux-android-4.9/prebuilt/$HOST_TAG/aarch64-linux-android/bin/nm and HOST_TAG have different values in different operating systems, darwin-x86_64 in macOS, windows-x86_64 in Windows ↩