Fire at Engineering Corruption | Manifest Management

Author: Liu Tianyu (Qianfeng)

Engineering corruption is a very difficult problem in the process of app iteration. It involves extensive and fine-grained specific details. It has relatively "hidden" and indirect effects on R&D efficiency & experience, engineering & product quality, stability, package size, and performance. Impact. Generally, it will not cause unbearable obstacles, but it often pops out and causes "labor pains", which are a bit like tooth decay or wisdom teeth. It cannot be removed after reaching a certain level. After any "pulling out", an effective sustainable governance plan is required to form a normalized anti-corrosion system.

From the perspective of project corruption and disassembly, it is the corruption of the project structure itself and various "elements" (manifest, code, resources, so, configuration) in the code project that composes the app. In recent years, the Youku Architecture team has continued to think, practice and manage, and has accumulated some technologies, tools and solutions. It is now classified and summarized one by one, supplemented by the explanation of relevant field knowledge, and organized into a series of technical articles on "Burning the Corruption of Engineering" to share with you. I hope that more students will join in this protracted battle against engineering corruption.

The first article in a series of articles " at Engineering Corruption | Proguard Governance ". This article is the second in a series of articles that will focus on the subdivision of manifest. For engineering corruption, shoot directly!

background

The manifest refers to the AndroidManifest.xml file in the apk. As the overall information list of the apk, it contains a lot of important information, which has a crucial impact on the app construction period processing, runtime behavior, and application store filtering.

Manifest Content & Impact

When the content of the AndroidManifest.xml file changes unexpectedly, it will bring unexpected consequences. For example: minSdkVersion becomes smaller, and users of low-version os upgrade to the latest apk after going online, resulting in serious user experience problems; targetSdkVersion increases, os' specific processing of app runtime changes, and unsuitable code crashes/functions abnormally; new permissions Introduced, privacy agreement not stated, discovered by regulators. The above problems are only caused by a "minor" configuration value change in the manifest file. The corruption of the manifest will lead to such unexpected changes, and the possibility of occurrence is getting higher and higher. Manifest management is based on the content sorting and prevention and control of AndroidManifest.xml, which is gradually developed.

Basic knowledge

This chapter briefly introduces some basic knowledge, so that everyone can have a clear understanding of the "framework" of the manifest. First, take a look at the generation (merging) process of the AndroidManifest.xml file.

Merge process

The app project, the aar-type subproject project, and the externally-dependent aar module all contain the AndroidManifest.xml file. During the apk build process, these AndroidManifest.xml files are merged (+ some additional processing) to generate a unique AndroidManifest.xml file, which is finally placed in the apk root directory after compilation.

Merging is done gradually from low priority to high priority. Horizontal is the priority of different sources; the priority between modules is from high to low, which is the declaration order in the app project; the priority between build variant, build type, and product flavor gradually decreases; if the product flavor contains multiple dimensions, The priority is the order specified in flavorDimensions from high to low.

AndroidManifest.xml priority & merge order

During the merging process, the attributes of the same xml element (usually the android:name attribute value, or the element tag) will have merge conflicts. The basic principle is: high-priority and low-priority attribute values. for conflict. Due to the diversity of elements/attributes in the manifest file, the actual rules are much more complicated. For details, please refer to the official google documentation. To resolve merge conflicts, in addition to modifying the corresponding AndroidManifest.xml file, you can also add a "merge rule tag" to the app project AndroidManifest.xml. In addition, even if no conflict occurs, when it is necessary to control the contents of the manifest, it can be achieved in the same way, which is described next.

Merge Control

The "merge rule markup" mentioned above realizes the control of the merge result by specifying the merge rules for the two different granularities of xml nodes and attributes. First, you need to add the tools namespace in the manifest root node:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.myapplication"
    xmlns:tools="http://schemas.android.com/tools">

Then, according to the specific situation, add the corresponding tools: attribute to the node.

Merge rule tag description

For the specific rules of merge control, please refer to the official documentation, which will not be described in detail here.

manifest placeholder

In addition to the above merge controls, you can also control the attribute values of nodes in the manifest through manifest placeholders.

# build.gradle文件中定义变量和值
android {
    defaultConfig {
        manifestPlaceholders = [customKey:"customValue", ...]
    }
    ...
}

# AndroidManifest.xml文件中使用占位符
<intent-filter ... >
    <data android:scheme="https" android:host="${customKey}" ... />
    ...
</intent-filter>

<meta-data android:name="sampleMeta" android:value="${customKey}"/>

In addition, there is a default placeholder ${applicationId}, which is bound to the applicationId configuration value in the android DSL. During the build process, all placeholders are replaced with corresponding values.

Merge decision logs

Finally, each node and attribute of AndroidManifest.xml, which manifest file comes from, and which strategy is used to generate these information are recorded in the merge decision log, which provides important auxiliary information for problem analysis and troubleshooting. The file is located in the app project build/outputs/logs/manifest-merger[-productFlavor]-<buildType>-report.txt , and the example content is as follows:

activity#com.example.myapplication.MainActivity
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:18:9-24:20
  android:name
    ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:18:19-47
intent-filter#action:name:android.intent.action.MAIN+category:name:android.intent.category.LAUNCHER
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:19:13-23:29
action#android.intent.action.MAIN
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:20:17-69
  android:name
    ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:20:25-66
category#android.intent.category.LAUNCHER
ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:22:17-77
  android:name
    ADDED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:22:27-74
...
uses-permission#android.permission.READ_EXTERNAL_STORAGE
IMPLIED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:2:1-28:12 reason: com.example.libraryaar1 requested WRITE_EXTERNAL_STORAGE
MERGED from [com.youku.arch:Hound:2.8.15] /Users/flyeek/.gradle/caches/transforms-2/files-2.1/d42ba59a47f7160082879236533c4582/AndroidManifest.xml:11:5-80
MERGED from [com.youku.arch:Hound:2.8.15] /Users/flyeek/.gradle/caches/transforms-2/files-2.1/d42ba59a47f7160082879236533c4582/AndroidManifest.xml:11:5-80
uses-permission#android.permission.WRITE_CALL_LOG
IMPLIED from /Users/flyeek/workspace/code-lab/android/MyApplication/app/src/main/AndroidManifest.xml:2:1-28:12 reason: com.example.libraryaar1 has targetSdkVersion < 16 and requested WRITE_CONTACTS

After reading the basic knowledge of this article, you should be able to understand the content here, so I won't repeat it.

Several interesting configurations

So far, we have a "framework" overall understanding of the manifest file. Finally, let's take a look at a few more interesting configurations.

package vs applicationId

These two concepts are easy to confuse. From the perspective of the final apk file, the only thing that identifies the apk is the package attribute value of the manifest node in AndroidManifest.xml, which is often referred to as "appId" and "app package name". Directly above:

package vs applicationId

The package value in the app project only affects the build process. The applicationId value in the android DSL will finally replace the package attribute value in AndroidManifest.xml and become the unique identifier of the final apk.

Implicit system permissions

Under certain conditions, the merging process of manifest files will automatically add additional system permission declarations. If they are not processed, and the app privacy agreement is not declared, it will cause compliance risks. The automatic addition of permission declarations is as follows (directly taken from the official documentation):

List of permission declarations added by the merge process

For example, the targetSdkVersion of the app is 28. In the form of an external dependency, a module is introduced. The AndroidManifest.xml (low priority list) contained in the targetSdkVersion is 14 and the READ_CONTACTS permission is declared. Then the final apk manifest file will contain the READ_CALL_LOG permission. statement.

Component export control

Component export means that the android:exported attribute is true (explicit/implicit), and the component can be called by other apps. If the android:exported value is explicitly set in the manifest, then that shall prevail; if it is not set, the implicit rule is: if intent-filter is set, the exported value is true, otherwise false. Many apps will use the in-app routing mechanism of components (especially activities), so some intent-filters will be set, which will cause components to be exported unexpectedly and bring security risks. This requires special attention, and will be discussed later.

It is worth noting that when targetSdkVersion is set to 31 (Android 12) and above, if the component sets the intent-filter, then the android:exported value must be explicitly set at the same time. If the exported value is not explicitly set, the build of the IDE will fail for higher versions of Android Studio, and the build can succeed for lower versions of Android Studio, but it will fail when installing to Android 12 and above devices.

governance practice

The basic knowledge of manifest and its engineering application have been explained before. I believe that everyone has formed a preliminary overall understanding. With the increase of engineering modules/codes, the controllability of the manifest file gradually decreases: whether it is an unexpected change of key configuration values, the introduction of unexpected permissions, or even the accumulation of useless/redundant/risky nodes and attributes. In the struggle against manifest "corruption", Youku has established effective detection capabilities through relevant tools based on the actual needs of the upper layers (such as privacy compliance, security loopholes, and online problems), and based on this, a daily R&D bayonet mechanism has been formed. On the premise of ensuring zero new problems, gradually digest existing problems.

Global configuration

Some global configurations in the manifest have an important impact on apk installation and runtime behavior. The most typical ones are minSdkVersion and targetSdkVersion. Once unexpected changes are brought online, the consequences will be disastrous.

A global configuration detection tool that provides global configuration detection capabilities based on whitelists, including the following situations:

• Configured in the whitelist, but does not exist in the list;
• It is configured in the whitelist and exists in the manifest, but the configuration values are inconsistent.

At the same time, an option is provided to terminate the build process when the global configuration is inconsistent with the whitelist. The sample detection results are as follows:

[absent] [uses-feature] android.hardware.camera    # 白名单中的这个uses-feature在清单中不存在

[conflict] [uses-sdk]    # 白名单中的uses-sdk节点，属性值与清单中不一致
|-- com.youku.arch:testlib:0.1-SNAPSHOT    # 包含uses-sdk节点的模块
|-- project:library-aar-1:1.0    # 包含uses-sdk节点的模块
|-- com.youku.arch:testlib2:0.1-SNAPSHOT    # 包含uses-sdk节点的模块
|-- [attr] targetSdkVersion    # targetSdkVersion属性值不一致
|   |-- [whitelist] 29
|   |-- [current] 28
|-- [attr] minSdkVersion    # minSdkVersion属性值不一致
|   |-- [whitelist] 14
|   |-- [current] 21

Youku's global configuration whitelist and the newly added prevention and control are as follows:

Global configuration management

Through this detection capability and bayonet mechanism, the protection of key global configurations is achieved, thereby effectively avoiding unexpected changes.

permission

Permission statement, under the current privacy compliance supervision situation, needs to be strictly controlled. The "strict" here is reflected in neither more nor less, and must be consistent with the app privacy agreement. In the basic knowledge section above, we know that the AndroidManifest.xml in the apk is merged, and there is also an implicit import of system permissions, which increases the difficulty of "strict" control of permissions.

In this regard, two detection capabilities have been developed: the module contains a permission list, and permission detection.

The module contains a permission list, which lists the uses-permission and permission definitions contained in each module, which is convenient for locating the source of the permission. Example result:

com.youku.android:YPx:1.20.10.19
|-- [uses-permission] android.permission.ACCESS_NETWORK_STATE
|-- [uses-permission] android.permission.BLUETOOTH
|-- [uses-permission] android.permission.VIBRATE

com.taobao.android:ls:4.10.6.6
|-- [uses-permission] android.permission.READ_PHONE_STATE
|-- [uses-permission] android.permission.ACCESS_WIFI_STATE

Permission detection, providing bidirectional detection capabilities based on whitelists:

• The permissions in the whitelist do not exist in the list;
• Permissions in the list, not in the whitelist.

[excess] [uses-permission] android.permission.CALL_PHONE    # 清单中CALL_PHONE权限声明，不在白名单中
|-- project:app:1.0    # 权限声明，来自app工程

[absent] [uses-permission] android.permission.ACCESS_NETWORK_STATE    # 白名单中ACCESS_NETWORK_STATE，在清单中不存在
|-- com.youku.arch:testlib:1.0    # com.youku.arch:testlib模块，包含此权限声明
|-- com.youku.arch:testlib2:1.0    # com.youku.arch:testlib2模块，包含此权限声明

Going a step further, provide the option to terminate the build process when the test fails. Through this detection capability and bayonet mechanism, the continuous consistency of the permission statement and the app privacy agreement is guaranteed. Youku's governance and prevention and control are as follows:

permission management

Four components

The four major components need to be declared in the manifest file in order to be recognized by the system after the apk is installed and run, and thus function normally. At the same time, some key behaviors of the four components also need to be configured in the manifest. In the process of Youku practice, there are mainly two types of problems: the lack of corresponding classes of components, and the export of non-essential components.

The absence of the corresponding class of the component refers to the four components declared in the manifest. The android:name attribute value corresponds to the java class, which does not exist in the apk. The negative effects of missing component classes are as follows:

• A proguard useless keep rule will be generated, which will increase the construction time (even though a keep is small, it will gather sand into a tower, which is also considerable);
• Once the component is called (started) at runtime, a java exception (crash/function unavailable) or security vulnerability will be generated. Even if it is a useless component, consider that there are still some black production organizations, which will automatically scan the component and start it (the crash rate curve will have spikes).

Exporting non-essential components (see above for the definition) will increase the risk of security vulnerabilities at runtime. Youku has received many related security vulnerabilities. The processing principles of exported components are as follows:

• There is no need to export, and it is self-developed code. close export;
• It is not necessary to export, but it is a second or third party code. In the manifest file of the app project, modify the value of the android:exported attribute to false through "Merge Rule Marker";
• It needs to be exported and is self-developed code for debugging during development. Close the export and converge to the unified R&D debugging toolbox;
• It needs to be exported, and it is a self-developed code for online actual business (external call terminal, etc.). Close the export and converge to the unified routing center;
• It needs to be exported, but it is a two- or three-party code, which is used for online actual business (external call terminal, etc.). Add whitelist.

For this, three detection capabilities have been developed:

• A list of modules that a component belongs to, listing all four major components, and the module that contains the declaration of this component:

# 在manifest合并后不存在的组件，前面会加上[delete]
# 被超过两个以上模块包含的组件，前面会加上[duplicate]

[duplicate] [activity] com.example.myapplication.MainActivity
|-- project:app:1.0
|-- project:library-aar-1:1.0

[deleted] [service] com.example.myapplication.FirstService
|-- project:app:1.0

[receiver] com.example.myapplication.FirstReceiver
|-- project:library-aar-1:1.0

[provider] com.example.myapplication.FirstProvider
|-- com.youku.arch:testlib:1.0

• Missing component reference detection, identifying missing referenced component names, and which modules declare this component. At the same time, provide options and a whitelist, and terminate the build process when the detection result fails. Example detection results are as follows:

[activity] org.cocos2dx.javascript.AActivity
|-- com.youku.android:interactive-engine:0.2.9

[activity] com.ali.lv.HLActivity
|-- com.ali.phone.wt:n-build:10.2.3.592

[activity] com.youku.pc.debug.DActivity
|-- com.youku.android:YKPChannel:2.14.1.28

[service] com.youku.feed.utils.FAService
|-- com.youku.android:FBase:1.5.20.8

• Exported component detection, identifying exported components, and which modules declare this component. At the same time, provide options and a whitelist, and terminate the build process when the detection result fails. For the behavior change of the more secure export component of Target31, the "Prohibit Implicit Export" configuration item is specially provided, which will ignore the whitelist and add an identifiable flag to the analysis result. Example detection results are as follows:

# 对于白名单中组件，会在名称前加上[ignored]标识；如果开启「禁止隐式导出」配置项，对于隐式导出组件，会在名称前加上[implicit]标识

[activity] com.youku.app.NPageActivity
|-- com.youku.android:YoukuHPage:1.9.43.8

[ignored][activity] com.ali.MIPreviewActivity
|-- com.ali:m-image-selector:10.1.6.190

[implicit] [activity] com.youku.fbiz.RPageActivity
|-- com.youku.android:fbizSDK:1.0.2.48

In the governance practice of Youku, considering the impact on the R&D students of various businesses, the existing problems were added to the whitelist, and the clean-up actions were launched in a unified manner. With the iteration of the version, in addition to the effective interception of new problems, there are also some "natural fixes" for the existing problems. The overall situation is as follows:

of the four major components

In addition, Youku's current targetSdkVersion is 30, and target31 adaptation will be carried out next year. There are 161 implicit export components in stock, accounting for about 50% of all export components. All these need to be solved by then. Under the current tool and bayonet system, I believe that the rectification of this problem will become easy and controllable.

governance panorama

So far, for the manifest list, a more comprehensive and effective anti-corrosion capacity building and governance have been carried out. Finally, give a panorama:

Manifest governance panorama

what else can be done

The content contained in the manifest is relatively limited. Therefore, the above governance should have covered most of the problems, but there are still some low-probability edge cases, which can be identified and solved in advance through the same idea, for example, the scheme definitions of multiple activities are repeated, resulting in implicit startup. activity, the selection popup appears.

The confrontation with the corruption of the project is still difficult, and there is a long way to go, and I will share with you.

[Reference document]

• 【google】App Manifest Overview:https://developer.android.com/guide/topics/manifest/manifest-intro
• 【google】Manage manifest files:https://developer.android.com/studio/build/manage-manifests

Follow [Alibaba Mobile Technology] WeChat public account, 3 mobile technology practice & dry goods every week for you to think about!