The core channel of Google Play cannot be bypassed when applications go overseas. Follow [Rongyun Global Internet Communication Cloud] to learn more
According to data from the data platform Statcounter, as of June 2022, Android has a market share of 72.12% in the global mobile device operating system. And Google Play is the absolute Android traffic giant and the first choice for Android applications.
As of August 2021, Google requires apps published in Google Play to use the Android App Bundle (AAB) format. AAB can provide a smaller App size, improve the user's download conversion rate and reduce the amount of uninstallation, and it requires that the size of the application does not exceed 150MB.
For apps that require more than 150MB, App Bundles have introduced Play Asset Delivery (PAD), which dynamically distributes assets for smoother publishing and faster updates since the update package doesn't contain all the content.
This article takes an overseas app as an example to share its practical process of "slim down" by adopting the Android App Bundle format and the Play Asset Delivery solution.
Android App Bundle
Advantages of AAB
Publishing using AAB, the APK will be generated and signed by Google Play. And the package size has changed from APK's 100M limit to AAB's 150M limit.
Using the App Bundle will transfer the generation and signing of the APK to Google Play. When the user downloads, Google Play will generate an optimized APK based on the device used by the user, which contains only the code and resources required by the device at runtime , you can get a smaller, more targeted download package.
(Use the App Bundle for a smaller download package)
The composition of the AAB package
The Android App Bundle is a new distribution format that includes all compiled code and resources. There are three types of modules in an AAB package:
- Base APK (Base.apk): Provides basic functions. When the user downloads the application, the APK will be downloaded and installed first.
- Configuration APK: Corresponding APK files will be generated for different screen densities, CPU architectures or languages. When the user downloads the application, only the APK of the corresponding configuration will be downloaded and installed.
- Function module APK: We can package non-basic and relatively independent functions into APK, and install them when users need to use them.
(AAB package main module)
As shown in the image above:
base/: This directory contains the application base module code.
feature1/ and feature2/: This directory contains code that needs to be installed dynamically.
asset_pack_1/ and asset_pack_2/: This directory contains assets that need to be loaded dynamically.
BUNDLE-METADATA/: This directory contains metadata files.
BundleConfig.pb: Provides information about the Bundle itself, such as the version of the tools used to build the App Bundle.
manifest/: The AndroidManifest.xml file for each module is stored separately in this directory.
dex/: Stores all dex files for each module.
How to use AAB?
To generate an AAB package is very simple, we only need to change from selecting APK to selecting Andriod App Bundle when packaging, and the rest of the process is exactly the same as generating the APK file.
Note here:
- After using the AAB format, APK expansion files (*.obb) are no longer supported.
- To publish in AAB format, Google Play's app signing feature must be turned on.
Google Play will use this key to sign the optimized APK. It is recommended to check Export encrypted key for enrolling published apps in Google Play App Signing to export and upload the key to Google Play, as shown in the following figure.
AAB resource configuration
By default, when building an App Bundle, configuration APKs are supported for each set of language resources, screen density resources, and ABI libraries. If you want to disable support for a certain configuration APK type, you can configure it in the base module's build.gradle file.
android {
// Instead, use the bundle block to control which types of configuration APKs
// you want your app bundle to support.
bundle {
language {
// Specifies that the app bundle should not support
// configuration APKs for language resources. These
// resources are instead packaged with each base and
// feature APK.
enableSplit = true|false
}
density {
// This property is set to true by default.
enableSplit = true|false
}
abi {
// This property is set to true by default.
enableSplit = true|false
}
}
}Test AAB with Bundletool
During development, we may need to analyze and debug AAB files. At this time, you need to use the Bundletool tool. Using this tool, we can accomplish the following functions:
1. Convert AAB to APKS
The AAB format cannot be directly installed on the mobile phone. It is necessary to convert the AAB format to an APKS file, and then install the corresponding APK.
During the development phase, we can use the Build Bundle to generate AAB files.
Then use the following command to output the APKS file:
java -jar bundletool-all-1.10.0.jar build-apks --bundle=app-debug.aab --output=app.apks --local-testing
Use the Zip tool to decompress the generated APKS file, you can see that in the splits directory, different APK files are generated for different languages, resolutions, and ABIs.
If language's enableSplit is set to false, no different APK files will be generated for languages.
If you only generate the APKS file for the device currently connected to the PC, you can use the following command:
java -jar bundletool-all-1.10.0.jar build-apks --connected-device --bundle=app-debug.aab --output=app.apks
By using this command, the generated APKS file only contains the base APK + configuration APK for the device.
We can get the configuration json file of the currently connected device using the following command:
java -jar bundletool-all-1.10.0.jar get-device-spec --output=device.json
{
"supportedAbis": ["arm64-v8a", "armeabi-v7a", "armeabi"],
"supportedLocales": ["zh-CN", "ar-JO", "en-US"],
"deviceFeatures": ["reqGlEsVersion\u003d0x30002", "android.hardware.audio.low_latency", "android.hardware.audio.output", "android.hardware.audio.pro", "android.hardware.bluetooth", "android.hardware.bluetooth_le", "android.hardware.camera", "android.hardware.camera.any", "android.hardware.camera.autofocus", "android.hardware.camera.capability.manual_post_processing", "android.hardware.camera.capability.manual_sensor", "android.hardware.camera.capability.raw", "android.hardware.camera.concurrent", "android.hardware.camera.flash", "android.hardware.camera.front", "android.hardware.camera.level.full", "android.hardware.context_hub", "android.hardware.device_unique_attestation", "android.hardware.faketouch", "android.hardware.fingerprint", "android.hardware.identity_credential\u003d202101", "android.hardware.location", "android.hardware.location.gps", "android.hardware.location.network", "android.hardware.microphone", "android.hardware.nfc", "android.hardware.nfc.any", "android.hardware.nfc.ese", "android.hardware.nfc.hce", "android.hardware.nfc.hcef", "android.hardware.nfc.uicc", "android.hardware.opengles.aep", "android.hardware.ram.normal", "android.hardware.reboot_escrow", "android.hardware.screen.landscape", "android.hardware.screen.portrait", "android.hardware.se.omapi.ese", "android.hardware.se.omapi.uicc", "android.hardware.security.model.compatible", "android.hardware.sensor.accelerometer", "android.hardware.sensor.barometer", "android.hardware.sensor.compass", "android.hardware.sensor.gyroscope", "android.hardware.sensor.hifi_sensors", "android.hardware.sensor.light", "android.hardware.sensor.proximity", "android.hardware.sensor.stepcounter", "android.hardware.sensor.stepdetector", "android.hardware.strongbox_keystore", "android.hardware.telephony", "android.hardware.telephony.carrierlock", "android.hardware.telephony.cdma", "android.hardware.telephony.euicc", "android.hardware.telephony.gsm", "android.hardware.telephony.ims", "android.hardware.touchscreen", "android.hardware.touchscreen.multitouch", "android.hardware.touchscreen.multitouch.distinct", "android.hardware.touchscreen.multitouch.jazzhand", "android.hardware.usb.accessory", "android.hardware.usb.host", "android.hardware.vulkan.compute", "android.hardware.vulkan.level\u003d1", "android.hardware.vulkan.version\u003d4198400", "android.hardware.wifi", "android.hardware.wifi.aware", "android.hardware.wifi.direct", "android.hardware.wifi.passpoint", "android.hardware.wifi.rtt", "android.software.activities_on_secondary_displays", "android.software.app_enumeration", "android.software.app_widgets", "android.software.autofill", "android.software.backup", "android.software.cant_save_state", "android.software.companion_device_setup", "android.software.connectionservice", "android.software.controls", "android.software.cts", "android.software.device_admin", "android.software.device_id_attestation", "android.software.file_based_encryption", "android.software.home_screen", "android.software.incremental_delivery\u003d2", "android.software.input_methods", "android.software.ipsec_tunnels", "android.software.live_wallpaper", "android.software.managed_users", "android.software.midi", "android.software.opengles.deqp.level\u003d132383489", "android.software.picture_in_picture", "android.software.print", "android.software.secure_lock_screen", "android.software.securely_removes_users", "android.software.sip", "android.software.sip.voip", "android.software.verified_boot", "android.software.voice_recognizers", "android.software.vulkan.deqp.level\u003d132383489", "android.software.webview", "com.google.android.apps.dialer.SUPPORTED", "com.google.android.feature.ADAPTIVE_CHARGING", "com.google.android.feature.AER_OPTIMIZED", "com.google.android.feature.D2D_CABLE_MIGRATION_FEATURE", "com.google.android.feature.DREAMLINER", "com.google.android.feature.EXCHANGE_6_2", "com.google.android.feature.GOOGLE_BUILD", "com.google.android.feature.GOOGLE_EXPERIENCE", "com.google.android.feature.GOOGLE_FI_BUNDLED", "com.google.android.feature.NEXT_GENERATION_ASSISTANT", "com.google.android.feature.PIXEL_2017_EXPERIENCE", "com.google.android.feature.PIXEL_2018_EXPERIENCE", "com.google.android.feature.PIXEL_2019_EXPERIENCE", "com.google.android.feature.PIXEL_2019_MIDYEAR_EXPERIENCE", "com.google.android.feature.PIXEL_2020_EXPERIENCE", "com.google.android.feature.PIXEL_2020_MIDYEAR_EXPERIENCE", "com.google.android.feature.PIXEL_EXPERIENCE", "com.google.android.feature.TURBO_PRELOAD", "com.google.android.feature.WELLBEING", "com.nxp.mifare", "com.verizon.hardware.telephony.ehrpd", "com.verizon.hardware.telephony.lte"],
"glExtensions": ["GL_OES_EGL_image", "GL_OES_EGL_image_external", "GL_OES_EGL_sync", "GL_OES_vertex_half_float", "GL_OES_framebuffer_object", "GL_OES_rgb8_rgba8", "GL_OES_compressed_ETC1_RGB8_texture", "GL_AMD_compressed_ATC_texture", "GL_KHR_texture_compression_astc_ldr", "GL_KHR_texture_compression_astc_hdr", "GL_OES_texture_compression_astc", "GL_OES_texture_npot", "GL_EXT_texture_filter_anisotropic", "GL_EXT_texture_format_BGRA8888", "GL_EXT_read_format_bgra", "GL_OES_texture_3D", "GL_EXT_color_buffer_float", "GL_EXT_color_buffer_half_float", "GL_QCOM_alpha_test", "GL_OES_depth24", "GL_OES_packed_depth_stencil", "GL_OES_depth_texture", "GL_OES_depth_texture_cube_map", "GL_EXT_sRGB", "GL_OES_texture_float", "GL_OES_texture_float_linear", "GL_OES_texture_half_float", "GL_OES_texture_half_float_linear", "GL_EXT_texture_type_2_10_10_10_REV", "GL_EXT_texture_sRGB_decode", "GL_EXT_texture_format_sRGB_override", "GL_OES_element_index_uint", "GL_EXT_copy_image", "GL_EXT_geometry_shader", "GL_EXT_tessellation_shader", "GL_OES_texture_stencil8", "GL_EXT_shader_io_blocks", "GL_OES_shader_image_atomic", "GL_OES_sample_variables", "GL_EXT_texture_border_clamp", "GL_EXT_EGL_image_external_wrap_modes", "GL_EXT_multisampled_render_to_texture", "GL_EXT_multisampled_render_to_texture2", "GL_OES_shader_multisample_interpolation", "GL_EXT_texture_cube_map_array", "GL_EXT_draw_buffers_indexed", "GL_EXT_gpu_shader5", "GL_EXT_robustness", "GL_EXT_texture_buffer", "GL_EXT_shader_framebuffer_fetch", "GL_ARM_shader_framebuffer_fetch_depth_stencil", "GL_OES_texture_storage_multisample_2d_array", "GL_OES_sample_shading", "GL_OES_get_program_binary", "GL_EXT_debug_label", "GL_KHR_blend_equation_advanced", "GL_KHR_blend_equation_advanced_coherent", "GL_QCOM_tiled_rendering", "GL_ANDROID_extension_pack_es31a", "GL_EXT_primitive_bounding_box", "GL_OES_standard_derivatives", "GL_OES_vertex_array_object", "GL_EXT_disjoint_timer_query", "GL_KHR_debug", "GL_EXT_YUV_target", "GL_EXT_sRGB_write_control", "GL_EXT_texture_norm16", "GL_EXT_discard_framebuffer", "GL_OES_surfaceless_context", "GL_OVR_multiview", "GL_OVR_multiview2", "GL_EXT_texture_sRGB_R8", "GL_KHR_no_error", "GL_EXT_debug_marker", "GL_OES_EGL_image_external_essl3", "GL_OVR_multiview_multisampled_render_to_texture", "GL_EXT_buffer_storage", "GL_EXT_external_buffer", "GL_EXT_blit_framebuffer_params", "GL_EXT_clip_cull_distance", "GL_EXT_protected_textures", "GL_EXT_shader_non_constant_global_initializers", "GL_QCOM_texture_foveated", "GL_QCOM_texture_foveated_subsampled_layout", "GL_QCOM_shader_framebuffer_fetch_noncoherent", "GL_QCOM_shader_framebuffer_fetch_rate", "GL_EXT_memory_object", "GL_EXT_memory_object_fd", "GL_EXT_EGL_image_array", "GL_NV_shader_noperspective_interpolation", "GL_KHR_robust_buffer_access_behavior", "GL_EXT_EGL_image_storage", "GL_EXT_blend_func_extended", "GL_EXT_clip_control", "GL_OES_texture_view", "GL_EXT_fragment_invocation_density", "GL_QCOM_motion_estimation", "GL_QCOM_validate_shader_binary", "GL_QCOM_YUV_texture_gather"],
"screenDensity": 440,
"sdkVersion": 31
}It can be seen from the generated json file that the current regional languages supported by the device are Chinese, Arabic and English. Therefore, the APKs of these three languages are also included in the previously generated APK.
2. Deploy the APKS to the connected device
After generating the apks file, use the following command to deploy the apks file to the currently connected device:
java -jar bundletool-all-1.10.0.jar install-apks --apks=app_release.apks
After the installation is successful, we can use the adb command to confirm whether the configuration APK was successfully installed:
adb shell pm path "package name"
You can also use the following command to achieve the same purpose:
adb shell dumpsys package package name | findstr split
This is useful when debugging AAB packages are installed correctly.
Play Asset Delivery
After using the AAB format, we found that the final output AAB file of the case application is still large, and the upload to Google Play still exceeds the limit.
At this time, the Play Asset Delivery (PAD) function is used. PAD is widely used in game apps. It can publish game resources (textures, sounds, etc.) separately, and calculate the size separately when uploading the AAB package to Google Play. We can put resources that take up a lot of space in the App into a separate Asset Pack, bypassing the limit of 150M for uploading AAB packages on Google Play.
PAD has three distribution modes:
- install-time: Asset Packs are distributed when the user installs the app, also known as "up-front" asset packs, and can be used immediately when the app starts. These resource packs increase the size of the app listed on the Google Play Store and cannot be modified or deleted by the user.
- fast-follow: Asset Packs are automatically downloaded as soon as the user installs the app. Users don't need to open the app to start downloading, and it doesn't block users from using the app. These resource packs increase the size of the app listed on the Google Play Store.
- on-demand: Asset Packs are downloaded when the app is running.
Different distribution modes have different upper size limits for Asset Packs:
- The asset pack size limit for each fast-follow and on-demand mode is 512MB.
- The total asset pack limit for all install-time modes is 1GB.
3. The total size of all Asset Packs in an AAB package is limited to 2GB.
4. A maximum of 50 Asset Packs can be used in one AAB pack.
Dynamic loading of resources
install-time scheme
The first step is to create a new Module to store the resources.
The second step is to modify the code in the build.gradle of the install_time_asset_pack Module to:
// In the asset pack’s build.gradle file:
apply plugin: 'com.android.asset-pack'
assetPack {
packName = "install_time_asset_pack"
dynamicDelivery {
//只能指定一种分发模式
deliveryType = "install-time"
}
} Step 3 Add the following code to the build.gradle of the App Module:
Step 4 Add the install_time_asset_pack module to the setting.gradle file.
include ':install_time_asset_pack'
Step 5 Put the resources that take up a lot of space into the install_time_asset_pack Module.
Here you need to create an assets directory under the mian directory, and put the resources in this directory.
Step 6 Since we put the resources into the Asset Pack package, we need to delete these resources in the original directory.
applicationVariants.all {
variant ->
variant.mergeAssetsProvider.configure {
doLast {
def file = fileTree(dir: outputDir, includes: ['model/ai_body.bundle',
'model/ai_face.bundle',
'model/ai_green.bundle',
'model/ai_human.bundle',
'graphics/body.bundle',
'graphics/controller.bundle',
'graphics/face.bundle',
'graphics/tongue.bundle'])
delete(file)
}
}
}Step 7 Write the code to copy the assets in the Asset Pack to the project's private directory and return the path. Modify it to this path in the original loading logic.
public String copyResource(String relativeAssetPath){
AssetFileDescriptor openFd = mAssetManager.openFd(relativeAssetPath);
String filePath = mContext.getExternalFilesDir(null).getAbsolutePath() + File.separator + relativeAssetPath;
File file = new File(filePath);
if (file.exists()) {
return filePath;
} else {
new File(file.getParent() + "/").mkdirs();
}
copyFile(openFd.createInputStream(), filePath)
return filePath;
}
private void copyFile(FileInputStream fileInputStream, String outFilePath) throws IOException {
if (fileInputStream != null) {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(outFilePath);
byte[] bytes = new byte[1024];
int temp = 0;
while ((temp = fileInputStream.read(bytes)) != -1) {
fos.write(bytes, 0, temp);
}
} catch (Exception exception) {
Log.e(TAG, "copyFile: e=" + exception.getMessage());
} finally {
if (fos != null) {
fos.close();
}
}
}
}fast-follow, on-demand Asset solutions
In the fast-follow or on-demand Asset scheme, the deliveryType property needs to be modified in the build.gralde of the Asset Pack.
apply plugin: 'com.android.asset-pack'
assetPack {
packName = "on_demand_asset" // Directory name for the asset pack
dynamicDelivery {
deliveryType = "fast-follow | on-demand"
}
}Then it is necessary to determine whether the resource package exists. If it does not exist, the download needs to be started and its download status is monitored.
private void getAssetResource() {
if (mAssetPackManager != null) {
AssetPackLocation assetLocation = mAssetPackManager.getPackLocation(AssetPackName);
if (assetLocation == null) {
//跟踪资源包的安装进度
mAssetPackManager.registerListener(new AssetPackStateUpdateListener() {
@Override
public void onStateUpdate(@NonNull AssetPackState assetPackState) {
switch (assetPackState.status()) {
case AssetPackStatus.PENDING:
break;
case AssetPackStatus.DOWNLOADING:
//这里监控下载进度
break;
case AssetPackStatus.TRANSFERRING:
// 100% downloaded and assets are being transferred.
// Notify user to wait until transfer is complete.
break;
case AssetPackStatus.COMPLETED:
//下载成功后加载数据
loadData();
break;
case AssetPackStatus.FAILED:
//如果下载失败了在这里进行处理
break;
case AssetPackStatus.CANCELED:
// Request canceled. Notify user.
break;
case AssetPackStatus.WAITING_FOR_WIFI:
if (!waitForWifiConfirmationShown) {
mAssetPackManager.showCellularDataConfirmation(MainActivity.this)
.addOnSuccessListener(new OnSuccessListener<Integer>() {
@Override
public void onSuccess(Integer resultCode) {
if (resultCode == RESULT_OK) {
Log.d(TAG, "Confirmation dialog has been accepted.");
} else if (resultCode == RESULT_CANCELED) {
Log.d(TAG, "Confirmation dialog has been denied by the user.");
}
}
});
waitForWifiConfirmationShown = true;
}
break;
case AssetPackStatus.NOT_INSTALLED:
// Asset pack is not downloaded yet.
break;
case AssetPackStatus.UNKNOWN:
break;
}
}
});
//下载资源包
mAssetPackManager.fetch(Collections.singletonList(AssetPackName));
} else {
//资源如果已经下载，直接进行加载
loadData();
}
}
}During the download process, if the downloaded content exceeds 150M and the user is not connected to Wi-Fi, it will not be downloaded until the user explicitly agrees to use the mobile network to download.
Likewise, if the download is large and the user is disconnected from Wi-Fi in the middle, the download will be paused and will require the user's explicit consent to use the mobile network for the download to continue. At this time, the monitored status is WAITING_FOR_WIFI. To trigger a prompt for the user to download using the mobile network, the showCellularDataConfirmation() method needs to be called.
Summarize
Using install-time mode, you can use the AssetManager API to access resources after the Asset Pack is ready.
When using fast-follow or on-demand mode, it is necessary to first determine whether the Asset Pack has been downloaded. If the download is successful, directly obtain the path to use. If the download has not yet started, you need to trigger the download of the resource pack and monitor its status in order to give feedback to the user.
SO dynamic loading
SO dynamic loading is slightly different from dynamic loading of resources. We know that there are generally two ways to load SO files in Android:
- System.load()
- System.loadLibrary()
So if we want to load SO dynamically, we have to know the difference between these two ways of loading SO, and find solutions according to their differences.
First of all, when using the method, the System.load() method needs to pass in a complete file path. And System.loadLibrary() only needs to pass in the library file name.
Secondly, the System.load() method needs to load the dependent library first. For example: LibA.so depends on the LibB.so file. When using the load() method to load the LibA.so file, even if the LibB.so file and the LibA.so file are in the same directory, the loading will fail because LibB.so cannot be found. .
You need to use the load() method to load LibB.so and then load LibA.so. This requires us to know the file's dependencies when loading the SO file. And using the loadLibrary() method only needs to put the dependent SO in the same directory.
System.load() scheme
To dynamically load SO using the System.load() method, we first need to resolve the SO library dependencies. The dependencies of the SO library must be defined somewhere, and as long as we can find this place and get the dependencies recursively, we can get the complete dependency path.
ELF structure
We know that Android is based on the Linux operating system, and SO files are stored in ELF format under Linux. So we only need to parse the SO file in ELF format to get the dependencies.
To parse ELF, we need to know the structure of ELF.
The information in ELF is stored in the form of Segment (segment), and the more common segments are listed in the figure above:
.text: Also known as the code segment, the machine instructions compiled from the source program are often stored here.
.data: The data segment is used to store global variables and local static variables.
.bss: Uninitialized global variables and local static variables are generally stored here. Because these variables are not initialized, their default value is 0. It is not necessary to place them in the data segment. A place is reserved for them in the .bss segment alone, so . The bss section also does not take up space in the ELF file.
.got: The Global Offset Table (Global Offset Table) is designed to solve the problem that the dynamic link library can be shared by multiple processes. Each application collects the referenced dynamic library symbols, saves them in the got table, and uses this table to record the address of each referenced symbol. When these symbols need to be referenced in the program, the address of each symbol is queried through this table.
The advantage of this is that only one dynamic library needs to be loaded in the memory. When different programs are running, they only need to modify their respective got tables, and the symbols they reference can all point to the same dynamic library, so that different programs can share the same dynamic library. The purpose of a dynamic library.
.plt: The PLT table is used to resolve late binding. It is not advisable to collect symbols in all dynamic libraries when the program starts running, and save them in the GOT table. In order to implement delayed binding, a layer of jumps can be added through the PLT table.
test@plt:
jmp *(test@GOT)
push n
push moduleID
jump _dl_runtime_resolveIn the above pseudo code, the first instruction of test@plt is to jump to the test@GOT table to query the address of the test() method.
Since it is the first time to access the test() method, there is no real address of the test() method in the test@GOT table at this time, but the address of the second instruction push n in the saved test@plt table, so it will continue to jump Go to the test@plt table to perform the push n operation, the number n is the subscript of the symbol reference test in the relocation table (.rel.plt).
Until the _dl_runtime_resolve instruction is executed, the real address of the test() method will be saved in the test@GOT table, and when test@plt is called again, it will jump to the real address of the test() method.
Among the current Android Native Hook solutions, one of them is to perform Hook based on the PLT/GOT table.
.dynamic: The basic information required by the dynamic linker is stored in the dynamic segment, including the location of the dynamic link symbol table, which shared objects it depends on, and the location of the dynamic link relocation table. You can view it with readelf -d.
Section header table: The section table describes the information of each section of the ELF, including the section name, the length of the section, and the offset in the file. We can use readelf -S or objdump -h to check.
program header table: The program header table is an array, each element in the array is called "program header", and each program header describes a Segment (segment) information. You can use readelf -h to see.
string table: The string table contains null-terminated strings. These strings may be the names of symbols or segments. When you need to refer to a string, you only need to provide the serial number of the string in the string table. That's it.
It should be noted that the first string in the string table is always an empty string (null). Since each string is null-terminated, the last byte must also be null.
symbol table: The symbol table records all symbols (functions, variables) used in the ELF file. Each symbol has a corresponding value, which, for variables and functions, is their address.
Get dependency information
After having a general understanding of the ELF file format, we know that the dependency information of the current SO file is stored in the .dynamic section. As long as we can get this information, we can recursively get the complete dependencies. To read the contents of a .dynamic section, you must know the offset of the section in the file. The offset of the section can be found in the section header table or the program header table.
And the addresses of the section header table and the program header table can be obtained from the ELF header.
The overall idea is as follows:
- First read the ELF header information
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Shared object file)
Machine: ARM
Version: 0x1
Entry point address: 0x0
Start of program headers: 52 (bytes into file)
Start of section headers: 12588 (bytes into file)
Flags: 0x5000000, Version5 EABI
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 7
Size of section headers: 40 (bytes)
Number of section headers: 20
Section header string table index: 19 From the header information we can see the ELF magic number:
The first 4 bytes are the same identification code for all ELF files. When parsing the ELF file, we need to use this identification code to determine whether the current ELF format is being parsed.
You can see that the ELF file we are currently parsing is a 32-bit ELF file and is in little endian order. These two pieces of information are very important and are directly related to the correctness of our parsing ELF files. For example, little endian is used here, so when judging the magic number of ELF, it should be compared with 0x464C457F. With this information, we have a basis when reading the ELF file header. Since it is a 32-bit ELF file, we need to parse it according to the 32-bit data structure.
typedef struct {
unsigned char e_ident[16]; //0x00-0x0f
Elf32_Half e_type; //0x10-0x11
Elf32_Half e_machine; //0x12-0x13
Elf32_Word e_version; //0x14-0x17
Elf32_Addr e_entry; //0x18-0x1b
Elf32_Off e_phoff; //0x1c-0x1f
Elf32_Off e_shoff; //0x20-0x23
Elf32_Word e_flags; //0x24-0x27
Elf32_Half e_ehsize; //0x28-0x29
Elf32_Half e_phentsize; //0x2a-0x2b
Elf32_Half e_phnum; //0x2c-0x2d
Elf32_Half e_shentsize; //0x2e-0x2f
Elf32_Half e_shnum; //0x30-0x31
Elf32_Half e_shstrndx; //0x32-0x33
} Elf32_Ehdr; Among them, Elf32_Half, ELF32_Word, etc. are all custom types, and the lengths are as follows:
Next we need to get several important fields in the ELF header:
1.e_type: What type of tag file belongs to.
Not all values are listed in the table, here we only need to judge whether the current type is ET_DYN.
2. e_phoff: the offset of the program header table, that is, the address of the program header table, in bytes. This corresponds to:
Start of program headers: 52 (bytes into file)
4 bytes need to be read from 0x1C.
3.e_shoff: The offset of the segment header table, that is, the address of the segment header table, in bytes. This corresponds to:
Start of section headers: 12588 (bytes into file)
4 bytes need to be read from 0x20.
4.e_phentsize: The size of each segment in the program header table, in bytes. This corresponds to:
Size of program headers: 32 (bytes)
2 bytes need to be read from 0x2A.
5.e_phnum: the number of segments in the program header table. This corresponds to:
Number of program headers: 7
2 bytes need to be read from 0x2C.
6.e_shentsize: The size of each segment in the segment header table, in bytes. This corresponds to:
Size of section headers: 40 (bytes)
Need to read 2 bytes from 0x2E.
7:e_shnum: The number of segments in the segment header table. This corresponds to:
Number of section headers: 20
2 bytes need to be read from 0x30.
8.e_shstrndx: The index of the string table in the segment header table. This corresponds to:
Section header string table index: 19
Need to read 2 bytes from 0x32.
After getting the offset of the program header table, we can traverse the program header table. There are 7 program headers in this example.
Elf file type is DYN (Shared object file)
Entry point 0x0
There are 7 program headers, starting at offset 52
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000034 0x00000034 0x00000034 0x000e0 0x000e0 R 0x4
LOAD 0x000000 0x00000000 0x00000000 0x02160 0x02160 R E 0x1000
LOAD 0x002eac 0x00003eac 0x00003eac 0x00158 0x00158 RW 0x1000
DYNAMIC 0x002eb8 0x00003eb8 0x00003eb8 0x00100 0x00100 RW 0x4
GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0
EXIDX 0x002088 0x00002088 0x00002088 0x000d8 0x000d8 R 0x4
GNU_RELRO 0x002eac 0x00003eac 0x00003eac 0x00154 0x00154 RW 0x4
Section to Segment mapping:
Segment Sections...
00
01 .dynsym .dynstr .hash .rel.dyn .rel.plt .plt .text .ARM.extab .ARM.exidx
02 .fini_array .init_array .dynamic .got .data
03 .dynamic
04
05 .ARM.exidx
06 .fini_array .init_array .dynamic .got
我们从偏移位置为 0x52 处开始遍历，每次增加步长为 32（e_phentsize）。同样的，程序头也有自己的数据结构：
typedef struct {
Elf32_Word p_type;//0x52-0x55
Elf32_Off p_offset; //0x56-0x59
Elf32_Addr p_vaddr; //0x5a-0x5d
Elf32_Addr p_paddr; //0x5e-0x62
Elf32_Word p_filesz; //0x63-0x66
Elf32_Word p_memsz; //0x67-0x6a
Elf32_Word p_flags; //0x6b-0x6e
Elf32_Word p_align; //0x6f-0x73
} Elf32_Phdr;We need to read the following information from the program header:
1.p_type: The type of the segment described by the program header, here we only need to find the PT_DYNAMIC type.
2.p_offset: The offset of the segment described by the program header, in bytes relative to the offset at the beginning of the file.
3.p_vaddr: The virtual address of the starting position of the content of this section in the process, in bytes.
4.p_memsz: The size of the content of this section, in bytes.
As you can see from what readelf prints, all we're actually looking for is this line:
DYNAMIC 0x002eb8 0x00003eb8 0x00003eb8 0x00100 0x00100 RW 0x4
The offset address here is the offset address of the .dynamic segment.
With the offset address of the .dynamic section, we can read the dependent SO file.
Dynamic section at offset 0x2eb8 contains 27 entries:
Tag Type Name/Value
0x00000003 (PLTGOT) 0x3fd4
0x00000002 (PLTRELSZ) 64 (bytes)
0x00000017 (JMPREL) 0xb28
0x00000014 (PLTREL) REL
0x00000011 (REL) 0xae8
0x00000012 (RELSZ) 64 (bytes)
0x00000013 (RELENT) 8 (bytes)
0x6ffffffa (RELCOUNT) 6
0x00000006 (SYMTAB) 0x114
0x0000000b (SYMENT) 16 (bytes)
0x00000005 (STRTAB) 0x494
0x0000000a (STRSZ) 1238 (bytes)
0x00000004 (HASH) 0x96c
0x00000001 (NEEDED) Shared library: [libhello.so]
0x00000001 (NEEDED) Shared library: [libstdc++.so]
0x00000001 (NEEDED) Shared library: [libm.so]
0x00000001 (NEEDED) Shared library: [libc.so]
0x00000001 (NEEDED) Shared library: [libdl.so]
0x0000000e (SONAME) Library soname: [libhellojni.so]
0x0000001a (FINI_ARRAY) 0x3eac
0x0000001c (FINI_ARRAYSZ) 8 (bytes)
0x00000019 (INIT_ARRAY) 0x3eb4
0x0000001b (INIT_ARRAYSZ) 4 (bytes)
0x00000010 (SYMBOLIC) 0x0
0x0000001e (FLAGS) SYMBOLIC BIND_NOW
0x6ffffffb (FLAGS_1) Flags: NOW
0x00000000 (NULL) 0x0The data structure of the .dynamic segment is relatively simple
typedef struct {
Elf32_Word p_type;//0x52-0x55
Elf32_Off p_offset; //0x56-0x59
Elf32_Addr p_vaddr; //0x5a-0x5d
Elf32_Addr p_paddr; //0x5e-0x62
Elf32_Word p_filesz; //0x63-0x66
Elf32_Word p_memsz; //0x67-0x6a
Elf32_Word p_flags; //0x6b-0x6e
Elf32_Word p_align; //0x6f-0x73
} Elf32_Phdr;We need to traverse the .dynamic section to find the contents of d_tags: NEEDED and STRTAB.
Among them, NEEDED indicates the dependent library, but the element itself is not a string, it points to the index in the STRTAB table. So we also need to get the offset to STRTAB.
In this example we get 5 NEEDEDs:
0x00000001 (NEEDED)
Shared library: [libhello.so]
0x00000001 (NEEDED)
Shared library: [libstdc++.so]
0x00000001 (NEEDED)
Shared library: [libm.so]
0x00000001 (NEEDED)
Shared library: [libc.so]
0x00000001 (NEEDED)
Shared library: [libdl.so]
and the offset of STRTAB:
0x00000005 (STRTAB)
0x494
With this information, we can obtain the dependent SO files, and then recursively search for the dependencies of these SOs to get the complete dependency path. With the dependency path, we can load SO files in the order of dependencies.
Replace the load method globally
Since the SO file is loaded dynamically, the SO file address may be different from before. After solving the dependency problem of the load() method, we need to modify the new address and load logic.
At this time, you can encapsulate the entire SO obtaining dependency information and loading logic. After encapsulation, you can modify the bytecode at compile time through ASM, and replace all the calling system System.load() method instructions with your own encapsulated methods. If you don't want to package it yourself, you can also use ReLinker or Facebook's open source SoLoader.
System.loadLibrary() scheme
The loadLibrary() solution is simpler than load(), it does not require us to parse the ELF file to read the dependency information. Many times we also use this method to load the SO library.
Let's first analyze how the loadLibrary() method loads the SO library.
public static void loadLibrary(String libname) {
Runtime.getRuntime().loadLibrary0(Reflection.getCallerClass(), libname);
}
void loadLibrary0(Class<?> fromClass, String libname) {
ClassLoader classLoader = ClassLoader.getClassLoader(fromClass);
loadLibrary0(classLoader, fromClass, libname);
}
private synchronized void loadLibrary0(ClassLoader loader, Class<?> callerClass, String libname) {
if (libname.indexOf((int)File.separatorChar) != -1) {
throw new UnsatisfiedLinkError(
"Directory separator should not appear in library name: " + libname);
}
String libraryName = libname;
if (loader != null && !(loader instanceof BootClassLoader)) {
String filename = loader.findLibrary(libraryName);
if (filename == null &&
(loader.getClass() == PathClassLoader.class ||
loader.getClass() == DelegateLastClassLoader.class)) {
filename = System.mapLibraryName(libraryName);
}
if (filename == null) {
throw new UnsatisfiedLinkError(loader + " couldn't find \"" +
System.mapLibraryName(libraryName) + "\"");
}
String error = nativeLoad(filename, loader);
if (error != null) {
throw new UnsatisfiedLinkError(error);
}
return;
}
getLibPaths();
String filename = System.mapLibraryName(libraryName);
String error = nativeLoad(filename, loader, callerClass);
if (error != null) {
throw new UnsatisfiedLinkError(error);
}
}The above is the calling sequence of loadLibrary, and the key logic is in the loadLibrary0() method. In this method, the findLibrary() method of ClassLoader will be used to obtain the SO file. After the acquisition is successful, it is handed over to the native method nativeLoad() to load the SO file.
//BaseDexClassLoader
public String findLibrary(String name) {
return pathList.findLibrary(name);
}
//DexPathList.java
/** List of native library path elements. */
// Some applications rely on this field being an array or we'd use a final list here
@UnsupportedAppUsage
/* package visible for testing */ NativeLibraryElement[] nativeLibraryPathElements;
/** List of application native library directories. */
@UnsupportedAppUsage
private final List<File> nativeLibraryDirectories;
/** List of system native library directories. */
@UnsupportedAppUsage
private final List<File> systemNativeLibraryDirectories;
public String findLibrary(String libraryName) {
String fileName = System.mapLibraryName(libraryName);
for (NativeLibraryElement element : nativeLibraryPathElements) {
String path = element.findNativeLibrary(fileName);
if (path != null) {
return path;
}
}
return null;
}
private static NativeLibraryElement[] makePathElements(List<File> files) {
NativeLibraryElement[] elements = new NativeLibraryElement[files.size()];
int elementsPos = 0;
for (File file : files) {
String path = file.getPath();
if (path.contains(zipSeparator)) {
String split[] = path.split(zipSeparator, 2);
File zip = new File(split[0]);
String dir = split[1];
elements[elementsPos++] = new NativeLibraryElement(zip, dir);
} else if (file.isDirectory()) {
// We support directories for looking up native libraries.
elements[elementsPos++] = new NativeLibraryElement(file);
}
}
if (elementsPos != elements.length) {
elements = Arrays.copyOf(elements, elementsPos);
}
return elements;
}
DexPathList(ClassLoader definingContext, String dexPath,
String librarySearchPath, File optimizedDirectory, boolean isTrusted) {
...
this.definingContext = definingContext;
ArrayList<IOException> suppressedExceptions = new ArrayList<IOException>();
this.dexElements = makeDexElements(splitDexPath(dexPath), optimizedDirectory,
suppressedExceptions, definingContext, isTrusted);
this.nativeLibraryDirectories = splitPaths(librarySearchPath, false);
this.systemNativeLibraryDirectories =
splitPaths(System.getProperty("java.library.path"), true);
this.nativeLibraryPathElements = makePathElements(getAllNativeLibraryDirectories());
if (suppressedExceptions.size() > 0) {
this.dexElementsSuppressedExceptions =
suppressedExceptions.toArray(new IOException[suppressedExceptions.size()]);
} else {
dexElementsSuppressedExceptions = null;
}
}
private List<File> getAllNativeLibraryDirectories() {
List<File> allNativeLibraryDirectories = new ArrayList<>(nativeLibraryDirectories);
allNativeLibraryDirectories.addAll(systemNativeLibraryDirectories);
return allNativeLibraryDirectories;
}The above source code is a bit more, the brief summary is:
1. Obtain the directory of the system native library from java.library.pah;
2. Put the directory of the native library of the application into a List and pass it to the makePathElements() method;
3. The makePathElements() method returns a NativeLibraryElement[] array to the nativeLibraryPathElements variable after processing;
4. When searching for SO libraries, search from the directory contained in the variable nativeLibraryPathElements.
After knowing the principle, it is very simple to implement loadlibrary() to dynamically load SO.
We only need to add the dynamically loaded SO library storage directory to the first position of the nativeLibraryPathElements array through reflection, so that the system can find our SO file when searching according to the directory contained in nativeLibraryPathElements.
private static void install(ClassLoader classLoader, File folder) throws Throwable {
final Field pathListField = ReflectionUtils.findField(classLoader, PATH_LIST);
final Object dexPathList = pathListField.get(classLoader);
final Field nativeLibraryDirectories = ReflectionUtils.findField(dexPathList, NATIVE_LIBRARY_DIRECTORIES);
List<File> origLibDirs = (List<File>) nativeLibraryDirectories.get(dexPathList);
if (origLibDirs == null) {
origLibDirs = new ArrayList<>(2);
}
//去重
final Iterator<File> libDirIt = origLibDirs.iterator();
while (libDirIt.hasNext()) {
final File libDir = libDirIt.next();
if (folder.equals(libDir)) {
libDirIt.remove();
break;
}
}
origLibDirs.add(0, folder);
final Field systemNativeLibraryDirectories = ReflectionUtils.findField(dexPathList, SYSTEM_NATIVE_LIBRARY_DIRECTORIES);
List<File> origSystemLibDirs = (List<File>) systemNativeLibraryDirectories.get(dexPathList);
if (origSystemLibDirs == null) {
origSystemLibDirs = new ArrayList<>(2);
}
//创建新的list，方式并发修改异常
final List<File> newLibDirs = new ArrayList<>(origLibDirs.size() + origSystemLibDirs.size() + 1);
newLibDirs.addAll(origLibDirs);
newLibDirs.addAll(origSystemLibDirs);
final Method makeElements = ReflectionUtils.findMethod(dexPathList, MAKE_PATH_ELEMENTS, List.class);
final Object[] elements = (Object[]) makeElements.invoke(dexPathList, newLibDirs);
final Field nativeLibraryPathElements = ReflectionUtils.findField(dexPathList, NATIVE_LIBRARY_PATH_ELEMENTS);
nativeLibraryPathElements.set(dexPathList, elements);
}Since the implementation of each version of Android is slightly different, we need to adapt the version. For details, please refer to the implementation of Tencent Tinker.
dlopen problem
Everything was fine until Android N (7.0) arrived.
The Android platform has always been highly fragmented, and device manufacturers are reluctant to upgrade older devices to the new Android platform because of the amount of work required, which forces developers to test their applications on a large number of devices.
To solve this problem, Google released Project Treble.
Treble divides the Android platform into two parts: Framework and Vendor, and they interact with each other through stable interfaces. Therefore, through Treble, it is possible to upgrade the Android framework while keeping the supplier part unchanged.
But this led Treble to introduce two sets of native libraries: framework and vendor. In some cases, there may be libraries with the same name but different implementations in the two parts. Conflicts arise because the library's symbols are exposed to all code in a process.
To solve these problems, the Android dynamic linker introduces namespace based dynamic linking, which is similar to how Java class loader isolates resources.
With this design, each library is loaded into a specific namespace, and libraries in other namespaces cannot be accessed unless they are shared through a namespace link.
We know that whether it is the load() method or the loadLibrary() method, the dlopen() method is ultimately called to open the SO library. The dlopen() method eventually calls into the code in Linker.cpp.
Starting from Android N, the loader_library() method of Linker.cpp has performed permission judgment.
static bool load_library(android_namespace_t* ns,
LoadTask* task,
LoadTaskList* load_tasks,
int rtld_flags,
const std::string& realpath,
bool search_linked_namespaces) {
...
if ((fs_stat.f_type != TMPFS_MAGIC) && (!ns->is_accessible(realpath))) {
...
return false;
}
...
return true;
}The is_accessible() method will determine whether the given absolute path is in the following three lists:
- ld_library_paths
- default_library_paths
- permitted_paths
bool android_namespace_t::is_accessible(const std::string& file) {
if (!is_isolated_) {
return true;
}
if (!allowed_libs_.empty()) {
const char *lib_name = basename(file.c_str());
if (std::find(allowed_libs_.begin(), allowed_libs_.end(), lib_name) == allowed_libs_.end()) {
return false;
}
}
for (const auto& dir : ld_library_paths_) {
if (file_is_in_dir(file, dir)) {
return true;
}
}
for (const auto& dir : default_library_paths_) {
if (file_is_in_dir(file, dir)) {
return true;
}
}
for (const auto& dir : permitted_paths_) {
if (file_is_under_dir(file, dir)) {
return true;
}
}
return false;
}If the given path is not in the above three lists, the load_library() method will return false, causing the load to fail. The program will report the following exception:
java.lang.UnsatisfiedLinkError: dlopen failed: library "/storage/emulated/0/Android/data/org.zzy.nativetest/files/bundle/jni/arm64-v8a/libnativetest.so" needed or dlopened by "/apex/com.android.art/lib64/libnativeloader.so" is not accessible for the namespace "classloader-namespace"
at java.lang.Runtime.loadLibrary0(Runtime.java:1077)
at java.lang.Runtime.loadLibrary0(Runtime.java:998)
at java.lang.System.loadLibrary(System.java:1656)
at org.zzy.nativetest.so.SoTestActivity.onCreate(SoTestActivity.java:28)
at android.app.Activity.performCreate(Activity.java:8051)
at android.app.Activity.performCreate(Activity.java:8031)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1329)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3608)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3792)
at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:103)
at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2210)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:201)
at android.os.Looper.loop(Looper.java:288)
at android.app.ActivityThread.main(ActivityThread.java:7838)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)What could be wrong with this? We previously added the SO library storage directory to the nativeLibraryPathElements array by reflection, but since Android N, if the SO library storage directory is not in the above three lists, it will cause dlopen to fail to open.
Fortunately, there is no way out. In the log of Logcat, the values of ld_library_paths, default_library_paths, and permitted_paths are printed.
[name="classloader-namespace", ld_library_paths="", default_library_paths="/data/app/~~-ARvezkrvMHNPn30p76eTg==/org.zzy.nativetest-oouE9DDeRvsCdlkHBgca1g==/lib/arm64:/data/app/~~-ARvezkrvMHNPn30p76eTg==/org.zzy.nativetest-oouE9DDeRvsCdlkHBgca1g==/base.apk!/lib/arm64-v8a", permitted_paths="/data:/mnt/expand:/data/data/org.zzy.nativetest"]We can find that permitted_paths contains the sandbox directory of the application. That is to say, we can solve this problem as long as we put the SO file that needs to be dynamically loaded into the sandbox directory of the application.
Asset Delivery dynamically loads SO
After understanding the SO dynamic loading scheme, you can start to use Asset Delivery to dynamically load the SO library. The solution we use is still install-time mode, and the application is distributed when it is installed.
First of all, we still have to remove the original SO file from the project. It can be removed in the build.gradle file in the App module in the following way:
packagingOptions {
exclude 'META-INF/DEPENDENCIES'
if (packAAB) {
exclude 'lib/arm64-v8a/libxxxSDK.so'
exclude 'lib/arm64-v8a/libxxx.so'
exclude 'lib/arm64-v8a/libxxx_view.so'
exclude 'lib/armeabi-v7a/libxxxSDK.so'
exclude 'lib/armeabi-v7a/libxxx.so'
exclude 'lib/armeabi-v7a/libxxx_view.so'
}
}A variable is used here to control whether to generate an AAB package. If an APK package is generated, the SO file will not be removed.
Then put the SO library into the install_time_asset_pack Module created earlier. Remember to differentiate by ABI version. When the Application is initialized, copy the SO library to the application's sandbox directory. You need to check it here. If it already exists, don't copy it again, otherwise, it will cost performance to copy it every time the application starts. It is also necessary to judge whether the current device is 64-bit or 32-bit, and just copy the SO file corresponding to the corresponding ABI. Finally, use the solution we mentioned earlier to add the storage directory of the SO file to the nativeLibraryPathElements array through reflection.
To sum up , for those who use the System.load() method to load SO, it is necessary to encapsulate the parsing of ELF and the loading logic of SO, and replace the original loading logic by instrumenting at compile time.
For using the System.loadLibrary() method to load SO, the SO loading directory needs to be injected into the nativeLibraryPathElements variable through reflection. No matter which method is used to dynamically load SO, the storage path of SO must be placed in the sandbox directory of the application.
Every time the SO file is updated, the SO in the Asset Pack also needs to be updated accordingly.
Practice Results
Before publishing in AAB format, the APK size of the case application was 227.49 MB, which was far more than the limit of Google Play.
After publishing in AAB format, Google Play's package size limit changed: When a user downloads your app, the combined size of the compressed APK (eg base APK + config APK) required to install your app must not exceed 150 MB .
On a Pixel 5 phone, if you install the Case App, you will get the following APK:
By calculation, the total size of basic APK + configuration APK is: 84M. Asset bundle size does not count towards the Google Play upload limit.
References:
1. https://developer.android.com/guide/playcore/asset-delivery?hl=en-us2. https://jackwish.net/blog/2016/android-dynamic-linker.html
3. https://cloud.tencent.com/developer/article/1592672?from=article.detail.1751968
android
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。