"Business development is encountering more and more environmental problems, which seriously affect development efficiency. Some seem to be packaging problems on the surface, but behind it is the corruption of the engineering structure."
background
In recent years, the negative impact of the high complexity of iOS engineering has gradually been exposed. Many students have been "ruined" by the slow and complex packaging of iOS, and the efficiency of business development has been greatly affected. I remember a classmate once complained to me. He packaged several modules and integrated them into the main project. In this process, every step in the process failed to package, and it took more than half a day in total.
Alibaba.com is a cross-border type B e-commerce business. It started to develop an iOS client in 2012. In order to support business development, component transformation was carried out in 2016, evolving modular architecture from a single engineering architecture. With the development of business and wireless technology, the client has evolved from a small modular project to a giant project. The team has built more than 100 self-maintenance modules, including business modules, architecture facilities, Hybrid containers, Flutter containers, dynamic technology, basic middleware and other capabilities. On the surface, the engineering architecture is evolving in an orderly manner, but there are already chaos inside. The module relationship is chaotic, and there are more and more circular dependencies and reverse dependencies. A large number of modules do not conform to the LLVM Module standard, the spec file is incomplete, and the header file reference is not standardized. Because of the irregular engineering, Cocoapods cannot be upgraded and can only use the old version 1.2 and 1.5, which is technically behind for more than 3 years.
In order to completely solve the problem and improve the business development experience, the Alibaba ICBU-side architecture team conducts a comprehensive management of the iOS engineering architecture. I also wrote an article to record my thoughts. Interested students are welcome to guide and communicate.
Steve Mcconnell "Code Complete": "The primary technical mission of software: managing complexity."
What are the problems caused by architecture corruption?
Problem 1: The complexity of module packaging is high
Mixed engineering environment
In 2016, Alibaba's client-side componentization was not completely done. Many modules were only separated in form. In fact, there are problems of reverse dependency and circular dependency. In 2017, the team wanted to do Frameworkization and found that the modules could not be packaged and compiled separately. Therefore, in order to compile and pass the module, we developed a compatible script, added all the framwwork and header files to the project searchPath, and let the module directly read all the dependencies in the profile of the synchronized main project. Since there is compatible logic, the spec file can be compiled without writing dependency descriptions, so no one maintains the spec file anymore, and the cross-module header file references are getting more and more messy.
Environment incompatibility & module build failure
Because of the existence of circular dependencies and irregular header files, the module compilation script adds a lot of workaround logic and is compatible with header file indexes. As a result, the module Cocoapods environment cannot be upgraded, and it stays at version 1.2. With more and more middleware and community swift technologies, the main project Podfile uses the new syntax of cocoapod 1.5. The environment is starting to be incompatible. At the same time, when the module parsed the Podfile of the main project, it could not recognize the new syntax of cocoapod 1.5, and the module construction failed.
90 man-days of development resources are wasted every year
After the module packaging fails, the development needs to analyze the log to investigate the cause of the packaging failure. If the analysis fails, you need to find the architecture team for support. If a module fails to be packaged, the requirements will remain stuck and cannot be integrated, which will block testing or other development work.
According to the development feedback, it is estimated that an average module packaging failure will consume 2 hours of R&D resources. According to statistics, during Q1, the total number of module packaging failures reached more than 200 times, and 70% of the packaging failures were caused by high complexity. Each packaging failure wasted 2 hours, which is equivalent to a waste of 90 man-days of R&D resources each year.
Robert Martin "Clean Architecture": "No matter how dedicated you are and how many overtime you work, you will still struggle with a bad system, because most of your energy is not dealing with development needs, but dealing with chaos."
Problem 2: Slow packaging of the main project
If the module is not standardized, and swift middleware needs to be referenced, it cannot be an independent static library and can only be integrated into the main project in the form of source code. This leads to the need to compile a large amount of source code when the main project is packaged, and the average packaging time is 12 minutes slower than projects such as Handtao and Youku. The main project packaging is required for requirements testing, integration, bug fixes, and troubleshooting. Slow packaging will block development and testing. A bi-weekly iteration packaged 70 times and wasted 14 hours.
Problem 3: The engineering environment is unstable
The Cocoapods environment cannot be upgraded, only the old version 1.2 and 1.5 can be used. But the old version of the environment is not maintained, and the environment is extremely fragile. For example, if someone publishes an illegal spec, the Pod Update will hang. Because the modules are not standardized, various inexplicable compilation problems will occur during source code development. Business development and debugging efficiency will be very low, wasting a lot of time.
Problem 4: Swift development is difficult
In recent years, swift has become popular, and there have been more and more swift middlewares in iOS communities and groups. However, Swift modules strictly abide by the "LLVM Modules" specification, and do not allow circular dependencies, external dependencies to be explicitly declared, and angle brackets for header file references, otherwise errors such as "could not build module xxx" and "No such module" will appear. . Under high standards, it is difficult to introduce Swift into our engineering development. Although we can not use Swift ourselves, the trend of Swiftization of middleware between the group and the three parties is irreversible.
In the past two years, the project of Alibaba.com has introduced many Swift middlewares, and has also independently developed many swift components, which has completely detonated the problem of R&D efficiency. Related modules are frequently packaged abnormally. Irregularities are complex and complicated. Often, after solving one compiler error, another error occurs, and there is no way for future generations to exhaust it. Finally, various uncontrollable risks began to appear in the system.
In addition, most of our modules do not comply with the LLVM Modules specification. If the business needs to use Swift or reference Swift middleware, it will take a lot of time to solve the adaptation problem. According to the data of agile iteration, requirement A plans 10 man-days, and the actual consumption is 20 man-days, and requirement B plans 6 man-days and the actual consumption is 10 man-days.
Deterioration of complexity to a certain extent, it must enter the level where there are many unknown unknowns
Question 5: Difficulties in cleaning up historical code
In recent years, many old businesses have been offline or transformed. However, due to the severe coupling between modules, many old codes have been afraid to delete, which also led to the continuous expansion of package size.
Difficulties and strategies of structure corruption governance
The scope of influence is wide, and governance is difficult to promote
In 2020, I initiated the architecture governance project in the iOS technology stack, and launched the iOS development of various business lines to manage together, but I was in a dilemma. On the one hand, no resources were invested in business development. On the other hand, the calling relationships among many business modules are chaotic, and governance risks are high, and everyone is afraid to move casually.
Data analysis, push from top to bottom
The chaos of the iOS project has seriously affected business development, and everyone's time is wasted on resolving compilation and packaging issues. IOS development students in various businesses are troubled, and many start to report that the difficulty of packaging has seriously affected the development efficiency.
To this end, I began to comprehensively sort out the data of the R&D process. On the one hand, I counted the module construction failure data and the time-consuming package of the main project, and then compared it with the data of other clients; on the other hand, I conducted interviews on business development to understand the waste of resources data from the user’s perspective, and supplement Links that cannot be counted in the R&D platform. Finally, successfully quantified the negative impact of engineering chaos on R&D efficiency into concrete data.
With the results of data analysis, there will be a hand to promote, and the governance of the structure can be promoted from the top to the bottom.
solution
Look at the big picture and sort out module dependencies
The first difficulty is the unclear relationship between modules. The dependency list in the module description file is empty, and the relationship between the modules is like a ball of wool.
If the relationship between the modules is not clear, the governance project cannot be dismantled, and the cost cannot be estimated. Therefore, we must first look at the overall situation and analyze the overall module dependencies.
I developed a tool for analysis. First find all the files of the module, use regular matching to find the external header files that it imports, and get the set of externally referenced header files. Then search the Pods directory of the main project, match the external module to which the header file belongs, and finally aggregate to get a complete module dependency tree.
The next step is visualization. After visualization, the complexity of the module relationship can be viewed more intuitively, which facilitates the formulation of governance plans. I used Dot language to describe the module relationship, which can automatically generate a dependency graph of the entire project, or a dependency graph of a specific module.
Rely on inversion and hierarchical governance
The second difficulty is the complex dependency conditions of governance.
The criterion for successful module governance is that all modules in the entire dependency tree have no circular dependencies and all comply with the LLVM Module specification. For example, in the governance business module A, there is a module C in the dependency tree of module A. Module C has a circular dependency or does not conform to the Module specification. When module A is packaged, an exception will be reported. However, Cocoapod and XCode only report one exception each time and cannot be analyzed. All the problems of the entire dependency tree.
Our project maintains more than 130 modules, including more than 200 third-party libraries and middleware modules. In addition to its own dependencies, business modules also have many indirect dependencies, and the dependency tree is very complex. In this case, the complexity of direct governance business modules is extremely high, and the governance process will be chaotic.
In the example above, module C, module I, and module G are central modules with complex relationships. For example, "Module I" directly depends on 30 external modules and indirectly depends on more than 100 modules. Its direct coupling relationship has 5 cycles, and its indirect coupling relationship has 15+ cycles. If you directly manage "Module I", you need to decouple 15 cyclic relationships and transform more than 100 modules into Module. According to this kind of thinking, the logic of modification is extremely complicated, and it is very likely that the governance will not proceed in half.
In order to solve this dilemma, I layered and classified the modules. There are three basic logics of division:
- The more the underlying module dependency, the simpler;
- Modules without circular dependencies are easier to manage;
- The completed governance module can be ignored.
According to this idea, I first sort out the level of the module, and then manage it from the bottom layer to layer. When the underlying modules are all managed, the burden of relying heavily on modules will also be greatly reduced. When the decoupling of the low-level cyclic dependencies is completed, the upper-level modules do not need to deal with the indirect cyclic dependencies.
Finally, using the four-quadrant analysis method, the modules are divided into 4 groups, 1 basic module has no circular dependency, 2 basic module has circular dependency, 3 business module has no circular dependency, 4 business module has circular dependency, and each group is managed in order.
Automated repair
The third difficulty is the large amount of code changes. Module governance faces many sub-problems, such as "incomplete description of the dependencies of module spec files", "umbralla header files are not missing", "public header file references are not standardized", and "circular dependency decoupling". It is very difficult to fix only "the dependency description of the module spec file is incomplete".
The way to complete the dependency is to find the import description "(import <xxxFramework/xxx.h)" of all source files, and all the frameworks since the statistics. Then look up the module to which it belongs based on the name of the framework. In addition, there are a lot of import formats that are not standardized. Some are directly quoting the file name (import "xxx.h"), and some are path-based quoting (import <xxx/xxx/xxx.h>). When encountering this kind of non-standard quotation, You also need a global search to find which module belongs to. For example, the dependency description of module A is empty, but in fact it depends on more than 20 modules. Module A has more than 60 source files, each source file import reference is an average of 10 lines, a total of 600 lines of reference code. If you manually analyze these 600 lines of code, it is estimated that it will take a day. This is just to modify one of the issues, not including "umbralla header files are not missing", "public header file reference is not standardized", "circular dependency decoupling".
Therefore, purely manual governance simply does not work, and efficiency must be improved through automated methods. So I developed an architecture management engine that can be used to analyze module dependencies, fix incomplete spec dependency descriptions, automatically generate umbralla header files, modify non-standard header file references, and so on. Automated repair tools can cover 95% of the code changes, and the development is only responsible for modifying routing, service API, code migration, module splitting and merging, and other logical changes with major changes.
The architecture management engine can not only do architecture governance, it can also be used as a team management tool, such as analyzing the activity of the git warehouse, setting CodeReview rules in batches, and recording the log of the R&D process.
The following code uses the ruby language and the cocoapods-core framework. The main function is to analyze the module import code and repair the module's podspec dependency.
require 'cocoapods'
require 'cocoapods-core'
require 'xcodeproj'
def DependencesAnalyser.main(contextHelper, projectToolPath, moduleName, allModuleNames)
# 1修复import格式
iOSProjectDir = contextHelper.projectDir
podDir = contextHelper.podDir
iOSProjectName = contextHelper.projectName
# 读取source_files路径
sourceDir = contextHelper.sourceDir
if sourceDir.nil?
puts '[error]依赖修复失败，找不到正确的sourceDir'
return nil
end
# 1 读取源文件目录下的所有.h和.m文件的路径
allheadPaths = getSourceHeaderPath(sourceDir)
# 2 遍历所有源文件，读取文件的每一行，正则匹配出所有import的代码行
# 2.2 如果是import "" 或者 import <xx.h> 规则引用的,解析出依赖的头文件
importHeaders = parseHeaderNameFromQuotationImport(allheadPaths)
# 2.1 如果是import <xx/xx.h> 规则引用的直接截断出framework名
dependences = parseFrameworkNameFromAngleBracketsImport(allheadPaths)
# 3 如果是import "" 规则引用的，判断引用的头文件是否存在Pod目录下，如果存在记录所在Pod的Framework名
# 3.1 读取主工程Pod文件目录下所有依赖库的.h文件的路径
dependencesFromQuatationImport = findFrameNameFromQuatationImportHeader(podDir, importHeaders)
dependences = dependences + dependencesFromQuatationImport
filtedDependences = filterDepencences(dependences, projectToolPath, moduleName, allModuleNames)
# 4 读取podspec，修改dependence后，输出新的podspec文件
modify_spec_file(filtedDependences, contextHelper)
# 5 输出依赖关系文件
return filtedDependences
endArchitecture and business cooperation governance
The fourth difficulty is that decoupling involves a lot of business logic. A lot of code is branch logic of the business, it is difficult to test after refactoring, and it is easy to fail online if it is not fully verified.
Decoupling involves a lot of business logic, and the best way to reduce risk is to leave it to business development for modification. Therefore, the architecture group takes the lead in the horizontal iOS engineering governance project, the architecture group provides governance solutions and tools, and the business development is responsible for business logic decoupling. Business decoupling uses four methods, routing scheme, service API, sinking of public components, and module merging.
Give a few typical decoupling scenarios:
Scenario 1: product module that is product recommendation, and the order module also needs to be used, so the order module will rely on the product module in a reverse direction, forming a circular relationship. The way to decouple this scenario is to separate the basic components from the product module, and the order module depends on the basic components.
Scenario 2: When the product module jumps to the order module, the product model is used as the input parameter of the API. The order module relies on the product module in order to reference the product model. This way of scene decoupling is to use the routing URL Scheme protocol to convert the model into the input parameter of the query in the URL.
Long-term guarantee mechanism
After structural governance, the circular dependency of modules and the modula specification have been resolved, but there may be secondary corruption in the future. Of course, we do not want to re-governance after a period of time, so we start from the bayonet of architecture design and R&D process, optimize the architecture and process, and prevent subsequent secondary corruption.
Architecture optimization
- Systematically define and divide modules, increase the cohesion of module logic, and avoid the need to develop multiple modules at the same time for one demand. Converge the number of modules and reduce the maintenance cost of modules;
- ICBU business modules will eventually be integrated into the host and guest, and the unified version arbitration in the main project can reduce complexity and avoid conflicts in the version statements of the modules. The module dependency description only declares the module name, not the version number, and the module version of the main project is synchronized as the version arbitration when packaging.
Convergence module engineering
If the modules maintain their own construction projects, long-term maintenance will inevitably lead to great differences in the build configuration. On the one hand, if the build configuration cannot be upgraded in a unified manner, the cost of architecture governance and technology upgrade will be high; on the other hand, if the module has a build problem, the troubleshooting cost will also become high.
Therefore, we have built a packaging script to dynamically generate a module project every time it is packaged. The module no longer maintains an independent project, and the build configuration is unified to the podspec file.
When the module is packaged, the construction project of the module is dynamically created
require 'cocoapods'
require 'cocoapods-core'
require 'xcodeproj'
require 'rubygems'
project_creater = ProjectCreater.new(ContextHelper.tempProjectPath, ContextHelper.projectName)
project_creater.transform
require 'pathname'
class ProjectCreater
def initialize(root, name)
@project_path = Pathname.new(root).realpath
@project_name = name
end
def transform
puts "ProjectCreater-开始"
prepare
puts "ProjectCreater-开始重命名"
rename
puts "ProjectCreater-完成"
end
private
def prepare
xcodeproj_path = @project_path.join("#{@project_name}.xcodeproj").to_s
if File.exist?(xcodeproj_path)
`rm -rf #{xcodeproj_path}`
end
end
def rename
Dir.glob(File.join(@project_path.join("Podfile").to_s)).each do |file|
content = File.read file
content = content.gsub(/POD_NAME/, @project_name)
File.open(file, 'w') { |f| f << content }
end
Dir.glob(@project_path.join('PROJECT.xcodeproj').to_s + '/**/*').each do |name|
next if Dir.exist? name
if File.extname(name) == '.xcuserstate'
next
end
text = File.read name
text = text.gsub("PROJECT",@project_name)
File.open(name, "w") { |file| file.puts text }
end
scheme_path = @project_path.join("PROJECT.xcodeproj/xcshareddata/xcschemes/").to_s
File.rename(scheme_path + "PROJECT.xcscheme", scheme_path + @project_name + ".xcscheme")
File.rename(@project_path.join("PROJECT.xcodeproj").to_s, @project_path.join(@project_name + ".xcodeproj").to_s)
end
endCocoaPod and Xcode compilation bayonet
- The CocoaPods environment of the main project is upgraded to version 1.9.1, and circular dependencies will be detected during update;
- Remove the compatible Header search Path logic, the module must use the standardized header file reference method to compile and pass;
- Open XCode modular compilation check, if the module header file reference is not standardized, it will not compile.
Devops build bayonet
- Strictly follow the integration order process, the integration order needs to be compiled and passed before it can be integrated;
- Add static scanning plug-ins in the build process to detect module specifications.
Summarize
Architecture corruption is like a "flu virus", and its negative effects are difficult to perceive and quantify.
For the technical team, to avoid architecture corruption, the technical team must have a higher awe of technology. Compared with waiting for the fire to spread, we should give more substantive support and encouragement to those who put out the fire in time.
For architects, architects are required to be proficient in developing tools. In the face of complex architecture problems, we must first conduct a comprehensive analysis, disassemble system problems, find the least complex governance path, and consciously seek data support to obtain the support of the team.
Finally, from the perspective of architectural governance. Client engineering is a natural centralized architecture, and it is easy to cause compilation problems due to environmental conflicts. Therefore, when we design a componentized architecture, we must ensure that the environment of the modules is completely independent to avoid a centralized architecture. Structural governance is not the end point. After the governance is completed, there must be a mechanism to prevent corruption to avoid secondary corruption.
reference
- 《Clean Architecture》https://book.douban.com/subject/26915970/
- 《Code Complete》https://book.douban.com/subject/1432042/
- DOT Language https://graphviz.org/doc/info/lang.html
- LLVM Module https://clang.llvm.org/docs/Modules.html#introduction
We are hiring!
Alibaba.com is the world's largest Class B international e-commerce platform, with technical talents in the fields of long-term signature terminal architecture, live broadcast, short video, IM, and e-commerce. If you are passionate about iOS, Android, Flutter and other mobile technologies, welcome to join the Alibaba.com client R&D team, you can Base Hangzhou and Shenzhen.
Resume
Contact email: blacktea.hw@alibaba-inc.com
WeChat ID: blackteachinese
" Taobao client diagnostic system upgrade actual combat "
, 3 mobile dry goods & practice for you to think about every week!
ios
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。