System Dilemma and Software Complexity, Why Are Our Systems So Complex

Author: Nie Xiaolong (led pigeons)

Read A Philosophy of Software Design, the complexity of software design and architecture, are you a tactical tornado?

foreword

One day, a doctor and a civil engineer were arguing about "who is the oldest profession in the world". The doctor said: "God made Eve out of Adam's rib. This was the first surgical operation in history, so the oldest profession should be a doctor", and the civil engineer said: "Before Genesis, God created heaven from chaos With human beings, this is an earlier civil operation, so the oldest occupation should be civil engineering.” Then the software engineer came out dragging the keyboard and said, "Then who do you think created that chaos?"

It's not easy for an architect to add a basement to a 100-story building, but we do it all the time, and someone will always say to you, "It's a simple need." It's really not complicated to bury a mine in the soil, but the real scenario we often face is actually: "Add a mine to this minefield", and no one knows where there are mines in the minefield.

what is complexity

We keep saying that systems are complex, so what exactly is complexity? There are many definitions of complexity, among which the more representative ones are the complexity measure of the rational school proposed by Thomas J. McCabe in 1976, and the complexity cognition of the perceptual school proposed by Professor John Ousterhout.

measure of rationality

Complexity is not a new concept. As early as the 1970s, software was extremely complex, and the cost of development and maintenance was very high. In 1976, McCabe&Associates company began to test the structure of software and proposed McCabe Cyclomatic Complexity Metric, which we also call McCabe cyclomatic complexity. It measures the complexity of the software through multiple dimensions, so as to judge the current development/maintenance cost of the software.

perceptual cognition

High-complexity code is not necessarily good code, but low-complexity code is not necessarily good code. Professor John Ousterhout believes that the complexity of software is relatively rational analysis, and may be more biased towards perceptual cognition.

Complexity is anything that makes software hard to understand or to modify\
例句：Complexity is anything that makes software difficult to understand and modify.
John Ousterhout "A Philosophy of Software Design"

Today, 50 years later, Professor John Ousterhout in his book A Philosophy of Software Design mentions a very subjective insight that complexity is anything that makes software difficult to understand and modify.

Fuzziness and dependencies are the two main factors that cause complexity. Fuzziness produces the most direct complexity, making it difficult for us to understand what the code really wants to express. It's hard to change it. And dependencies lead to the continuous transmission of complexity, and the continuous overflow of complexity eventually leads to the infinite corruption of the system. Once the code becomes spaghetti, it is almost impossible to repair, and the cost will increase exponentially.

manifestations of complexity

Complex systems often have some very obvious features, which Professor John abstracts into three categories: Change amplification, Cognitive load, and Unknown unknowns. When our system has these three characteristics, it means that our system has gradually become more complex.

Symptom 1 - Change Zoom

Change amplification: a seemingly simple change requires code modifications in many different places.
例句：A seemingly simple change requires code changes in many different places.
John Ousterhout "A Philosophy of Software Design"

Change amplification refers to seemingly simple changes that require code changes in many different places. The typical representative is Ctrl-CV code development. The domain model lacks cohesion and convergence. When a certain segment of business needs to be adjusted, multiple modules need to be changed to adapt to the development of the business.

 /**
 * 销售捡入客户
 */
public void pick(String salesId, String customerId) {
  // 查询客户总数
  long customerCnt = customerDao.findCustomerCount(salesId);
  // 查询销售库容
  long capacity = capacityDao.findSalesCapacity(salesId);
  // 判断是否超额
  if(customerCnt >= capacity) {
    throws new BizException("capacity over limit");
  }
  // 代码省略 do customer pick
}

In the field of CRM, when sales pick up customers, they need to judge the storage capacity, and this code can indeed meet the needs. However, with the development of the business, the contracted customers should be adjusted to not occupy the storage capacity. In addition to sales pick-up, customers also include multiple scenarios such as supervisor distribution, leads distribution, manual entry, data purchase, etc. If the model is not collected for the storage capacity domain, a simple logic adjustment requires us to make appropriate adjustments in multiple scenarios. matching to meet the demands.

Symptom 2 - Cognitive Load

Cognitive load: how much a developer needs to know in order to complete a task.\
例句：How much knowledge does a developer need to complete a task.
John Ousterhout "A Philosophy of Software Design"

Cognitive load refers to how much knowledge a developer needs to complete a task. When using a functional framework, we want it to be simple to operate, and when deploying a complex system, we want it to have a clear architecture, but in fact, it reduces the cost of a task. Blindly pursuing high-end technology, designing complex systems, and increasing the cost of learning and understanding are all putting the cart before the horse.

TMF is the backbone of the entire Starlink and the core of the reusable and scalable architecture of the business platform. However, TMF is too complicated, and the cost of cognition and learning is very high. 99% (or should say 100%) of some of the expansion demands we face in our daily life are not suitable for TMF. Maybe through some design patterns or some if else, it may be more Suitable for solving our problems.

In addition, there are also some simple search scenarios that use streaming engines such as blink, simple background systems are built through DDD, and rule engines are used for state machine transitions for several commodity releases, all of which are complex cognitive loads. a kind of degree.

Symptom 3 - Unknown Unknown

Unknown unknowns: it is not obvious which pieces of code must be modified to complete a task\
例句：What code must be modified to accomplish the task.
John Ousterhout "A Philosophy of Software Design"

Unknown unknowns refer to what code must be modified to complete a task, or what information a developer must obtain in order to perform a task successfully. This is also the worst form of complexity that Professor John Ousterhout believes.

When you're maintaining a 20-year-old project, this kind of problem is relatively unsurprising. Due to the clutter of code and lack of documentation, you can't control an application with 5 million lines of code, and the code itself doesn't clearly express what it's supposed to say. At this time, the "unknown unknown" appears. You don't know whether the changed line of code can make the program run normally, and you don't know whether the change of this line of code will cause new problems. At this time, we found that only God can save those "Gods".

Why complexity arises

Why is the software becoming more and more complicated, and is it possible to avoid a catastrophe by reducing some mistakes? Looking back at those complex systems, we can find many factors that lead to system corruption.

I want to keep things simple and save trouble, but I don’t manage unreasonable content in a timely manner.
Lack of ingenuity and blindness to dirty code
Insufficient technical capabilities to deal with complex systems
The handover transition is missing, and Sanwu products are almost impossible to maintain

In addition to the above, many reasons can be thought of. But we found that they seem to have a common point - software engineers, it seems that the source of all complexity is the unqualified software engineers, so the root cause of some evil is ourselves?

1. A unified China and a divided Europe

The European continent is roughly the same size as China, but why is Europe divided and China unified? Some people say that their culture is different, some people say that their language barrier is the main reason, and some people say that they lack a Qin Shihuang. In fact, when we look back on European history, Europe really does not lack a unified empire. The Roman Empire once made the Mediterranean its own inland sea, and Napoleon ruled over 13 million square kilometers in his heyday. There have been great empires in Europe, but none of them have moved toward unity.

If we look at the map again, in fact, except for China and Russia, 99% of the countries in the world are small countries. Separation is the norm, and unity is abnormal. Teacher Ma also said that success is accidental, only failure is inevitable. Only very few countries have achieved great unification, so we should not ask why Europe is divided, but why China is unified. The same is true for our software. Complexity is the norm, and uncomplicatedness is abnormal.

2. The inherent complexity of software

The Complexity of software is an essential property, not an accidental one.
例句：The complexity of software is an essential characteristic, not an accident.
Grady Booch "Object-Oriented Analysis and Design with Applications"

Grady Booch put forward such a concept in Object-Oriented Analysis and Design with Applications, he believes that the complexity of software is inherent, including the complexity of the problem domain, the difficulty of managing the development process, the flexibility and characterization possible through the software The problem of discrete system behavior, these four aspects analyze the development of software must be accompanied by complexity, which is a characteristic that must be accompanied by the science of software engineering.

Everything, without exception, requires additional energy and order to maintain itself. I knew this in the abstract as the famous second law of thermodynamics, which states that everything is falling apart slowly.
Translation: Everything in the world needs extra energy and order to maintain itself, without exception. This is the famous second law of thermodynamics, which states that all things slowly fall apart.
--Kevin Kelly, "The Inevitable"

Kevin Kelly made a similar point in The Inevitable. He believes that everything in the world needs extra energy and order to maintain itself, and everything is slowly falling apart. Without the injection of external forces, things will gradually collapse. This is the law of everything in the world, not something we are doing wrong.

Software Architecture Governance Complexity

The external force injected into a software system is our software architecture, and every line of our future code. There are many kinds of software architectures, from the earliest single architecture to the later distributed architecture, SOA, microservices, FaaS, ServiceMesh and so on. All software architectures remain the same and address the complexity of the software.

The essence of architecture

The programming paradigm refers to the programming mode of the program. There are only three programming paradigms (paradigms) in the development of software architecture today, namely structured programming, object-oriented programming and functional programming.

Structured programming cancels goto, removes jump statements, and restricts and regulates the direct transfer of program control.
Object-oriented programming restricts the use of pointers, and restricts and regulates the indirect transfer of program control
Functional programming takes the lambda algorithm as the core idea, and restricts and regulates the assignment in the program

Five Object-Oriented Design Principles SOLID. Dependency inversion restricts the dependency order of modules, single responsibility restricts the scope of responsibility of modules, and interface isolation restricts the provided form of interfaces.

The essence of software is constraints. The code of the product cannot be written in the order field, and the method of the data layer cannot be written in the business layer. 70 years of software development did not tell us what to do, but taught us what not to do.

increasing complexity

Software complexity doesn't just disappear out of thin air, it just increases. There are 3 perspectives on increasing complexity:

Ambiguity creates complexity, dependencies propagate complexity
Complexity is often not caused by a single disaster
We can easily convince ourselves that a little complexity from the current changes is no big deal

Xiao Li once complained to me that this code is really disgusting, it took a long time to understand, and the code is very rigid, and just this requirement needs to be changed here, the code is really like a mess. I asked what he did with it in the end, and he said, I added another lump to it.

programming thinking

tactical programming

In fact, Xiao Li's approach is not an individual behavior. Maybe we have all struggled with this when we encounter complex codes. Professor John's programming method is called "tactical programming". The most important feature of tactical programming is fast, and it also has the following characteristics.

Must be the fastest
Don't spend too much time searching for the best design
Every programming task introduces some complexity
Refactoring will slow down the current task, so keep it as fast as possible

 @HSFProvider(serviceInterface = AgnDistributeRuleConfigQueryService.class)
public class AgnDistributeRuleConfigQueryServiceImpl implements AgnDistributeRuleConfigQueryService {

    @Override
    public ResultModel<AgnDistributeRuleConfigDto> queryAgnDistributeRuleConfigById(String id) {
        logger.info("queryAgnDistributeRuleConfigById id=" + id);
        ResultModel<AgnDistributeRuleConfigDto> result = new ResultModel<AgnDistributeRuleConfigDto>();
        if(StringUtils.isBlank(id)){
            result.setSuccess(false);
            result.setErrorMsg("id cannot be blank");
            return result
        }
        try {
            AgnDistributeRuleConfigDto agnDistributeRuleConfigDto = new AgnDistributeRuleConfigDto();
            AgnDistributeRuleConfig agnDistributeRuleConfig = agnDistributeRuleConfigMapper.selectById(id);
            if(agnDistributeRuleConfig == null){
                logger.error("agnDistributeRuleConfig is null");
                result.setSuccess(false);
                result.setErrorMsg("agnDistributeRuleConfig is null");
                return result
            }
            this.filterDynamicRule(agnDistributeRuleConfig);
            BeanUtils.copyProperties(agnDistributeRuleConfig, agnDistributeRuleConfigDto);
            result.setSuccess(true);
            result.setTotal(1);
            result.setValues(agnDistributeRuleConfigDto);
        } catch (Exception e) {
            logger.error("queryAgnDistributeRuleConfigById error,", e);
            result.setSuccess(false);
            result.setErrorMsg(e.getMessage());
        }
        return result;
    }
}

Let's look at the above code, which is a piece of business logic for querying distribution rules. Although the function can work, there are many irregularities

Facade layer defines all logic - no structural layering
Business and technology are not separated - coupling interface information and business data
Try catch all over the sky - lack of unified exception handling mechanism
No normalized log format - log format confusion

But it is undeniable that he must be the fastest at the moment. This is one of the characteristics of tactical design, always advancing according to the current fastest delivery solution, and even many organizations encourage this way of working, in order to make the function work faster, only focus on short-term benefits and ignore long-term value.

Tactical Tornado

Almost every software development organization has at least one developer who takes tactical programming to the extreme: a tactical tornado.
例句：Almost every software development organization has at least one developer who takes tactical programming to the extreme: Tactical Tornado.
John Ousterhout "A Philosophy of Software Design"

Those who take tactical programming to the extreme are called tactical tornadoes. Tactical Tornado trades corrupt systems for the most efficient solution at the moment (perhaps he doesn't think so). Tactical tornadoes also have the following characteristics:

is a prolific programmer, no one gets things done faster than a tornado
It can always leave traces of the destruction after the tornado for future generations to clean up
it's really curly

Some organizations even see the tactical tornado as a hero, so why can it be done so much and so quickly? Because he puts the cost into the future. The biggest cost of software engineering is maintenance. Every time we change the code, it should be a sorting of the historical code, not a single function accumulation. Tornado can win the present, but will lose the future, and this lost future may require the whole team to pay with him.

strategic programming

Professor John proposed that strategic programming is the opposite of tactical programming. Strategic programming focuses more on long-term value, is not satisfied with functional work, and is committed to making excellent designs to meet the demands for future expansion (be careful, don't overdo it). Strategic design has the following four characteristics

Working code is far from enough
Introducing unnecessary complexity is unacceptable
Continually making small improvements to the system design
Investment mentality (every engineer needs to make a continuous small investment of 10-20% in a good design)

Professor John Ousterhout mentioned the total cost of strategic design and tactical design in the book A Philosophy of Software Design. Strategic design can effectively control software costs over time, but tactical design increases linearly over time. This is the same meaning that Martin Fowler mentioned in the book Patterns of Enterprise Application Architecture about data-driven and domain-driven governance of complexity, and to commit to long-term value investments.

The dilemma and evolution of the system

No system is inherently complex. In order to complete tasks quickly, new complexity is continuously introduced until the system gradually deteriorates. The complexity of infinite growth and infinite transmission makes it more and more difficult to "complete quickly" software requirements. When one day we realize the complexity of the system and then try to iterate the software through strategic design, you will find that it is difficult to move forward. A small modification requires a lot of infrastructure repairs. In the end, we have to bow down to the cost and continue to pass The tactical design is infinitely condescending.

A condition that is often incorrectly labeled software maintenance. To be more precise, it is maintenance when we correct errors; it is evolution when we respond to changing requirements; it is preservation when we continue to use extraordinary means to keep an ancient and decaying piece of software in operation. Unfortunately, reality suggests that an inordinate percent- age of software development resources are spent on software preservation.
例句：We always say we need to "maintain" these old systems. To be precise, in the process of software development, maintenance is only when we fix bugs; evolution is when we respond to changing needs; when we use extreme means to keep old and outdated software working , which is to protect (stubbornly). It turns out that we spend more time dealing with the last situation.
Grady Booch "Object-Oriented Analysis and Design with Applications"

As Grady Booch makes a point in Object-Oriented Analysis and Design with Applications, it's really a bit of sloppiness when we use extreme measures to keep old and stale software working. We are careful, integration tests, grayscale releases, timely rollbacks, etc., we are not "maintaining" them, but in an ugly way to allow these ugly code to continue to succeed. When the code becomes spaghetti, it will be nearly impossible to fix, the cost will increase exponentially, and it seems that such code already exists in our system and probably continues to increase.

Architectural Pseudo-theory

In architectural design, there are always poems and distant places that software engineers believe in, but the unreachable utopia is not necessarily a beautiful holy land that is far away, but may actually be an architectural design that is unhelpful or even harmful to the system. Here are two possible architectural pseudo-theories.

1. Good code is self-explanatory

Comments do not make up for bad code
Translation: Comments are not a remedy for poor code
Martin Fowler "Clean Code"

Martin Fowler mentions in Clean Code that comments are not a remedy for bad code, and I've always believed that if the code is good enough, comments aren't needed. But in fact this is a false proposition, Professor John commented that 'good code is self-documenting' is a delicious myth.

 /**
 * 批量查询客户信息
 */
public List<CustomerVO> queryCustomerList(){
  // 查询参数准备
  UserInfo userInfo = context.getLoginContext().getUserInfo();
  if(userInfo == null || StringUtils.isBlank(userInfo.getUserId())){
    return Collections.emptyList();
  }
  LoginDTO loginDTO = userInfoConvertor.convert(userInfo);
  // 查询客户信息
  List<CustomerSearchVO> customerSearchVOList = customerRemoteQueryService.queryCustomerList(loginDTO);
  Iterator<CustomerSearchVO> it = customerSearchVOList.iterator();
  // 排除不合规客户
  while(it.hasNext()){
    CustomerSearchVO customerSearchVO = it.next();
    if(isInBlackList(customerSearchVO) || isLowQuality(customerSearchVO)){
      it.remove();
    }
  }
  // 补充客户其他属性信息
  batchFillCustomerPositionInfo(customerSearchVOList);
  batchFillCustomerAddressInfo(customerSearchVOList);
  return customerSearchVOList;
}

In this code, we can easily see what this function does in 5 seconds, and know some business rules inside it. Infinite private method encapsulation will make the code chain too deep, and the disassembly of infinite classes will cause more mesh dependencies. There are at least 3 points, so we must not abandon comments.

The implication of the inability to accurately name the name is that the abstract entity hides the details, we cannot give it all the information on a name, and the necessary annotations can be perfectly assisted.
The elaboration code of the design idea can only realize the design but not the design. This is why some complex architecture designs need the support of the document rather than the 'self-explanation' of the code. The gap between the document and the code is filled by comments.
The power of the mother tongue is especially suitable for us Chinese. Sometimes it is not because there are fewer comments and more code, so we subconsciously look at the code first. It is the culture we have experienced for decades that gives us a completely different perception of Chinese and ABC.

2. Always pursue the most elegant

Rebs once boasted that the code he wrote was as elegant as poetry, and the pursuit of elegant code should be a holy place in the heart of every software engineer. But sometimes there is some inelegance, and the existence of some 'seemingly irrational' does not mean that it is wrong, but sometimes we continue to deviate on the road of pursuing more elegance.

The goal of software architecture is to minimize the human resources required\
to build and maintain the required system.
例句：The ultimate goal of software architecture is to meet the needs of building and maintaining the system with minimal human cost
Robert C.Martin "Clean Architecture"

In the book Clean Architecture, Robert C. Martin mentioned the ultimate goal of the architecture, which is to meet the needs of building and maintaining the system with minimal labor costs. Architecture is always a tool for us to solve complexity. If the current system is not complicated, we don’t need to over-modify and optimize it for so-called elegance. Keeping the cost at a low level is the best solution for software.

Simple business systems do not apply DDD architecture, and weak interaction scenarios do not require front-end and back-end separation. Even Chief Designer Deng has formulated a system of 'socialism with Chinese characteristics' in planning the development of new China. Don't blindly follow some dogmatic concepts, choose what suits you, and control it within a controllable range, neither excessive nor lacking. After all, there is no absolute elegance, or even absolute correctness.

write at the end

Many people think that doing business development is not that challenging, but the opposite is true. The hardest bugs to solve are those that cannot be reproduced, and the hardest problem domains are those of uncertainty. Business is often the most complex, and uncertainty-oriented design is the most complex design. The hardest thing about the software engineering discipline is abstraction, because it has no standards, no methods, or even right or wrong. How to find a path that is neither excessive nor lacking in the inherent complexity of software is a lifelong task for software engineers, and it may never be reached. Maybe we are already on the road.

See books:

Pay attention to [Alibaba Mobile Technology], Ali's cutting-edge mobile dry goods & practice will give you thoughts!