An unimagined path for front-end project dependency management

Brief introduction (convenient for students who want to quickly understand the conclusion of the article content)

1. First of all, the conclusion is that Node.js divides dependencies into two parts, dependency and devDependency, but the way to share the same node_modules folder is no longer applicable in the current increasingly complex front-end project development process.
2. Tool dependencies and business dependencies share the same node_modules folder, which will make the development and construction process inefficient and fragile, including the following problems: node_modules bloated and inefficient, version drift, Monorepo architecture is not compatible, and the CI construction process is fragile.
3. The solution is to separate concerns and place business dependencies and tool dependencies separately. One idea is to incorporate business dependencies into the version management system. Tool dependencies are precompileed and released by a dedicated build team. Developers can choose the version for global installation.

node_modules status

This picture must be familiar to the front-end students. The dependency problem of node_modules, which is currently complaining, has not been solved, but it has become more and more obvious from looking back in 2020. When we look at many packages, the status is "WTF, when did I install this dependency?", you can look at the node_modules in your front-end project. If there is no 500M, you are embarrassed to say that you are a front-end, and in these Among the dependencies, how many are the dependencies that really need to be used in the final product? How much is the dependence of tools in the development and construction process?

The author did a simple experiment:

• Separate installation of React and ReactDOM only takes up 3.9M space.
• Installing Vue alone will only take up 3.6M of space.
• Use create-react-app to create a blank React project, occupying 189.6M space.
• Use vue-cli to create a blank Vue project, occupying 164.5M space.

Although it is not representative of the whole situation, it must also reflect some problems. You can recall which tools you used in your development process: packaging tools, development servers, testing frameworks, various linters, TypeScript compilers, Babel, and so on. These tools all have their own dependencies. Children give birth to grandchildren, and grandchildren give birth to children. There are endless descendants and grandchildren. Soon, your hard drive will no longer fit. And these tool dependencies are only used in the development and construction process, or even at different stages. For example, many unit tests are actually run during the online CI process, but they will all be installed into the node_modules file. In the folder, mixed with business dependence.

Problems caused by DevDependency

Tool dependency and business dependency sharing node_modules will not only cause problems such as folder size increase inexplicably and slow npm install, but also cause dependency version drift and cause various inexplicable BUGs.

Front-end engineering has developed to today and has entered an era of skyrocketing complexity. This is due to the explosion in the types of resources to be processed by the front-end. Different resources require different tools for processing, and superimposed on the high-speed iteration of front-end technology. In 5 years, the build tools have changed from Grunt and Gulp to Webpack, Parcel, Rollup, and there will be Vite, Snowpack, Esbuild, etc. in the future. Such high-speed tool updates, multiplied by the growth of resource types, bring tools The rapid increase in complexity also brings a strong dependence on tool version control. Under the current semantic version management method, a small business dependent npm install may cause various unknown version drifts of tool dependence, which brings great challenges to the stability of the entire build process. Deleted and reinstalled for a while, the version is not correct for the crematorium.

I want to say two more words about Lock. The original intention of Lock is good. I hope to solve the problem of inconsistency of dependent versions through the Lock file. However, in the process of using it, I must have encountered a conflict with the Lock file when the new package of npm install was used. What should we do at this time? Delete node_modules and reinstall it, then congratulations, and congratulations to mention the version drift package~

On the other hand, as front-end projects become more and more complex, more and more front-end projects adopt the Monorepo architecture and need to go through the online CI process for release. The current devDependency design method cannot adapt to Such a construction method.

my-mono-repo
├── package.json
└── packages
    ├── A
    │   ├── package.json
    │   ├── node_modules
    │   └── src
    ├── B
    │   ├── package.json
    │   ├── node_modules
    │   └── src
    └── C
        ├── package.json
        ├── node_modules
        └── src

Let me talk about the Monorepo problem first. The above is a very natural directory design. A, B, and C each have their own node_modules, and this brings about a problem, where is devDependency installed, if installed in their respective node_modules, then a lot of The space is actually occupied by redundant tool dependencies. If it is said to be installed in the node_modules of the parent directory in a unified manner, then it is necessary to solve the version problem of solving different subdirectories, even if tools such as lerna can be used for automatic management, in the subdirectories The following npm install may also cause some common dependent versions in the parent directory to drift, and introduce unknown changes to the development and construction of other subdirectories.

In addition to the inapplicability of the Monorepo architecture, the design of devDependency will also bring various problems during the online CI process. First of all, redundant node_modules brings more space and network overhead, which makes the environment initialization process longer in the CI process. In fact, all the tool dependencies in devDependency are not used in the entire CI process, such as packaging. Processes such as, Lint, and testing require different tools, but each step requires downloading the full amount of devDependency; on the other hand, the upgrade of business dependencies often makes tool dependencies passively upgrade unknowingly, leading to previous caching The devDependency is invalid. If the cache is not cleared in time and the version is updated, it is easy to cause inconsistencies between the build and the development environment and cause unknown version problems. The source of all these vulnerabilities lies in the complexity of the current front-end project, which has exceeded the load of the originally designed devDependency. Putting devDependency and dependency indiscriminately in node_modules, just like when beating eggs, put the egg shell I also stirred it in, and then I had to pick out the egg shell from the beaten egg liquid, helpless~

Unimagined road

At this point, we have talked about a lot of problems caused by devDependency, so how do we solve these problems?

First of all, we must first define the root cause of the problem. Here I will directly state my conclusion. The cause of all these problems lies in the lack of separation of concerns between tool dependency and business dependency. If you have some experience in using other programming languages, you can recall that whether it is Python, Java, or C++, tool dependencies and business dependencies have never been mixed together. This is because of the role of the two, the frequency of updates, and The requirements for use are different. For business dependence, we ultimately want to integrate it into the product. It has business attributes and needs to be able to solve business problems in time. The update frequency will be more frequent, especially in the current Monorepo and private NPM is prevalent now; and for tool dependence, our needs are stability, uniformity, and efficiency, without frequent changes, or even if they change, they should be insensitive and transparent to business developers, not to mention business dependence. The changes lead to the instability of the tool.

Now that we have found the source of the problem, our solution is obvious:

On the one hand, for the tool dependency of devDependency, we detach it from node_modules. Furthermore, we can encapsulate these tool dependencies into a team-specific build tool, and then each business development student only needs to install it to On the whole, you don't even need Babel in your own project, just install a few dependencies that you need for business. This kind of development experience is not good! The packaged tools can be handed over to a dedicated construction team for maintenance, or even packaged into a two-level package, such as pkg , deno compile further improve performance.

On the other hand, for the business dependency of dependency, we can continue to stay in node_modules, and further, we can include node_modules into the version control of git. Since the tool dependencies have been detached, the rest are business dependencies, which were originally built into the final product. We need to ensure strong consistency in various environments. At the same time, the size of node_modules that the tool depends on will also be detached. Reducing it to a reasonable level and putting it under the control of git will not bring much extra overhead.

Regarding whether incorporating node_modules into git management will cause excessive overhead, here we can imagine that in any long-running project, the business relies on its own code, and the maximum ratio is 1:1, which is impossible. Appears in a mature commercial project, and the code written by ourselves has not yet introduced a large dependency. At the same time, because the business dependency is ultimately packaged into a product and released on the Internet, we also have the motivation to minimize the size of the business dependency. . In summary, the cost of incorporating business dependencies into version management is acceptable relative to the benefits of strong consistency.

Now that there is a direction for guidance, we can now begin to carry out specific transformations:

First of all, the easiest and quickest way is to split dependency and devDependency into two package.json respectively, and then raise the directory structure of devDependency to a level, using node.js's module layer-by-layer search feature, which is basically not You need to change any code to complete the split of dependency and devDependency. The specific directory structure is as follows

|-- node_modules # 安装 devDependency 的依赖
|-- package.json # 记录 devDependency 的依赖
|-- myApp
    |-- node_modules # 安装 dependency 的依赖
    |-- src # 业务代
    |-- package.json # 记录 dependency 的依赖
|-- .gitignore

Next, in the .gitignore file, we exclude the node_modules that install the devDependency dependency, and the node_modules that install the dependency dependency need to be kept in the git repository. The specific content is as follows

node_modules
!myApp/node_modules

Incorporating dependency into git management here has advantages and disadvantages. The disadvantage is that the files downloaded by git clone will become larger. The advantage is that on the one hand, we can use git to ensure strong consistency of business dependencies, as long as the code is checked out from the same branch. , The business dependencies must be exactly the same. On the other hand, if a classmate adds or modifies the business dependencies, they can also be recorded by git, so that the changes can be traced back. Furthermore, it can also be done in response to this situation. Special dependency review, which was easily overlooked when business dependencies could be changed by just changing package.json. Therefore, relative to the problem of size, the benefits of stability and consistency brought by git are included. I think it will be bigger.

Finally, it is recommended to lock the version of the dependent library in the outermost package.json or hand it over to specialized students for unified management. Business students only need to care about their business dependencies.

Of course, the above plan is just the simplest transformation, mainly to give you an idea that you can refer to. The basic idea is separation of concerns, tools belong to tools, and business belongs to business. For the actual situation of different projects, you can use the above On the basis of ideas, he further explored and found the maintenance method that best suits his team.

A little vision for the future

In the development process of front-end engineering, the role of node.js can be described as a great contribution. It can even be said that it is with node.js that the front-end is really led to the field of engineering, although it was also through Java or Python in the past. Dealing with the application of front-end code, but for front-end programmers who need to master another language, they always feel that there is a layer of window paper, and it is node.js that pierces this layer of window paper.

We should not ignore the contribution of node.js to front-end engineering. At the same time, we should also be aware of the limitations of node.js. After all, the original design purpose of node.js, whether it is Common.js module specifications, It is also the dependency path search method of module.paths. In the initial design, it was more for the use of node.js for server-side programming services. The dependency and devDependency installation methods used were not specifically for front-end engineering. It is designed, and one of the problems that this leads to is that when we enjoy the engineering capabilities brought by node.js, due to the characteristics of the front-end project itself, directly adopting the dependency management method of node.js becomes fragile and unreliable. .

The development of front-end engineering to this day is also facing more and more challenges:

1. Under the bundleless trend led by vite, what should be done with front-end engineering?
2. How to ensure the efficiency of development and construction under the monorepo architecture?
3. How to develop and build under the micro front-end architecture?

These new situations are brand-new situations that people have never faced when designing node.js. We can’t ask the designers of node.js to consider all situations from the beginning. That’s right. Realistically, what we should do more is to analyze the nature of the problem, and go further on the shoulders of our predecessors to find a solution that is more suitable for the current situation.

This article is just an attempt to start with dependency and devDependency to analyze some of the current problems of using node.js for front-end engineering. It is hoped that it will give readers a different perspective. If you have any questions, please feel free to comment. Leave a message and discuss together:)~

Author: ES2049 | Magic Circle

The article can be reprinted at will, but please keep this link to the original text.
You are very welcome to join ES2049 Studio if you are passionate, please send your resume to [e-mail]