node_modules slimming

cause

scene one:
The current project has undergone slash-and-burn development, and then access the cli tool for centralized management and packaging, so the dependencies in the project,
What is the degree of overlap with the dependencies in the cli tool, and whether his version is the same, and whether there is redundant code

Scenario two:
A library in the project has been upgraded, it depends on the V3 version of the A library, and the current project depends on the V2 version of the A library. At this time, the packaging is obvious, and different versions of this package will be entered at the same time.

Scenario three:
There is a corresponding dependency library in the current deps, but it is not used in the business code

Because of the above scenarios, we need a tool to address these situations

thinking 🤔

How to solve these scenarios, what is the solution

For scenario 3, there is already a library: depcheck

Simple principle: By comparing the files in the project import or require with the dependencies, and finally generate the dependency list

want a certain configuration
(Through the actual call, it is found that there are still some problems: the code in the submodule cannot be detected, and the same is true for the babel configuration plugin detection in the dependency)

Scenarios 1 and 2 are not the same as 3. They have existing libraries, but they are slightly duplicated. All need to be detected for the library.

The current plan is to run it through a node script

• Check node_modules or lock file for multiple versions of the same library
• The node_modules file has too many levels, and the lock file is a layer of mapping. Consider starting here
• Make sure the lock file is the latest
• Open the local website, and visualize the results for the visual display (after the actual operation, this scene is abandoned, and the specific reasons will be detailed below)

develop

Here we first solve the problem of scenario 1

scene one

In the above thinking, there is already a solution for this scenario, namely the depcheck scenario, but his configuration needs to be rewritten:

check configuration update

 const options = {
    ignoreBinPackage: false, // ignore the packages with bin entry
    skipMissing: false, // skip calculation of missing dependencies
    ignorePatterns: [
        // files matching these patterns will be ignored
        'sandbox',
        'dist',
        'bower_components',
        'tsconfig.json'
    ],
    ignoreMatches: [
        // ignore dependencies that matches these globs
        'grunt-*',
    ],
    parsers: {
        // the target parsers
        '**/*.js': depcheck.parser.es6,
        '**/*.jsx': depcheck.parser.jsx,
        '**/*.ts': depcheck.parser.typescript,
        // 这里 ts 类型可能会出问题, 但是经过实际的运行和文档说明是没问题的
        '**/*.tsx': [depcheck.parser.typescript, depcheck.parser.jsx],
    },
    detectors: [
        // the target detectors
        depcheck.detector.requireCallExpression,
        depcheck.detector.requireResolveCallExpression,
        depcheck.detector.importDeclaration,
        depcheck.detector.exportDeclaration,
        depcheck.detector.gruntLoadTaskCallExpression,
        depcheck.detector.importCallExpression,
        depcheck.detector.typescriptImportEqualsDeclaration,
        depcheck.detector.typescriptImportType,
    ],
    // specials: [
    //     // Depcheck API在选项中暴露了特殊属性，它接受一个数组，以指定特殊分析器。
    // ],
    // 这里将会覆盖原本的 package.json 的解析
    // package: {
    // },
};

Then call the configuration again:

 // 默认即当前路径
const check = (path = process.cwd()) => depcheck(path ,options)

Finally add the print result:

 console.log('Unused dependencies:')
unused.dependencies.forEach(name=>{
    console.log(chalk.greenBright(`* ${name}`))
})
console.log('Unused devDependencies:'); 
unused.devDependencies.forEach(name=>{
    console.log(chalk.greenBright(`* ${name}`))
})

An example showing the result of the call:

scene two

Instruction technology selection:

1. commander

The most recommended, but also the most downloaded, with a download volume of 8kw+

2. package-lock.json

For the lock file, the default npm and its corresponding analysis, and now there are yarn , pnpm are more popular, but they are generally used when packaging on the server. npm instruction

development of instructions

Planned Instructions

• check // operation of default scene one
• check json // Parse the .lock file and print the packages that take up space
• check json -d // print the result to a file

first step

Definition of directive:

 const main = () => {
    const program = new commander.Command();
    program.command('check')
        .description('检查使用库')
        .action((options) => {
            // 显示一个 loading
            const spinner = ora('Loading check').start();
            
            // check
            check()
            
        }).command('json').description('解析 lock文件').option('-d, --doc', '解析 lock 文件, 将结果保存')
        .action(async (options) => {
            // 显示 loading
            const spinner = ora('Loading check').start();
            // 执行脚本
            // 额外判断 options.open
            deepCheck(spinner, options)
        })
    
    program.parse();
}

The second step is to parse the file

First we get the file content through fs:

 const lockPath = path.resolve('package-lock.json')

const data = fs.readFileSync(lockPath, 'utf8')

For lock data analysis:

 const allPacks = new Map();
    
    Object.keys(allDeps).forEach(name => {
        const item = allDeps[name]
        if (item.dev) {
            // dev 的暂时忽略掉
            return
        }
        
        if (item.requires) {
            // 和item.dependencies中的操作类似
            setCommonPack(item.requires, name, item.dependencies)
        }
        
        if (item.dependencies) {
            Object.keys(item.dependencies).forEach(depsName => {
                const depsItem = item.dependencies[depsName]
                if (!allPacks.has(depsName)) {
                    allPacks.set(depsName, [])
                }
                const packArr = allPacks.get(depsName);
                
                packArr.push({
                    location: `${name}/node_modules/${depsName}`,
                    version: depsItem.version,
                    label: 'reDeps', // 标识为重复的依赖
                    size: getFileSize(`./node_modules/${name}/node_modules/${depsName}`)
                })
                allPacks.set(depsName, packArr)
            })
        }
    })

Finally, a loop is used to calculate the package with the largest temporary space:

 // 创建一个排序数据, push 之后自动根据 size 排序
    let topSizeIns = createTopSize()
    
    allPacks.forEach((arr, name, index) => {
        if(arr.length <= 1){
            return
        }
        let localSize = 0
        arr.forEach((item, itemIndex) => {
            const size = Number(item.size)
            localSize += size
        })
        
        topSizeIns.push({items: arr, size: localSize})
    })

    // 最后打印结果, 输出可选择文档
    if (options.doc) {
        fs.writeFileSync(`deepCheck.json`, `${JSON.stringify(mapChangeObj(allPacks), null, 2)}`, {encoding: 'utf-8'})
    }
    
    // 打印 top5
    console.log(chalk.yellow('占用空间最大的 5 个重复库:'))
    topSizeIns.arr.forEach(itemObj => {
        const common = itemObj.items.find(it => it.label === 'common')
        console.log(chalk.cyan(`${common.location}--${itemObj.size.toFixed(2)}KB`));
        itemObj.items.forEach(it => {
            console.log(`* ${it.location}@${it.version}--size:${it.size}KB`)
        })
    })

third step

Graphical solution ( deprecated )

Let's talk about the implementation first:

1. Convert the data generated by json to the data required by the chart
2. Start local service, reference echart and data

Data conversion:

 let nodes = []
let edges = []
packs.forEach((arr, name, index) => {
    let localSize = 0
    arr.forEach((item, itemIndex) => {
        const size = Number(item.size)
        nodes.push({
            x: Math.random() * 1000,
            y: Math.random() * 1000,
            id: item.location,
            name: item.location,
            symbolSize: size > max ? max : size,
            itemStyle: {
                color: getRandomColor(),
            },
        })
        localSize += size
    })
    
    topSizeIns.push({items: arr, size: localSize})
    
    const common = arr.find(it => it.label === 'common')
    if (common) {
        arr.forEach(item => {
            if (item.label === 'common') {
                return
            }
            edges.push({
                attributes: {},
                size: 1,
                source: common.location,
                target: item.location,
            })
        })
    }
})

Start the service:

The service does not use a third-party library, but adds a node http service:

 var mineTypeMap = {
    html: 'text/html;charset=utf-8',
    htm: 'text/html;charset=utf-8',
    xml: "text/xml;charset=utf-8",
    // 省略其他
}

const createServer = () => {
    const chartData = fs.readFileSync(getFile('deepCheck.json'), 'utf8')

    http.createServer(function (request, response) {
        // 解析请求，包括文件名
        // request.url
        if (request.url === '/') {
            // 从文件系统中读取请求的文件内容
            const data = fs.readFileSync(getFile('tools.html'))
            response.writeHead(200, {'Content-Type': 'text/html'});
            // 这里是使用的类似服务端数据的方案, 当然也可以使用引入 json 的方案来解决
            const _data = data.toString().replace(new RegExp('<%chartData%>'), chartData)
            // 响应文件内容
            response.write(_data);
            response.end();
        } else {
            const targetPath = decodeURIComponent(getFile(request.url)); //目标地址是基准路径和文件相对路径的拼接,decodeURIComponent()是将路径中的汉字进行解码
            console.log(request.method, request.url, baseDir, targetPath)

            const extName = path.extname(targetPath).substr(1);
            if (fs.existsSync(targetPath)) { //判断本地文件是否存在
                if (mineTypeMap[extName]) {
                    response.setHeader('Content-Type', mineTypeMap[extName]);
                }
                var stream = fs.createReadStream(targetPath);
                stream.pipe(response);
            } else {
                response.writeHead(404, {'Content-Type': 'text/html'});
                response.end();
            }
        }
    }).listen(8080);

    console.log('Server running at http://127.0.0.1:8080/');

    opener(`http://127.0.0.1:8080/`);
}

export default createServer

Rendering:

From this picture, you can see the approximate problem point:

1. Too many dependent packages, resulting in cluttered data display
2. The circle is displayed according to the real size of the package, the gap is too large, the large one is tens of thousands of kb, and the small one is dozens of kb
   The maximum size 200 is temporarily idle in the picture

So this feature is temporarily disabled

other solutions

When using pnpm , I found that he can solve the problem of the size of the redundant package, so I also list it here

Summarize

The current package has been built: @grewer/deps-check can try to use

For the three common scenarios proposed at the beginning of the article, this package can basically solve the problem

Some optimization points can be proposed later, such as the replacement of some packages ( moment replace dayjs , lodash and lodash.xx cannot exist at the same time. Wait)
These require long-term maintenance and management

After reading this article, if you have any good suggestions, please leave a message and let me know