Remember a pit caused by too smart production tools

background

Recently, I am working on a CLI tool for processing proto files. Before using proto files, there are generally two ways:

1. Directly reference the proto file and dynamically generate JS code at runtime
2. Generate the corresponding JS file through the protoc tool and reference it in the project

The latter will have higher performance, because the compilation process is before the program runs, so the latter is generally used.

Problem phenomenon

Because it is a general-purpose tool, the proto file will also be dynamic. A simple simulation of possible scenarios in the local environment, and then the terminal executes the protoc command:

# grpc_tools_node_protoc 为 protoc Node.js 版本的封装
grpc_tools_node_protoc --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto

It was found that everything was running normally, so I wrote the corresponding code into the script, replaced part of the path with variables, submitted the code, issued the package, installed locally, and verified it.

As a result, there was such a problem:

Could not make proto path relative: ./protos/**/*.proto: No such file or directory
/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc.js:43
    throw error;
    ^

Error: Command failed: /usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto
Could not make proto path relative: ./protos/**/*.proto: No such file or directory

    at ChildProcess.exithandler (child_process.js:303:12)
    at ChildProcess.emit (events.js:315:20)
    at maybeClose (internal/child_process.js:1021:16)
    at Socket.<anonymous> (internal/child_process.js:443:11)
    at Socket.emit (events.js:315:20)
    at Pipe.<anonymous> (net.js:674:12) {
  killed: false,
  code: 1,
  signal: null,
  cmd: '/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto'
}

Shocking, and even more incredible, when I cmd to the terminal and ran it again, I found that everything was normal.

After being shocked, I re-examined my own code implementation.

Troubleshooting

First of all, I suspect that the way to execute the command is wrong. The current one is exec , because grpc_tools_node_protoc is also a packaged Node.js module, so I took a look at its source code , and found that the source code was execFile , and then Go to the Node.js documentation to see if there is a difference between the two, because the No such file or directory error message is 060f19efeb21e2. First of all, I suspect that the path is incorrect because the CLI is installed globally, so I took a look at the two APIs for current In the definition of working directory, I found a few differences as expected:

exec the cwd parameters as described Current working directory of the child process. Default: process.cwd(). , and execFile the cwd parameters as described Current working directory of the child process. .

It seems that the latter does not have a default value, so is it caused by the wrong working directory, so we added the 060f19efeb227a parameter to the cwd

As a result, there is no difference, it is still an error.

So look a bit Node.js on exec and execFile difference API implementation, to confirm whether the cwd reason, and found exec internal call is execFile , it can be basically confirmed both in cwd the default value of the parameter processing and There will be no difference. At the same time, the DEBUG information output is added to the source code. Check that cwd is indeed the directory where we are currently running.

Since the problem is not here, then we have to analyze it from other places, because we are more confident in our own code (and indeed there are not a few lines), so I carefully looked at grpc-tools and found that the code is like this:

var protoc = path.resolve(__dirname, 'protoc' + exe_ext);

var plugin = path.resolve(__dirname, 'grpc_node_plugin' + exe_ext);

var args = ['--plugin=protoc-gen-grpc=' + plugin].concat(process.argv.slice(2));

var child_process = execFile(protoc, args, function(error, stdout, stderr) {
  if (error) {
    throw error;
  }
});

cmd parameter output by the above program error is actually the result of args
Out of curiosity, we added a DEBUG log to the source code, and found a magical situation.

When we exec , the output is like this:

[
  '/usr/local/bin/node',
  '/usr/local/bin/grpc_tools_node_protoc',
  '--js_out=import_style=commonjs,binary:./src/main/proto',
  '--grpc_out=grpc_js:./src/main/proto',
  './protos/**/*.proto'
]

And we execute the command directly through the terminal, and the output is like this:

[
  '--plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin',
  '--js_out=import_style=commonjs,binary:./src/main/proto',
  '--grpc_out=grpc_js:/./src/main/proto',
  './protos/examples/example-base-protos/kuaishou/base/base_message.proto'
]

The last parameter of the two is actually different.

So I tried to put the detailed file path of proto into the command, and ran it through exec again, and found that everything was normal, so the problem lies in the last proto file path. The co-author protoc does not support wildcard files such as ** enter.
So a new question is here, why two different operating modes will cause the incoming parameters to change.

Because the executable files of the Node.js module are registered through the package bin, there is reason to suspect that NPM did some small actions, so I wrote a shell file, and the output is very simple:

echo $* # 输出所有的参数

Using the reverse elimination method, if we can get **/*.json sh test.sh **/*.json , then it can basically be determined that it is the ghost of NPM.

The result output is:

package-lock.json package.json proto.json

You can get a complete file path by outputting through the terminal, which means that at least it is not some operations of NPM.

Suddenly I thought of a possibility, type bash and then run the same command sh test.sh **/*.json . Sure enough, we got **/*.json .

I thought that my terminal was using zsh , so I looked through the corresponding documents and found the corresponding instructions: https://zsh.sourceforge.io/Doc/Release/Expansion.html , [turn to 14.8.6 by yourself Recursive Globbing]
When I first realized the problem, a line of oh my f**king zsh drifted through my heart.

zsh will match the path recursively, and then expand it in the execution parameters, the final positioning of the reasons is because zsh a convenience feature led me mistaken protoc a function, culminating in a non- zsh environmental exposure problems.

to sum up

The problem encountered this time is very strange, but the reason is very helpless. Fortunately, the process of investigation is quite rewarding. I was forced to read the source code of some modules and have a deeper understanding of the entire compilation process of the proto file. .
After getting used to using zsh, some of the capabilities it provides make me mistakenly think that it is provided by the program, and I did not consider that aspect during the entire troubleshooting process, and I don’t know if such "easy to use" tools will be used. Other scenes give me some surprises.