Failure Analysis | MongoDB 5.0 error Illegal instruction solution

Author: Ren Zhongyu

A member of the DBA team of Aikesheng, good at failure analysis and performance optimization, welcome to discuss technical issues related to articles.

Source of this article: original contribution

*The original content is produced by the open source community of Aikesheng, and the original content shall not be used without authorization. For reprinting, please contact the editor and indicate the source.

---

At the sound of a cannon last July, MongoDB Inc sent us MongoDB 5.0. This version not only brings the core feature - time series collections, but if you use it carelessly, it will also bury some small "pits" for us; if your The environment is preparing to install, trial or upgrade to MongoDB 5.0, so please stop and discuss.

Phenomenon

First note, my Linux version is CentOS Linux release 7.2.1511.

After installing the latest mongodb-5.0.x, executing mongo or mongod directly reports the error Illegal instruction .

[root@10-186-61-38 mongodb]# cd mongodb-linux-x86_64-rhel70-5.0.5/bin/
[root@10-186-61-38 bin]# ./mongo --help
Illegal instruction
[root@10-186-61-38 bin]# ./mongod --help
Illegal instruction

Full of confusion, there is no problem with using the lower version of MongoDB 4.4.9.

[root@10-186-61-38 mongodb]# cd mongodb-linux-x86_64-rhel70-4.4.9/bin/
[root@10-186-61-38 bin]# ./mongo --help
MongoDB shell version v4.4.9
usage: ./mongo [options] [db address] [file names (ending in .js)]

Check

Execute the mongo or mongod command to get 2 lines of demsg log error:

[root@10-186-61-38 bin]# dmesg -T
······
[Thu Dec 23 18:05:13 2021] traps: mongo[16596] trap invalid opcode ip:7f0ad9fa90da sp:7ffe9deaa050 error:0 in mongo[7f0ad7f86000+2c8c000]
[Thu Dec 23 18:05:17 2021] traps: mongod[16597] trap invalid opcode ip:7f3b1e329a6a sp:7ffc8fb540e0 error:0 in mongod[7f3b1a355000+5110000]

It can be seen that the command execution failure is caused by invalid opcode, which seems to be related to a certain instruction set of the operating system.

Searching in the MongoDB community with doubts and keywords, I found that there are similar errors:

In this case, although the operating system is Ubuntu, the system error message is similar. The reason for the error is: The basic requirement of MongoDB 5.0 version is that the CPU of the server where it is located needs to support the AVX instruction set.

Looking carefully at the official documentation, you can see that installing MongoDB version 5.0 does require a CPU that supports the AVX instruction set:

The CPU models that currently support the AVX instruction set can refer to the link:

https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX

Check the CPU of my own server, it does not support AVX (note: if the AVX instruction set is supported, the flags field will print the 'avx' string):

[root@10-186-61-38 ~]# cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 13
model name  : QEMU Virtual CPU version 2.5+
······
flags       : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm
······

CPU testing with AVX support

Find a server with a CPU that supports the AVX instruction set for testing.

The new version of MongoDB 5.0 command is executed normally.

AVX instruction set

AVX (Advanced Vector Extensions, advanced vector extension instruction set) is an instruction set in the x86 architecture microprocessor

• Proposed by Intel in March 2008 and first supported in the Sandy Bridge series processors released in Q1 2011. AMD started supporting AVX in the subsequent Bulldozer series processors released in Q3 2011. So older CPU architectures may not support the AVX instruction set. AVX is the SSE extension architecture of the X86 instruction set, such as IA16 to IA32, which increases the register XMM 128bit to YMM 256bit, so theoretically, the computing performance of the CPU will be increased by 2 times.

At present, the official documentation of MongoDB only states that the installation of MongoDB 5.0 needs to rely on the server CPU to support the AVX instruction set, but it does not explain the specific reasons for the support;

Only one article about MongoDB with AVX "Getting storage engines ready for fast storage devices" was retrieved on the Internet. It mentioned that the highly optimized AVX-based memcpy method can be used to copy data from the memory-mapped area to the buffer of another application. ; Guess that the current release of version 5.0 includes the update at the WiredTiger storage engine level mentioned in the article (the article mentioned that the read throughput after updating the engine has increased by 63%), and the optimization update of the underlying storage engine depends on the support of avx.

https://engineering.mongodb.com/post/getting-storage-engines-ready-for-fast-storage-devices

in conclusion

If you need to install or upgrade to the new version of MongoDB 5.0, you must ensure that your server CPU can support the AVX instruction set architecture in advance. Check the command as follows:

grep avx /proc/cpuinfo

Reference documentation:

https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610

https://docs.mongodb.com/manual/administration/production-notes/