MQTT server security test

The IoT market is growing rapidly with explosive growth. As the scale of devices continues to grow and business logic becomes more complex, the security of the IoT platform infrastructure becomes more and more important. Whether the specific implementation of the protocol by the IoT platform is complete and whether the parsing process of a specific message is secure has become the top priority.

This requires comprehensive testing of the complex standards and specified behavioral norms in the protocol. At the same time, considering various interferences and attacks that may exist in actual use, the testing process also needs to cover various non-standard abnormal packets to analyze the fault tolerance and processing capability of the target platform for abnormal situations.

Fuzz testing is a very effective testing tool. This article will take EMQ X as an example to introduce how to use fuzzing tools to find possible defects and vulnerabilities in the implementation of the MQTT server/MQTT client protocol.

What is fuzzing

Fuzz testing (fuzzing, fuzzing) is a software testing technique. The core idea is to input automatically or semi-automatically generated random data into a program and monitor program exceptions such as crashes and assertion failures to detect possible program errors such as memory leaks. Fuzzing is often used to detect security vulnerabilities in software or computer systems.

Fuzz testing can be used as white box, grey box or black box testing. Usually used for black-box testing with higher returns.

Source:

Ready to work

Test tool and object selection

In this article, we choose Defensics as a fuzzing tool. Defensics is a black-box fuzzing tool developed by Synopsys, which provides fuzzing suites for a large number of file formats, network protocols, and interfaces. Based on the MQTT v3.1 protocol standard, it uses a large number of automatically generated MQTT packets to test the Broker/Client, helping developers and testers to improve software security.

The test suite for MQTT v5 is currently not released

We will take the open source MQTT message server - EMQ X as an example to test the security of its protocol implementation. EMQ X is a large-scale and elastically scalable cloud-native distributed IoT MQTT message server open sourced by EMQ Yingyun Technology. It can efficiently and reliably connect a large number of IoT devices, process messages and event stream data in real time with high performance, and help build critical business systems. IoT platforms and applications.

Test environment preparation

This test is carried out in the Arch Linux environment, rolling update to the latest version, using EMQX 5.0-beta.2-8be2aaf7 for testing.

In addition, before proceeding to the next step, you need to download the Defensics installation package, the test suite installation file with the suffix .install , and the DEFENSIC executable file from Synopsys to provide the FlexNet license server to verify the license status.

Deploy License Server (FlexNet)

Synopsys Defensics uses FlexNet to manage licenses. FlexNet Server needs to be deployed in the network environment where the Defensics fuzzer is executed to manage licenses obtained from Synopsys (ie license.lic file).

Optionally use Systemd User Unit to start FlexNet Daemon on a local deployment, configured as follows. The license.lic certificate file and the DEFENSIC ( Vendor Daemon ) executable file will be located in the same directory. FlexNet will search the $PATH for the Vendor Daemon executable for authentication.

Of course, if more testers need to use Defensics, it can also be deployed on a dedicated certificate server to provide certificate authentication services to more users. For other details and specific parameters, please refer to Defensics and FlexNet Publisher related documents.

lmgrd.service Systemd User Unit" title="lmgrd.service Systemd User Unit">

Then use the command systemctl --user enable --now lmgrd.service start the Vendor Daemon provided by the authentication server Synopsys. Defensics is ready to begin testing with licensing certification.

Other configuration

There may be display problems and blurred fonts in Linux systems. You can refer to Java Runtime Environment fonts - ArchWiki for configuration.

Install Defensics and Test Suite

Execute the .sh installer as root to install. And it is recommended to check the generation option /usr/local/bin/Defensics of the startup script during the installation process.

If all goes well, you can see the verified license status in File -> License Manager after starting Defenses. The test suite can then be installed and loaded.

Defensics Test

Basic configuration

Set the IP address and port number of the MQTT Server in the basic configuration, as well as the MQTT Client configuration for testing.

Where MQTT defaults to port 1883 (port 8883 when TLS/SSL is enabled).

If the MQTT Server has enabled client authentication or message topic permissions, more detailed configuration of the two clients used for testing is required.
In addition, Defensics also provides more advanced payload fuzzing and testing based on TLS/SSL connections. However, this test only involves fuzzing related to the MQTT v3.1.1 protocol standard, so no configuration is required.

After configuring the corresponding field values, Defensics will connect to the MQTT Server with the specified Client ID, username and password, and will use the specified MQTT topic for message publishing and topic subscription, namely PUBLISH and SUBSCRIBE .

For more advanced and complex tests, you can use the Edit sequence function to edit the corresponding configuration file to change the behavior when connecting or disconnecting, such as automatic subscription after connecting, and publishing messages immediately after connecting.

operability test

After completing the configuration, perform an operability test in Interoperability to verify whether different packets can be sent and received normally. With normal connectivity to the MQTT Server, test groups that can be executed will be marked in green.

Advanced configuration

In this part, the user is allowed to configure the specific test case execution process, but generally the default configuration is sufficient.

This includes control over the test case execution process, such as timeout thresholds, number of repetitions, number of attempts, etc.

In addition, users can also perform network-related configuration according to the actual situation to obtain test results under different network scenarios. At this point, you can also choose to perform automatic configuration based on the target IP of the MQTT Server.

Anomalies that may be detected can also be assessed according to the vulnerability classification method provided by CVSS (Common Vulnerability Scoring System).

Instrumentation

Instrumentation refers to the observation and control of the system under test during a test, with the goal of observation being to detect any failures caused by the test. Meters can also be used to restart or otherwise control the system under test while running a test.

Most test suites have default detection enabled and no additional configuration is required. And the nature of this default configuration varies from test suite to test suite.

Select test case

Defensics provides a total of more than 1 million test cases for the MQTT v3.1.1 protocol standard that can be used to perform comprehensive fuzz testing on the target system. At the same time, it also supports users to select and combine test cases based on these subdivisions to focus on analysis and solve problems.

This time, we selected some use cases for testing, including three groups of use cases CONNECT-DISCONNECT PUBLISH-qos-0 SUBSCRIBE-qos-0 , and selected all abnormal message use cases for testing at the same time.

In the test for abnormal messages, the number and degree of abnormal behaviors of various types of data can also be selected for testing. For example, text, binary data, numbers, characters, etc.; at the same time, you can configure the byte limit of overflow exception to test with malformed packets with value overflow.

execute the test

After selecting the type of test case and the number of abnormal data, the test can be started. The test took about 6 minutes, of which about 98% passed the test, and about 1% (2779 use cases) had unknown results.

Analyze and save results

Let's first select one of the abnormal packets with unknown results for analysis.

It can be seen that in order to evaluate the robustness of the tested object, Defensics tries to use abnormal data that does not conform to the protocol specification for testing during fuzzing. For example, the red highlighted part in the figure, this part of the two-byte data is in the SUBSCRIBE message of the MQTT v3.1.1 protocol, indicating the length of the UTF-8 string of the subscription topic. It means that the following data of length 0x009A (154 bytes) is the UTF-8 string of the subscription topic filter. But the actual length of the topic filter is 18 bytes, and the value should be 0x0012 , which does not match the actual length.

According to the mandatory specification of the protocol, in this case, the server MUST close the network connection [MQTT-4.8.0-1] that transmits this protocol violation control message.

However, it is not specified whether a message indicating the cause of the error must be returned. Therefore, EMQ X only performs internal error processing, and directly discards abnormal packets. Neither does any postback operation to the sender, so Defensics marks this result as unknown.

But according to the agreement, this result still meets the requirements.

After statistics, among the 2779 test cases with unknown results, the different types of errors are shown in the following table:

error type	quantity
Packet is too large (overflow)	190
Fixed header error (fixed-header)	44
Fixed header flags errors (flags)	34
The value of the remaining length of the packet is abnormal (remaining-length)	1167
Packet-identifer exception	626
Topic filter structure error (topic-filters)	304
Abnormal topic filter length value (topic-filter-length)	414

In the face of these abnormal messages, EMQ X directly discarded them, and did not send back an indication message about the error information.

For some of the error types, since the information at the error point is more critical, trying to guess the boundary of the critical information may even lead to larger and more unacceptable errors.
For example, the string length indicator value parsed above is wrong. Guessing about topic filters and their lengths may result in incorrect topic filters, causing clients to get unexpected topic subscriptions. It is even possible that the topic filter is of the correct length, and the topic filter's value was lost in transit and corrupted.

Such logical errors are more difficult to find and troubleshoot during system operation, and the consequences are more difficult to accept. Therefore, it is a better choice to directly discard abnormal packets at this time.

As for other types of errors, because the error points are too obvious, the more likely causes are data loss during transmission, or exceptions caused by data stream boundary errors. Therefore, it is more inclined to think that these data are not MQTT packets, and they are also discarded, so as not to spend extra resources to deal with these exceptions.

Summarize

This article roughly summarizes the fuzzing process of EMQ X using the Defensics fuzzer produced by Synopsys and the supporting MQTT v3.1 protocol test suite, and selects some use cases for testing and result cause analysis.

It can be seen that EMQ X is very complete in the implementation of the protocol. Even if a large number of error messages are used for testing, it will not cause EMQ X to lose the ability to provide services, which can ensure the security of the protocol and provide security for the stable operation of the actual project. .

EMQ is committed to providing highly available and highly reliable MQTT message servers and other data infrastructure software for the IoT field. Last year, we also reached a cooperation with Synopsys, which will be fully responsible for the safety and quality risk management of EMQ's various production lines throughout the product life cycle. We hope that users can build more stable and reliable IoT platforms and applications through EMQ products.

EMQ X open source project also welcomes your participation at any time. You are welcome to submit PR or Issue to us through GitHub: https://github.com/emqx/emqx .