EMQ X and HStreamDB integration practice: data storage through the rule engine

Faced with massive device connections and the large-scale data streams generated in real-time in the Internet of Things era, EMQ provides a modern data infrastructure from the edge to the cloud, facilitating the unified "connection, movement, processing, and analysis" of cloud-side IoT data.

Nowadays, the cloud-native distributed messaging middleware EMQ X that can "run anywhere, unlimited connections, and arbitrarily integrate" has solved the challenge of massive connections, while the streaming database HStreamDB is trying to solve the storage, processing and real-time analysis of massive IoT data.

As the first cloud-native streaming database designed specifically for streaming data, HStreamDB is committed to efficient large-scale data streaming storage and management. The combination of EMQ X and HStreamDB will make the one-stop management of massive data access, storage, real-time processing and analysis no longer difficult.

The recently released v0.6 adds a new data write Rest API, which can use any language to write data to HStreamDB through the Rest API, which is convenient for open source users to carry out secondary development around HStreamDB. We have also combined this function with the Webhook function of the EMQ X open source version to realize the rapid integration of EMQ X and HStreamDB.

This article will introduce in detail the specific operation of using HStreamDB to persistently store the access data of EMQ X.

Note: This article describes mirroring based on EMQ X 4.3 and hstreamdb/hstream:v0.6.1.

Start EMQ X and HStreamDB

First, we need a running EMQ X, how to install, deploy and start please refer to: EMQ X document .

At the same time, we need a running HStreamDB. For more detailed instructions on how to install, deploy and start, please refer to: HStreamDB Docs .

For users who are not familiar with HStreamDB, you can quickly start a single-machine HStreamDB cluster through docker-compose.

Start HStreamDB

First download the docker-compose.yaml file directly through the link

Create a file to store database data:

mkdir /data/store

Start HStreamDB in the background:

docker-compose -f quick-start.yaml up -d

pass through:

docker-compose -f quick-start.yaml logs hstream-http-server

You will see the following log:

Server is configured with:
     gRPCServerHost: hserver
     gRPCServerPort: 6570
     httpServerPort: 6580
 Setting gRPC connection
 Setting HTTP server
 Server started on port 6580 

Create the required Stream through HStreamDB CLI

Stream is an object used to store streaming data in HStreamDB, which can be seen as a collection of some data.

Start HStreamDB CLI

Start a command line interface of HStreamDB with docker:

docker run -it --rm --name some-hstream-cli --network host hstreamdb/hstream hstream-client --port 6570 --client-id 1

You will enter the following interface:

      __  _________________  _________    __  ___
     / / / / ___/_  __/ __ \/ ____/   |  /  |/  /
    / /_/ /\__ \ / / / /_/ / __/ / /| | / /|_/ /
   / __  /___/ // / / _  _/ /___/ ___ |/ /  / /
  /_/ /_//____//_/ /_/ |_/_____/_/  |_/_/  /_/

>

Create HStreamDB Stream to save the bridged data:

> CREATE STREAM emqx_rule_engine_output ;
emqx_rule_engine_output

Of course, we can also get the created Stream SHOW

> SHOW STREAMS;
emqx_rule_engine_output

Configure EMQ X

Then, we open the Dashboard of EMQ X, click Rule Engine, and enter the Resource interface.

We can first create a WebHook resource, as shown below:

In Request URL fill a column hstream-http-server listening address, <host>:6580/streams/emqx_rule_engine_output:publish , then click test connection test the link.

Next, let's create the required rule engine rules:

SELECT 
  payload,                 -- 在 HStreamDB 的 http 协议中，我们需要一个 payload 项
  str(payload) as payload, -- HStreamDB 要求 payload 是一个 JSON String
  0 as flag                -- HStreamDB 中 flag 为 0 表示 payload 是一个JSON String
FROM 
  "#"                      -- 这个符号会匹配所有的 topic

We need to add an Action Handler, select Action as Data to Web Server :

Set Method to POST , and Header to content-type application/json .

At this time, we have completed the most basic bridge settings, let us test it through websocket and hstreamdb-cli.

Observe whether the persistent storage of data is complete through HStreamDB CLI

First, we create a Query in the HStreamDB CLI that we just started:

> SELECT * FROM emqx_rule_engine_output EMIT CHANGES;

In HStreamDB, each Stream represents a series of dynamically changing data streams, so a Query does not simply read data, but will continue to read and output the data written in the Stream. In CLI, the starting point for reading and outputting data is the moment when the Query is successfully created. Currently, what we can observe is that there is no output in CLI.

At this point, we can write data to EMQ X through the WebSocket of EMQ X DashBoard or other MQTT clients (such as the cross-platform MQTT 5.0 desktop client tool-MQTT X).

The following uses WebSocket as an example, we can first connect to the EMQ X cluster we started:

Then send data to the specified topic:

If everything is normal, we can see the data we sent to EMQ X in the HStreamDB CLI in real time.

> SELECT * FROM emqx_rule_engine_output EMIT CHANGES;
{"location":{"lng":116.296011,"lat":40.005091},"speed":32.12,"tachometer":9001.0,"ts":1563268202,"direction":198.33212,"id":"NXP-058659730253-963945118132721-22","dynamical":8.93}

So far, we have completed the persistent storage of the data accessed by EMQ X in HStreamDB.

By integrating EMQ X with HStreamDB, we can not only achieve persistent storage of the data sent to EMQ X, but also perform real-time processing and analysis of these data to obtain further data insights. With the continuous improvement of the two products, we believe that in the future, the efficient combination of EMQ X + HStreamDB will play an important role in the analysis and processing scenarios of real-time streaming data in the IoT field, and become an important part of the process of data conversion and monetization. The value creation of enterprise data assets provides impetus.