Use the streaming computing engine eKuiper to process Protocol Buffers data

Protocol Buffers (Protobuf) is a language-neutral, platform-neutral, extensible mechanism for serializing structured data in a binary transfer format. Compared with conventional data transfer formats (such as JSON or XML), Protobuf is more efficient and faster and saves transmission bandwidth, so it has been widely used.

In the cloud-side collaboration architecture, it is often necessary to send data to the cloud, and at the same time, it is necessary to receive the data sent from the cloud to perform cloud-side collaborative computing. The total amount of data transmitted by large-scale cloud-side collaborative computing is huge. In the case of limited and expensive public network bandwidth resources, it is particularly important to adopt a more compact data transmission format.

LF Edge eKuiper is an ultra-light IoT edge data streaming analysis engine suitable for deployment at the edge with limited resources. It can connect external systems with various communication protocols such as MQTT and HTTP through source and sink. eKuiper supports the configuration of the codec format of source/sink transmission data, and currently supports JSON, ProtoBuf and Binary formats.

This article will take the Protobuf format as an example to explain how to set the codec format in eKuiper, read and parse the data in this format through the source, and use the format to encode and write in the sink, so as to achieve efficient cloud-edge collaborative data transmission and alleviate The problem of bandwidth shortage in cloud-side transmission.

This tutorial uses eKuiper Manager to create and manage rules, please refer to the UI tutorial . You can also use the REST API or run command line tools on the side where eKuiper runs to accomplish the same rule management operations.

Environmental preparation

Before starting the hands-on operation, you need to prepare the following environment:

• MQTT server is used for data transfer. This tutorial uses the MQTT server located at tcp://broker.emqx.io:1883 , broker.emqx.io is a public MQTT server provided by EMQX Cloud . If you run eKuiper locally, you need to change etc/mqtt_source.yaml and change the configuration item server to "tcp://broker.emqx.io:1883"; if you use docker to start, you should set the environment variable MQTT_SOURCE DEFAULT SERVER="tcp: //broker.emqx.io:1883".
• In order to observe the running results conveniently, we need to install an MQTT client , such as MQTT X.

Schema Registry

Compared with the schemaless JSON format, Protobuf needs to define the data structure in advance, that is, the schema. In the proto file, the definition of multiple messages and other entities can be included, but in the configuration of the codec format, only the definition of the message can be used. In this tutorial, we use the following schema for the definition of data structures. The file defines a message structure called Book, which contains a title of type string and a price of type integer. The transmitted data will encode and decode the binary data of the book data according to this structure.

 message Book {
  required string title = 1; 
  required int32 price = 2;
}

1. registration mode. In the Admin Console, open Configuration -> Schemas and click Create Schema.

   

2. In the pattern creation window, fill in as shown below. Among them, select the pattern type protobuf ; the pattern name can enter a custom and unique name as the identification id of the pattern in the subsequent rule creation; the pattern content can be filled in by file or text content. When selecting file, you need to fill in the url where the file is located; the mode used in this tutorial is relatively simple, so you can select content, and then fill in the text of the proto file in the content box.

   

3. Click Submit. The newly created schema should be visible in the schema list. Subsequent operations can be modified or deleted using the buttons in the action bar.

   

So far, we have registered a schema named schema1 , which defines the type Book , which can be used in the source and sink of the rule. Users can also continue to perform more schema registration and management work in this interface.

Read Protobuf data

In this section, we take MQTT source as an example to introduce how to access and parse the data transmitted based on Protobuf encoding, so that rules can be calculated in eKuiper. It should be noted that in Source, the encoding format and transmission protocol are not bound. Any source type such as MQTT, httpPull, etc. can be matched with different encoding formats, such as ProtoBuf and JSON.

Suppose we have an MQTT topic demo , for the purpose of saving transmission bandwidth, the data transmitted in it is Protobuf encoded binary data. Next, we'll configure the eKuiper data source to access the topic's data and process it.

1. Create a data flow: In the management console, select Source Management -> Flow Management, and click Create Flow.

2. Configure the data stream and its format: the stream name can be set to a user-defined unique name; the data source is the MQTT topic to be monitored; the stream type is set to mqtt; the stream format selection protobuf ; One-step registration schema1 ; the mode message is set to the message Book defined in the proto file.该配置表示数据流protoDemo MQTT 主题protoDemo ，收到schema1中的Book的格式进行protobuf decoding. Click Submit and the newly created flow should be listed in the flow list.

   

3. Create a rule: Select a rule and click New Rule to enter the rule creation interface. As shown in the figure below, click on the upper right corner to enter the text mode, enter the custom rule ID, rule name, and enter the JSON text of the rule in the text content. This rule indicates that the content in the selection stream protoDemo is sent to the MQTT topic result/protobuf .

    {
      "id": "ruleDecode",
      "sql": "SELECT * FROM protoDemo",
      "actions": [{
        "mqtt": {
          "server": "tcp://broker.emqx.io:1883",
          "topic": "result/protobuf",
          "sendSingle": true
        }
      }]
   }

   

4. Send data and view the result: We will use MQTTX to send the Protobuf encoded binary data to the topic protoDemo , and observe whether the received result is the correct data after decoding.

   1. Open MQTT X and connect to the cloud tcp://broker.emqx.io:1883 .
   2. Subscribe to the topic of the above-mentioned rule to send the result result/protobuf , so that it is easy to observe the result.

   3. In the message sending pane, set the subject to protoDemo and the Payload format to Hex , and send the binary data encoded according to the Book format in schema1, for example 0a1073747265616d696e672073797374656d107b .

      

   4. Make sure that the receive window receives the correct JSON data, as shown in the image below.

      

So far, we have finished reading and decoding Protobuf data and processing the output with simple rules. Users create various rules just like normal JSON format data. If the expected result is not obtained, you can view the rule status on the rule list page of the management console to ensure that the indicators of rule data input and output meet expectations.

Write Protobuf data

In this section, we will show the usage of reading JSON format data for processing and then sending it to the cloud MQTT broker in Protobuf format. In the scenario of IoT edge-cloud collaboration, this usage can save the bandwidth overhead of edge-cloud transmission. The eKuiper deployed at the edge can access the local MQTT broker without consuming bandwidth, and can be accessed by processing faster JSON format. After rule operation, when the calculation result needs to be sent to the cloud MQTT broker, Protobuf encoding can be used to save bandwidth.

1. Create a data flow: In the management console, select Source Management -> Flow Management, and click Create Flow. As shown in the figure below, create a stream that connects to the demo topic and JSON format data.

   

2. Create rules and send them to the cloud using Protobuf format.

   1. Click New Rule, enter a custom Rule ID and name, enter SQL SELECT * FROM demo .

   2. Click the New button to the right of the action to configure the MQTT action. Among them, the MQTT server address is configured as the cloud broker address, and the MQTT topic is result/protobufOut ; the data sent by bar is configured as true to ensure that the received data is configured in a matching format; the stream format is configured as protobuf , the schema name is schema1 registered in the first section, and the schema message is Book . This rule will read JSON data, and then encode it into binary data according to the format of Book and send it to the result/protobufOut topic. Click Submit to complete the action configuration.

      

      

   3. Each rule can have multiple actions, and the encoding format used by each action is independent. The user can continue to configure the remaining actions. After all configurations are complete, click Submit to complete the creation of the rule.

3. Send data and view the results, the process is similar to the previous section. This time we will send JSON data to the demo topic, and expect to see the protobuf-encoded binary data in the subscribed result/protobufOut topic. As shown in the figure below, pay attention to the configuration of the data format to avoid displaying garbled characters.

   

Summarize

This tutorial describes how to read and write Protobuf data in eKuiper. The ProtoBuf format is one of the external connection formats of eKuiper. Various formats can be combined arbitrarily. After accessing the system, the internal format representation is used. First, the user needs to define the Protobuf mode; then, the Protobuf format can be configured in the creation of the stream and the creation of the action, and the defined mode can be selected to encode and decode the data.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/using-ekuiper-to-process-protocol-buffers-data