eKuiper Newsletter 2022-05｜protobuf codec support, visual drag-and-drop writing rules

This May, eKuiper is in the early stage of new function development of version 1.6.0, and product function development mainly includes protobuf codec support. We are also carrying out prototype verification in response to the user's multiple requirements for writing data processing rules by dragging and dropping. Bug fixes for v1.5.1 are also in progress and are expected to be released in early June.

In addition, as an edge streaming data processing software, eKuiper also participated in the standard evaluation of the "Edge Streaming Data Processing Technology Capability Requirements" initiated by the China Academy of Information and Communications Technology. After passing, eKuiper will become the first product certified by this standard.

protobuf codec support

Currently, the default value of the codec configuration property format of source and sink is JSON. Users can select the codec capability of protobuf through the newly added protobuf option. Compared with JSON, the amount of data encoded by protobuf is smaller, which is beneficial to save the bandwidth transmitted between cloud edges. This function has been developed but not yet released, and users can try it out through the v1.6.0 branch of the edge source code.

Compared with the schema-less JSON format, protobuf needs to define the proto file as the encoding and decoding schema. Before using the protobuf format, the user needs to register the schema first, and specify the schema selected for encoding and decoding through the newly added schemaId attribute.

Schema management

Registered using the REST API, the schema content can be provided via file path or text.

 // POST /schemas
{
  "id":"fileName",
  "file":"http://myhost/files/abc.proto"
}

Or configure by text content:

 // POST /schemas
{
  "id":"schemaName",
  "content":"message Person {
    required string name = 1;
    required int32 id = 2;
    optional string email = 3;
  }"
}

Either way, the content will be stored in etc/schemas/protobuf.

Using protobuf in Sink

• format, which specifies the encoding to use
• schemaId, used to specify the schema used

 {
  "mqtt":{
    "server":"tcp://127.0.0.1:1883",
    "topic": "result",
    "format":"protobuf",
    "schemaId":"schemaName.Person"    // protobuf的 ID分为两部分，前面为文件名，后面为message名
  }
}

Using protobuf in Source

FORMAT, supports protobuf

SCHEMA_ID , used to specify the schema used

 CREATE STREAM demo() WITH (TYPE="NEURON", FORMAT="protobuf", SCHEMA_ID="schemaName.Person")

Visual drag and drop editing capabilities

Previously, eKuiper only supported the writing of data processing rules in SQL format for streaming data, and there were certain thresholds, which made it inconvenient for business personnel to directly participate in the writing of rules. In order to further reduce the use threshold, eKuiper is ready to support dragging and dropping data processing units in a visual way and perform simple editing and configuration, and finally connect multiple data processing units to automatically form data processing rules, so that more people can directly use eKuiper for business processing. The scheme is currently in the prototyping stage.

coming soon

Next month we will refine the visual drag-and-drop editing and merge it into the master branch. In addition, eKuiper will optimize the caching mechanism after sink error, realize offline storage of memory + disk, and retransmit sequentially after error recovery, so as to support stronger resilience in the event of network disconnection, and cache data for a longer time.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/ekuiper-newsletter-202205