New IoT Data Integration: Flow Visual Orchestration & Bidirectional Data Bridging

Introduction: New Data Integration Capabilities

Providing high-performance real-time data processing and integration for IoT platforms and applications has always been one of the most important capabilities of EMQX. The newly released EMQX 5.0 has been deeply refactored and optimized for data integration related functions, in order to help users use it more easily and flexibly.

EMQX 5.0 integrates Webhook, data storage/bridging plug-ins, and uses a unified interface to manage the north-south data flow. On the basis of the original data collection and reporting scenarios for rule processing, users can also process the south direction from the cloud to the device through rules. information.

At the same time, EMQX 5.0 also provides data integration visualization capabilities (Flows). Through the Dashboard page, users can clearly see how IoT data is processed through rules and how data flows to external data services or devices.

Subsequent versions will also support drag-and-drop arrangement of rules and data bridging (Flow Editor) on the Dashboard to easily connect IoT hardware data flows together through a visual interface.

This article will show readers in detail the value and application of this important capability of EMQX, as well as what upgrades and optimizations have been made in EMQX 5.0.

What is data integration

EMQX data integration is the combination of rule engine and bridging function in version 4.x. It is a data processing and distribution component based on the publish-subscribe model. Through simple and visualized configuration, it is possible to connect message streams , device events , message middleware such as Kafka, RabbitMQ, and various SQL / NoSQL / time series databases and other data system integration.

EMQX's real-time, concise and efficient data integration solution is mainly realized through the two functions of rules and data bridging .

Rules are used to process message and event data, and implement operations such as data format conversion, message encoding and decoding, and business logic processing through SQL syntax combined with built-in or user-extended custom functions. For more information on rules, see Rules .

Data bridging is used to connect data systems and realize bidirectional, high-performance data movement between EMQX and external systems. Through data bridging, users can send messages from EMQX to external data systems in real time, or pull data from external data systems and send them to a topic in EMQX. For more information about data bridging, see Data Bridging .

Why you need data integration

The MQTT protocol is designed for message transmission from hardware devices to servers. To implement a complete IoT application, it is necessary to connect the device with the business system to realize cloud storage of reported data and issuance of business instructions.

In traditional IoT application development, developers need to write corresponding codes to achieve this purpose, and subsequent business changes need to upgrade the entire application. The device side and the business platform side are often operated by independent teams. How to realize the decoupling of the device side and the cloud business platform and efficiently complete the business connection is the key to whether the project can be quickly delivered and upgraded.

EMQX data integration is designed to solve this problem. By providing flexible and low-code configuration capabilities, it helps users get started easily and quickly realizes various application integrations and business innovations.

After years of practice in IoT application scenarios in various industries, EMQX data integration capabilities have become an essential key function for building overall applications, and continue to create value for users.

Bidirectional data flow: both data reporting and message delivery support rule processing

The data between devices and business systems in IoT applications is usually bidirectional. There are both data collection and reporting scenarios and cloud message delivery requirements. The combination of the two realizes a complete business process.

In previous versions, EMQX rules were triggered and executed through device messages and events. This means that rules can only be used in data collection and reporting scenarios, while cloud-to-device message delivery scenarios cannot directly use the data processing capabilities of rules. Users need to use the transit scheme to send data to a topic in EMQX first, and then process it through rules.

EMQX 5.0 optimizes this: provides bidirectional data bridging capability - in addition to bridging device data to external systems, it can also bridge data from external data systems such as another MQTT service, Kafka to EMQX, and after rule processing sent to the specified device.

The data bridging of bidirectional data flow in the cloud delivery scenario decouples the connection between the business system and EMQX, enabling continuous large-scale message delivery and real-time processing, providing more possibilities for IoT business development.

Flow Editor: Process data flow through visual orchestration rules

In previous versions, data integration of EMQX was achieved by configuring SQL + rule actions. This mode has the following benefits:

• SQL syntax is widely used in the database field, and users with technical background can quickly get started and master writing complex rules in a short time;
• SQL rules are more readable than coding, and can be created and changed at runtime, which is suitable for rapidly changing businesses.

The data integration and data processing methods around SQL allow users to quickly develop and realize the required business, but it also brings some problems:

• Technicians need to be familiar with the EMQX rule SQL syntax before they can start writing business, while for non-technical personnel, SQL is not intuitive enough, and the threshold for getting started is high;
• Some complex scenarios cannot be implemented using SQL, such as distributing data to different data bridges based on conditions in the same rule, which will result in users having to create multiple similar rules;
• It is not possible to add an event to reuse the same rule without modifying the SQL;
• When there are many rules, it cannot provide a clear data processing and integration process, and it is difficult to maintain and manage.

In EMQX 5.0, we prioritized solving the problem of difficult maintenance and management of data integration under multiple rules through the visual viewing capability (Flows). Through the Dashboard page, users can clearly see the data integration topology of how IoT data is processed through rules and flowed to external data systems or distributed to devices after being processed by rules from external data systems, and monitor any rules or data in this link. The status of the bridge node.

In subsequent versions, we will build a data flow editor with visual orchestration capability -- Flow Editor on EMQX. It is based on the browser and used with the underlying execution engine, which will allow users to freely arrange rules and data bridges on the EMQX Dashboard by dragging and dropping. Easily connect IoT hardware data streams through a visual interface, enabling more flexible and faster IoT business development and delivery.

Data Integration Upgrade Guide: Migrating from v4 to v5

EMQX 5.0 rule SQL is fully compatible with the syntax of version 4.x, but in design, the actions originally under the rule are split into built-in actions (message republishing, console output) and data bridges (Webhook, MQTT Bridge), in order to achieve Multiplexing of rules and data bridging.

In addition to the architectural design, EMQX also integrates the functions related to data integration in the old version. Before the 4.x version of the rules engine functionality, users could use plugins for simple data integration. For compatibility reasons, we did not remove these plugins after the rule engine was launched, which brought a certain fragmentation of functions.

EMQX 5.0 officially removes related plug-ins in order to provide users with clearer and more complete data integration capabilities and reduce the difficulty of learning and selection.

The following is the comparison of the function changes of EMQX open source version 4.x to 5.0:

Epilogue

In the process of building IoT platforms and applications in various industries, it is an indispensable link and challenge to accurately and real-time process massive data according to requirements and connect with various upper-level business system applications. As an IoT MQTT message server that can "run anywhere, connect unlimitedly, and integrate at will", EMQX will help users easily apply data processing problems and realize business innovation through powerful, easy-to-use and constantly improving data integration capabilities.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/iot-data-integration