HStreamDB Newsletter 2022-06｜New cluster mechanism, visual monitoring, Python client release…

This month, the HStreamDB team focused on the development of v0.9, and has completed the switching of the HServer decentralized cluster, the development of HStream IO Embedded Runtime and CDC Source Connector, and brought the new Grafana monitoring integration and officially released The first Python client available. In addition, the integration of HStreamDB and EMQX was completed in collaboration with the EMQX team.

HServer adopts new clustering mechanism

At present, we have initially completed switching the HServer cluster mechanism from the ZooKeeper-based centralized solution to the SWIM[1]-based decentralized solution. The main purpose is to support larger clusters and better scalability, while reducing external requirements. system dependencies. In the future, we will continue to test and improve the new cluster mechanism, which will be officially released in v0.9.

HStream IO supports CDC Source

HStream IO is an internal data integration framework that HStreamDB v0.9 will release soon, including source connectors, sink connectors, IO Runtime and other components. It can realize the interconnection between HStreamDB and various external systems, thereby helping to promote data in the entire enterprise data stack. efficient circulation and real-time value release.

This month, we completed the development of Embedded IO Runtime and CDC Source Connector for various databases, including MySQL, PostgreSQL, SQL Server, etc., which can efficiently synchronize the data of these databases to HStreamDB in real time.

Added Grafana monitoring integration

In order to facilitate user operation, maintenance and management of HStreamDB clusters, we have added monitoring support based on Prometheus and Grafana, which are also the mainstream monitoring solutions in the industry. The monitoring data inside HStreamDB will be stored to Prometheus through the Exporter, and then displayed visually through the Grafana panel. The current effect is shown in the following figure.

For more information about monitoring, please refer to the documentation https://hstream.io/docs/en/latest/monitoring/grafana.html

Python client officially released

This month, we officially released HStreamDB's Python client hstreamdb-py https://github.com/hstreamdb/hstreamdb-pyy v0.1.0, supports HStreamDB v0.8, and currently has data batch writing, subscription consumption and resource management and other core functions, you are welcome to use and give feedback.

For related installation instructions, please refer to https://pypi.org/project/hstreamdb/ , and for more documentation, please refer to https://hstreamdb.github.io/hstreamdb-py/

Support integration with EMQX

EMQX is the world's leading open source MQTT message server developed by EMQ, which is widely used in the field of IoT. This month, through cooperation with the EMQX R&D team, we completed the efficient integration of EMQX and HStreamDB, which will help users achieve one-stop IoT device connection, data access, persistent storage and real-time analysis. For details, please refer to https://www.emqx.com/zh/blog/integration-practice-of-emqx-and-hstreamdb

[1]: Das, A., Gupta, I. and Motivala, A., 2002, June. Swim: Scalable weakly-consistent infection-style process group membership protocol. In Proceedings International Conference on Dependable Systems and Networks (pp. 303 -312).IEEE.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://hstream.io/zh/blog/hstreamdb-newsletter-202206