HStreamDB v0.9 released: Partition model extension to support integration with external systems

The latest version of HStreamDB v0.9 is now officially released!

v0.9 mainly has the following highlights and updates:

• The stream partition model is extended, allowing users to directly access data at the specified location on the partition;
• Added HStream IO, the internal data integration framework of HStreamDB;
• The cluster switched to a SWIM-based member discovery and fault detection mechanism;
• A new stream processing engine;
• Updated Java and Go clients and added a new Python client.

Stream Partition Model Extension

v0.9 extends the previous partitioning model, allowing users to directly manipulate and access partitions within a stream, thus enabling fine-grained control over data distribution and partition scaling in a stream. HStreamDB adopts a key-range-based partitioning mechanism. All partitions under the stream jointly divide the entire key space, and each partition belongs to a continuous subspace (key range). If the hash value of the partitionKey carried by the record falls within a certain subspace, the record will be stored in the corresponding partition.

Specifically, the v0.9 partition model adds the following capabilities:

• Configure the initial number of partitions when creating a stream
• Through partitionKey distribute the written records to the corresponding stream partitions
• Read the data of the specified partition directly from any location
• View the partitions contained in the stream and the key range corresponding to each partition

In a future release, we will support dynamic scaling of streams through partition splitting and merging.

HStream IO released

HStream IO is an internal data integration framework included in v0.9, including source connectors, sink connectors, IO runtime and other components. It can realize the interconnection between HStreamDB and various external systems, and promote the efficient flow of data in the entire enterprise data stack. and real-time value release.

v0.9 provides the following connectors to support incremental synchronization between multiple databases.

Source connectors:

• source-mysql
• source-postgresql
• source-sqlserver

Sink connectors:

• sink-mysql
• sink-postgresql

Users can create and manage IO tasks through corresponding SQL commands. For details, please refer to the documentation to learn about the functions and usage of HStream IO.

new stream processing engine

v0.9 reimplemented the stream processing engine based on iterative and differential computing principles, significantly improving throughput and reducing latency. In addition, the new engine supports multiple join statements, sub-queries and more general materialized views.

This feature is still in the development stage and is an experimental feature. Users can try it out by referring to the SQL guide .

Gossip-based HServer cluster

v0.9 reconstructs the cluster implementation of HServer. The new implementation mainly adopts the cluster mechanism of gossip style and the fault detection mechanism based on SWIM, which replaces the implementation based on ZooKeeper in the previous version. The new implementation will improve cluster scalability and reduce reliance on external systems.

Advertised Listeners

Deployment and use in production can involve complex network setups. For example, if a server cluster is hosted internally, it needs an externally visible IP address for customers to connect to the cluster, especially when using docker or cloud hosting, which makes the environment more complicated.

To ensure that clients from different network environments can interact with the cluster, HStreamDB 0.9 supports configuring advertised listerners. After configuring the advertised listerners, the server can return the corresponding address for different clients according to the port that the client sends the request.

Unified HStream CLI

To make the CLI more uniform and simple, we have migrated the old HStream SQL Shell and some other node management features to the new HStream CLI. HStream CLI currently supports functions such as launching an interactive SQL Shell, sending cluster bootstrap requests, and checking server node status. Users can learn the specific usage by referring to the CLI documentation .

Grafana-based monitoring

v0.9 adds support for monitoring HStreamDB clusters through Prometheus and Grafana. Metrics inside HStreamDB will be stored in Prometheus through the exporter and finally displayed on the Grafana panel. For specific deployment and usage procedures, please refer to the documentation .

Support for deployment on K8s with Helm

v0.9 provides the Helm Chart of HStreamDB. Now, HStreamDB cluster can be quickly deployed on K8s through Helm. For more detailed usage steps, please refer to the documentation .

Client version upgrades and improvements

Java client v0.9.0, Go client v0.2.0, Python client v0.2.0 have all been released, providing support for HStreamDB 0.9. For details, see:

• Java Client v0.9.0: https://github.com/hstreamdb/hstreamdb-java/releases/tag/v0.9.0
• Go client v0.2.0: https://github.com/hstreamdb/hstreamdb-go/releases/tag/v0.2.0
• Python client v0.2.0: https://github.com/hstreamdb/hstreamdb-py/releases/tag/v0.2.0

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://hstream.io/zh/blog/hstreamdb-v-0-9-release-notes