EMQX Newsletter 2022-05｜v5.0 releases 2 rc versions, cloud service adds external authentication and data integration

In May, EMQX 5.0.0-rc.3 & rc.4 versions were released one after another. This version added support for jq syntax to the rule engine, greatly simplified the default configuration file, and made further improvements to the Dashboard menu bar. Adjust and optimize. The official release of EMQX 5.0 is getting closer. At the same time, the next maintenance versions of v4.3 and v4.4 have also entered the testing phase and will be released soon.

In terms of cloud services, EMQX Cloud has added AWS private network connections and more regional deployment support to overseas stations this month. At the same time, there are more options for external authentication and data integration services.

EMQX

EMQX 5.0.0-rc.3 & rc.4 version overview

In the previous article, we introduced some of the new features of EMQX 5.0 from a Dashboard perspective through EMQX 5.0.0-rc.2. At present, EMQX 5.0.0-rc.3 and rc.4 have also been released, and the functions are more complete and stable.

Rules engine supports jq syntax

Now, we can use jq syntax in the rules engine's SQL to handle more complex JSON data. We can index and slice JSON arrays, filter data by conditions, perform fuzzy queries on keys, use pipeline commands to combine multiple filters, use built-in functions to calculate the average of JSON arrays, and even customize functions for more complex calculations. jq gives the rule engine SQL more powerful data processing capabilities, we can visit the jq Manual to learn more about its usage. The following is a simple example of the use of jq in the rules engine SQL:

 SELECT
  jq('.', payload) as example
FROM
  "t/#"

The default configuration file is reduced

In version 5.0.0-rc.4, we shortened the default configuration file to less than 100 lines, and additional configuration file examples will help users understand the use of all configuration items. This will effectively help users free from most of the configuration items that are not often used, and only focus on the necessary configuration items, further reducing the difficulty of their use.

Other optimizations

• Dashboard menu bar adjustment and optimization
• Dashboard supports relative paths or custom access paths, which is convenient for reverse proxies such as NGINX
• Optimized the use of the rate limit function in the Dashboard, now you can configure multiple Buckets for the Limiter, and select the Limiter in the Listener
• Optimized the configuration of TLS certificates in Dashboard
• Optimized the display logic of resource status in Dashboard
• Fixed inaccurate calculation of memory footprint
• Fixed an issue where manually leaving the cluster would render the node unavailable
• Fixed multiple bugs with gateway listeners
• Fixed multiple bugs related to health checks

To download a trial or learn more about optimizations and bug fixes, please visit EMQX 5.0.0-rc.3 & EMQX 5.0.0-rc.4 .

4.3 & 4.4 Maintenance Release Upgrade Preview

Maintenance releases such as v4.3.15 are about to be released, which are expected to bring more than 20 bug fixes and several improvements, including EMQX support to start under paths that contain spaces, and improved EMQX startup under Windows to avoid being unable to see when startup fails Error messages, increased version checking to avoid hot upgrades across major versions, etc. In addition, there are several enhancements:

The rule engine supports Alibaba Cloud TableStore

Included version: Enterprise Edition v4.4.4

Alibaba Cloud Tablestore is a cloud-based structured data storage product launched by Alibaba Cloud. It provides IoT storage, wide table engine, multi-index and other capabilities to meet the needs of time series data, message data, and metadata scenarios. need.

EMQX and the Tablestore team have done special optimization work for the efficient connection of products, realizing the efficient storage integration of diverse IoT data. Through the integrated structure of Tablestore, it provides IoTstore with a one-stop solution that is large-scale, free of operation and maintenance, low-cost and easy to expand, effectively solving technical difficulties such as technical selection and classified storage of database products.

At present, the EMQX rule engine Tablestore integration has passed the 10W TPS throughput performance test and can be officially put into production.

Rules engine SQL supports more functions

Included versions: open source v4.3.15, open source v4.4.4, enterprise v4.3.10, enterprise v4.4.4

1. time conversion function

   Now, we can use the format_date function in the rule engine SQL to convert the incoming integer timestamp or automatically exchange the current timestamp to a time string in the specified format, or use date_to_unix_ts The function converts a time string in the specified format to an integer timestamp. Example:

    SELECT
     format_date('nanosecond', '+08:00', '%y-%m-%d %H:%M:%S%Z') as date1
     format_date('nanosecond', '+08:00', '%y-%m-%d %H:%M:%S%Z', timestamp) as date2
   FROM
     "t/#"

2. Floating-point output precision control function

   Added float2str/2 function to support specifying the output precision of floating-point numbers.

Add Basic and JWT authentication support to Pulsar

Included versions: Enterprise Edition v4.3.10, Enterprise Edition v4.4.4

We've added Basic and JWT authentication support to Pulsar, which works with TLS for better security.

Support JWT for authentication

Included versions: open source v4.3.15, open source v4.4.4, enterprise v4.3.10, enterprise v4.4.4

Now, the JWT used for client connection authentication can continue to be used for authentication for more flexible rights management capabilities. This function requires that the acl declaration that meets the format requirements be carried in the JWT. For details, please refer to the official website documentation.

Authentication and authentication using built-in database (Mnesia) as data source supports multi-condition filtering and fuzzy query

Included versions: open source v4.3.15, open source v4.4.4, enterprise v4.3.10, enterprise v4.4.4

Similar to the query client, the authentication function that uses the built-in database as a data source now also provides these query options, such as _like_clientid , _like_username , topic etc. , of which _like_clientid and _like_username support fuzzy queries using substrings.

Support to configure the log time format to be compatible with the time format in older versions

Added log.formatter.text.date.format configuration item, which supports the configuration as rfc3339 or a format string, that is, YYYY-MM-DDTHH:mm:ss.SSSZZ in the form of a format compatible with the time in logs of older versions such as 4.2.

Community dynamics

The EMQX team participated in the 2022 Code BEAM Europe conference in Stockholm, Sweden on May 19-20.

EMQ Software Engineer William Yang delivered an inspiring keynote titled "QUICER: BEAM's Next-Generation Transport Protocol Library" introducing the world's first MQTT over QUIC implementation powered by EMQ and a new open source build for BEAM NIF library QUICER ( GitHub - emqx/quic: QUIC protocol for Erlang and Elixir ).

Dmitrii Fedoseev, another data engineer at EMQ, discussed how to use SNABBKAFFE to test distributed consistency fault tolerance, and introduced how EMQ successfully applied trace-based methods to real-world applications running in production.

EMQX Cloud

Overseas stations support the creation of AWS PrivateLink

The private network connection (PrivateLink) can realize the establishment of a secure and stable private connection between the private network (VPC) where EMQX Cloud is deployed and the services on the public cloud, simplify the network architecture, realize private network access services, and avoid accessing services through the public network. potential security risks.

The EMQX Cloud overseas station added support for the connection of PrivateLink deployed on AWS this month, connecting the VPC where the deployment is located and the VPC where the AWS (overseas) resources are located, which is equivalent to realizing communication within the same network.

Overseas stations support more regional deployments

In the regional selection of EMQX Cloud overseas station creation and deployment, AWS has added Hong Kong region, Google Cloud Platform has added Taiwan region, and the cost is the same as other overseas regions. Enterprise users involved in overseas business will have more deployment options.

External authentication supports Redis

Added support for users to use data stored in their own Redis service for client authentication and access control. Currently EMQX Cloud supports HTTP, MySQL, PostgreSQL, and Redis for authentication and access control. Learn more about external certification .

Data integration adds TDengine and Lindorm

Data integration adds new resource connections: TDengine and Lindorm. TDengine is an open source, high-performance, distributed time series database that supports SQL. Currently, both domestic and overseas stations have been supported. Lindorm is a database of Alibaba Cloud that supports various data models such as wide tables, time series, objects, files, queues, and spaces. Currently, it only supports domestic stations.

EMQX Kubernetes Operator

In May, EMQX Operator, a tool for automating deployment, configuration, and management of EMQX clusters on Kubernetes, released version 1.1.8, providing the following new features:

Feature update

1. Improve the resource operation log into the events event record
2. Convert some logs in EMQX Operator into events
3. EMQX Operator resource checklist implementation
4. Better EMQX Custom Resource Status

perfect optimization

1. Fixed the mirror tag problem, support tags based on private repositories
2. Fixed restart listener exception after updating .spec.listener.certificate

Test verification

1. EMQX Operator realizes the stress test of 1 million connections and 500,000 TPS of EMQX based on cloud environment

coming soon

EMQX Operator 1.2 and v1beta3 APIVersion are under development, v1beta3 APIVersion will bring a more reasonable .spec structure, and version 1.2 will introduce a more complete event log and cluster status description.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/emqx-newsletter-202205