This week in Databend #21

Databend aimed to be an open source elastic and reliable cloud warehouse, it offers blazing fast query and combines elasticity, simplicity, low cost of the cloud, built to make the Data Cloud easy.

This week, the migration of the Databend website to Docusaurus was completed; it is now hosted on a service sponsored by Vercel. Please enjoy it.

Big changes

Below is a list of some major changes that we don't want you to miss.

Features

• add show create database statement by @Junnplus (#3573)
• support engine options and refactor github engine by @Junnplus (#3542)
• impl Visitor for SQL AST by @jackwener (#2880)
• support drop stage by @GrapeBaBa (#3550)
• add mTLS support for http query server by @ZhiHanZ (#3507)

• add more strings functions

  • LOWER & LCASE, UPPER & UCASE by @Xuanwo (#3521, #3558)
  • Format by @eastfisher (#3466, merging in #3489 by @sundy-li)
  • SOUNDEX by @bestgopher (#3446)

• add uuid functions

  • generateUUIDv4, zeroUUID, isemptyUUID, isnotemptyUUID by @lianghanzhen (#3582)

• add more hash functions

  • BLAKE3, XXHASH32, XXHASH64 by @PsiACE (#3527, #3556)

Improvement

• optimize table statement by @dantengsky. (#3511)
• add more privileges by @flaneur2020. (#3567)
• move validity masking out from function to transform by @junli1026 (#3543)
• refine load configs from toml file by @liubaohai (#3545)
• refactor string hash functions using String2String template function by @mshauneu (#3536)
• use clap 3.0.0-rc.5 to unify the cli arg parsing by @koushiro (#3499)
• add query cache by @Veeupup (#3433)
• add created_on to TableMeta by @GanZiheng (#3319)
• support historical data truncation by @dantengsky. (#3463)

Performance Improvement

• improve constant filter by @sundy-li (#3537)
• add filters pushdown for read datasource by @zhyass (#3513)

Bug fixes

• check create table of immutable catalog by @Junnplus (#3554)
• fix ClickHouse progress hang in some queries by @BohuTANG. (#3544)
• make pprof work again by @PsiACE (#3523)
• fix intdiv return type by @zhyass (#3497)

Tips

Let's learn a weekly tip from Databend.

What's happening in Databend SQL layer

With #2983 ready for review, Databend's SQL layer will see the arrival of a new planner framework.

In fact, we have a number of plans for refactoring the SQL layer, which may even include a rewritten parser.

• Read the planner RFC

  RFC: New SQL planner framework

  In summary, this RFC will:

  Introduce a new framework to support planning JOIN and correlated subquery
  Introduce a rule system that allows developer to write transformation rules easily

• Explore related issues

  • [[Tracking] Refactor SQL layer](https://github.com/datafusela...)
  • [[Tracking] Refactor SQL Parser](https://github.com/datafusela...)
  • [[Tracking] Refactor PlanBuilder](https://github.com/datafusela...)

Changlogs

You can check the changelogs of Databend nightly to learn about our latest developments.

• v0.6.11-nightly
• v0.6.10-nightly
• v0.6.9-nightly
• v0.6.8-nightly
• v0.6.7-nightly
• v0.6.6-nightly
• v0.6.5-nightly

Ecosystem/Upstream

From open source, for open source. Our team is also committed to contributing to the Rust ecosystem and upstream dependencies.

• succinct-rs - succinct data types: A compressed prefix tree in Rust. by @drmingdrmer

Meet Us

Please join the DatafuseLabs Community if you are interested in Databend.

We are looking forward to seeing you try our code. We have a strong team behind you to ensure a smooth experience in trying our code for your projects.
If you are a hacker passionate about database internals, feel free to play with our code.

You can submit issues for any problems you find. We also highly appreciate any of your pull requests.

• Databend Website
• Twitter
• GitHub Discussions
• Slack Channel