• Rust for Various Applications: Rust is great for terminal apps, fantasy consoles, and web3. It often gets mentioned when C++ is discussed. ClickHouse is in C++, but efforts are being made to add Rust.
• First Steps in Rust Integration: The first step was to find a small non-critical component for Rust integration. An undergrad student was tasked with integrating the BLAKE3 hash function. In a few months, the first Rust code was in ClickHouse using "corrosion" integration and a dedicated Rust directory.
• External Contributions: The first external pull request with Rust was to improve the clickhouse-client with interactive history navigation. However, the Rust library had a bug. Later, PRQL was added, allowing for querying in a pipelined form. It has support for switchable dialects in ClickHouse.
• Delta Lake Integration: Delta Lake is an open data lake format with no good C++ implementations. Databricks created a Rust library, which was used to replace the old C++ code. This was the most complex Rust integration.
• Challenges in Rust Integration: Building Rust during the ClickHouse build had hermeticity and reproducibility issues. Rust code often led to segmentation faults due to interop with C++. It lacks exceptions, and fuzzers found PRQL query-related panics. Sanitizers are needed for the C++ application with Rust, and MSan is only in the "nightly" Rust toolchain. Thread Sanitizer had to be disabled for Rust. Cross-compilation with Rust and C++ together is not easy. Delta-kernel-rs depends on "reqwest" with OpenSSL linking issues. Rust can cause symbol name bloat and requires disabling debug info. There are convention differences between C++ and Rust code. The build infrastructure needs to support Rust caching and profiling. Rust has a modular ecosystem but can lead to dependency blow-up.
• Conclusion: Rust is going well in ClickHouse. Those interested in writing in Rust are welcome. ClickHouse Cloud offers a $300 credit trial and volume-based discounts.