Cloudflare 选择 PostgreSQL 扩展而非专门的 OLAP 来实现每秒 10 万行的分析

发布于 7 月 31 日

Cloudflare's engineering team chose TimescaleDB over ClickHouse for adding analytics and reporting capabilities to its internal platform.
- The simplicity of storing analytical data with configuration data and the performance of a specialized OLAP system was highlighted.
Cloudflare built Digital Experience Monitoring (DEX), an internal observability platform, with a configuration plane and an analytics plane.
Since 2017, Cloudflare had been using ClickHouse but chose not to use it for this project due to issues like write amplification and resource contention with its default MergeTree table engine when dealing with millions of device log events in small batches.
PostgreSQL was used initially for both configuration data and analytical logs but couldn't handle the growing data volume and performance degradation.
To improve performance, the team explored precomputing aggregates (downsampling) and turned to TimescaleDB for its support of columnstore and sparse indexes. TimescaleDB is an open-source time-series database built as a PostgreSQL extension, optimizing storage and querying for time-stamped data.
Benchmarking showed TimescaleDB had performance improvements of 5x to 35x over PostgreSQL depending on the query type and time range.
The community mainly questioned the decision not to use ClickHouse, but TimescaleDB provided a good balance between simplicity and performance for Cloudflare.
After implementing the DEX project, TimescaleDB has been adopted in other Cloudflare projects like Zero Trust Analytics & Reporting for generating analytics and long-term reports.

阅读 26