• The Database and its Open-Source Dependencies: Built ScopeDB in 4 months with 3 devs using Rust. It's a shared-disk arch db in cloud managing petabytes of observability data. Used 50,000 lines of Rust code with 100 direct & 647 total deps. Leveraged Apache OpenDAL, SQLx, SeaQuery, Jiff, and ordered-float. Updated dep count: 623 (2025-01-15), 394 (2025-02-17). Spawned common libs & made them open-source, like a message queue demo system.
• Involve and Contribute Back to the Upstreams: When using open-source projects, encounter bugs etc. Motivated to contribute. Examples include implementing new layer for OpenDAL, developing benchmark tool for OpenDAL, extending types in Jiff and ordered-float, contributing to Apache Arrow, sharing code even if specific to ScopeDB, maintaining open-source projects, and contributing to documentation. Sometimes contribute in non-code ways like giving feedback. Many times, it's about communication and collaboration.
• Inside Out: The Database’s Open-Source Components: During ScopeDB dev, wrote code for common requirements as no existing open-source software met them. Examples are Fastrace (tracing lib), Logforth (logging lib), Fastimer (for scheduled tasks), and Cronexpr (for cron expressions). ScopeDB's SDK is also open-source.
• An Open-Source Twin and the Commercial Open-Source Paradigm: Developed Morax as an open-source twin with similar arch to ScopeDB for sharing engineering experience. It's used to test container reuse in ScopeDB. Followed the idea from The Cathedral & the Bazaar that it's reasonable to keep commercial software private while being involved in open-source.
• Future Works: ScopeDB is in early stages, facing challenges in improving performance and supporting more features. Working on accelerating async scheduling and supporting variant data. Also aiming to provide an online service for free trial with real cloud resources. Interested people can email mailto:tison@scopedb.io. Will keep sharing engineering experience.