介绍 twom 数据库格式

发布于 8 月 3 日

New Database Format: Developed twom based on twoskip for Cyrus IMAP server. It's called "twom" with "two" for dual level0 pointers and "m" for mmap and MVCC. All code in merged pull request.
- xxHash: Faster than CRC32 for small data. Chose xxHash for its license and performance.
- MVCC Repacks: Allows repacking without holding a lock, reduces lock time from 30 minutes to 1 minute and enables writes during repack.
- Starvation-free Locking: Uses two-offset locking strategy to avoid writer starvation. Had to patch during testing on statuscache file.
- MMAP for Reading and Writing: Uses mmap for both reads and writes to reduce syscalls and simplify code. Still uses msync for reliable commits.
- Pre-emptive Allocation: Extends file by 25% and fills empty space to reduce munmap and mmap calls.
- Just straight POSIX: Written standalone with fcntl locking and mmap manipulation for portability and optimizations.
Database Operations: Can use 'cyr_dbtool' to interact with twom database. Set, delete, and show records, and use prefix iterators and dump output.
Repacking: Removes tombstones and reorganizes the file. New writes update back pointers.
Raw File: Not much overhead per record. About 50 bytes of overhead including trailing nulls, headers, checksums, and pointers.
Production Use: Since February 12, 2025, Fastmail email servers have been using twom backend in all places that used to use twoskip. Switch done in three phases over two days.

阅读 10