• Key takeaways: Bufstream is a Kafka-compatible queue 8x cheaper than Apache Kafka. It's great with Protobuf, enforces data quality and governance without opt-in client config, plans to add granular RBAC and Iceberg support, and runs in AWS or GCP without sending data to Buf.
• What is Bufstream: A fully self-hosted replacement for Apache Kafka that writes to S3-compatible storage, compatible with Kafka protocol including EOS and transactions, enforces data quality with Protovalidate and includes Iceberg metadata.
• Why did we do this: Customers wanted to use Protobuf in streaming data pipelines after adopting gRPC and Connect but Kafka didn't support their workloads well. Kafka is expensive and lacks data quality and governance features. Bufstream was built to address these challenges.
• Quality and governance as platform capabilities: Treats data quality as critical, can reject messages not matching schema and enforce protovalidate constraints. Bakes in granular data access controls and plans to evolve to full-fledged field-level RBAC.
• Optimized for cloud economics: Writes directly to S3 to reduce costs by eliminating expensive disks and inter-zone networking. Benchmarks show it's 8x cheaper than Kafka with median latency of 260 ms and p99 latency of 500 ms. Can opt to store some topics as Iceberg tables to save more.
• Air-gapped deployment and straightforward pricing: Deploys with simple Helm chart in AWS or GCP, self-contained, charges $0.002 per GiB of write traffic without additional costs. Interactive demo, benchmarks, and Slack available for getting started or scheduling demos.