领英宣布 Northguard 和 Xinfra：超越 Kafka 进行日志存储和发布/订阅的扩展

发布于 6 月 25 日

LinkedIn Announces New Systems: Today, LinkedIn announced Northguard (a scalable log storage system replacing Kafka) and Xinfra (a virtualized Pub/Sub layer).
Challenges with Kafka at LinkedIn's Scale: Kafka became difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).
Northguard's Architecture and Features:
- Consists of shared data & metadata, decentralized coordination, and minimal global state.
- Organizes logs into records, segments, ranges, and topics.
- Enables balanced load, high availability, and seamless scaling.
- Data model provides more flexible scaling than traditional indexed partitions.
- Metadata model uses sharded, Raft-backed replicated state machines.
Northguard's Optimization: Optimized protocols for performance and durability, with strong durability guarantees.
Xinfra for Kafka Migration: A virtualized Pub/Sub layer for seamless, zero-downtime migrations.
Segment Storage in Northguard: Default "fps-store" implementation is optimized for durability and latency.
Rigorous Testing at Scale: Undergoes deterministic simulation to ensure reliability.
Migration Success and Adoption: Successfully migrated thousands of topics and over 90% of applications are using Xinfra.
Open-Source Plans: LinkedIn is focused on finalizing and iterating on the tools before exploring open-sourcing.

阅读 242