• Blog Series Overview: In the "Let’s Take a Look at…​!" blog series, the author explores data and streaming space projects. They mention a specific Kafka-related topic and encourage readers to suggest others.
• Apache Kafka Basics: Kafka is a distributed durable event log. Producers write to topics divided into partitions among brokers. Consumers in groups divide partitions. This design defines key characteristics like maximum consumer parallelism and ordered message processing.
• Kafka vs Queues: Kafka is great for high-volume data streaming but not for queuing semantics. Queues are better for job queueing where items are processed independently. Some efforts were made to support queue-like use cases in Kafka, but actual queue implementations like ActiveMQ Artemis and RabbitMQ were better.
• KIP-932: Queues for Kafka: With Kafka 4.0, Early Access of KIP-932 is added. It introduces share groups, where multiple consumers can process messages on the same partition, enabling high consumer parallelism.
• Share Group Details: Share groups use a new consumer rebalance protocol. Messages can be individually acknowledged, and there's a share-partition leader managing in-flight messages. Messages have different states like Archived, Available, In-flight, Acknowledged, and Rejected.
• Java Usage: In Java, to use share groups, Kafka and client libraries need to be built from source. Share groups have implicit and explicit acknowledgement modes. New configuration options are needed to enable the feature.
• Retry Behavior and State Management: Retries in share groups have some nuances. Messages in Acquired status are returned indefinitely until acknowledged. The share-group state is made durable through a special Kafka topic.
• Summary and Outlook: KIP-932 adds queue-like semantics to Kafka, useful for job queuing. It's an early access feature with limitations like lack of DLQ support. Performance characteristics need further exploration. Voting for Kafka 4.0.0. RC1 has started, and readers can provide feedback on the mailing list.