Illustrated Kafka, the drawing is great!

Kafka is a mainstream message flow system, and there are many concepts in it. Let's sort out the core concept through the diagram below, so that we can have a clear understanding in our minds.

basis

Kafka ( What is Kafka and what scenarios is it mainly used in? ) is a stream processing system that allows back-end services to communicate with each other easily. It is a commonly used component in microservice architecture.

Producer consumer

The producer service Producer Kafka , and the consumer service Consumer monitors Kafka to receive messages.

A service can be a producer and a consumer at the same time.

Topics

Topic is the target address of the message sent by the producer, and the listening target of the consumer.

A service can monitor and send multiple topics.

Kafka has a concept of [consumer-group (consumer group)].

This is a set of services, acting as a consumer.

If the consumer group receives the message, Kafka will route a message to a certain service in the group.

This helps to balance the load of messages and facilitates the expansion of consumers.

Topic acts as a queue of messages.

First, a message is sent.

Then, the message is recorded and stored in this queue, and it is not allowed to be modified.

Next, the message will be sent to consumers of this topic.

However, this message will not be deleted and will remain in the queue.

Continue to send messages.

As before, this message will be sent to the consumer, not allowed to be changed, and stay in the queue forever.

(How long the message can stay in the queue, you can modify the configuration of Kafka)

Partitions

In the description of Topic above, Topic is regarded as a queue. In fact, a Topic is composed of multiple queues and is called [Partition].

This can facilitate the expansion of Topic.

When the producer sends a message, the message will be routed to a Partition in this topic.

Consumers monitor all partitions.

When the producer sends a message, it is Topic-oriented by default. The Topic decides which Partition to put in, and the polling strategy is used by default.

You can also configure Topic so that all messages of the same type are in the same Partition.

For example, to process user messages, all messages of a certain user can be in one Partition.

For example, user 1 sends 3 messages: A, B, and C. By default, these 3 messages are in different Partitions (such as P1, P2, P3).

After configuration, you can ensure that all messages of user 1 are sent to the same partition (such as P1).

What is the use of this function?

This is to provide the orderliness of the message.

The order of messages in different Partitions cannot be guaranteed. Only the messages in one Partition are orderly.

Architecture

Kafka is a cluster architecture, and ZooKeeper is an important component (but ZooKeeper was discarded after version 2.8: Goodbye ZooKeeper! ).

Topic and Partition owned by ZooKeeper manager.

Topic and Partition are stored in Node physical nodes, and ZooKeeper is responsible for maintaining these Nodes.

For example, there are 2 Topics, each with 2 Partitions.

This is a logical form, but the actual storage in the Kafka cluster may look like this:

There are 3 copies of Partition #1 of Topic A, which are distributed on each Node.

This can increase Kafka's reliability and system flexibility.

In 3 Partition #1, ZooKeeper will designate a Leader to be responsible for receiving messages from producers.

The other two Partition #1 will serve as Followers, and the messages received by Leader will be copied to Followers.

In this way, each Partition contains the full amount of message data.

Even if a Node fails, there is no need to worry about the damage of the message.

The distribution of all Partitions of Topic A and Topic B may look like this:

Thanks for reading, I hope it helps you😃

Translated from: https://timothystepro.medium.com/visualizing-kafka-20bc384803e7
Transfer from: https://blog.csdn.net/duysh/article/details/116355977