Kafka -- message sending storage process

The directory structure of kafka is different from RocketMQ -- message sending and storage process , but some ideas are still the same.

The first is under kafka-logs, where the topic+partition folder is placed. For example, if the topic is test, there are two partitions (assuming both are on this Broker), then there are two files, test-0, test- 1.
Under each file, there are many log files and two corresponding index files. Each log file is also called a segment.

The size of the segment file is also 1G, and each file name is the physical offset of the message.

When the broker receives a message, it first makes some judgments, such as permission verification and whether the parameters of acks are legal.

After the verification is passed, traverse each partition in the message to process the data. For example, the partition 0 corresponding to test, according to the above directory structure diagram, we know that the copy of test-0 will be found.

After it is found, it will apply for a read lock and obtain the leader's partition, because only the leader can write data.

The read lock is added above, and another lock is added when the segment file is written.

After applying for the lock, start to select the segment file. If the log file is first written, there is no segment file at this time, so a log file with a size of 1G and a name of 00000000000000000000 will be created. If multiple log files already exist, take the last log file directly, because a new log file will be created after the log file is written, and the last log file is the one that needs to be written.

If the size of the written message and the size of the currently selected segment file add up to more than 1G, a new segment file will be created.

If the index or timeIndex corresponding to the segment is full, a new segment file will also be created.

After the segment file is obtained, the message is written into the memory.

Every time a 4096-byte message is written, two index files are written. Of course, when writing, it is still necessary to lock first.

So our index files are not continuous.

After writing the memory, it will determine whether to flash the disk. The default is not to flash the disk. The flashing operation is handed over to the operating system for regular execution, so there will be data loss.