RocketMQ -- message sending storage process

I have already talked about how the message is sent to the broker. After the broker receives the message, how does it handle it?

In order to ensure that the message is not lost, RocketMQ will persist the message, that is, it will write the message to the commitlog log, which is under the store. The size of each log is 1G, so the commitlog will have multiple disk files, and each file name is the physical offset of the message.

When the broker receives a message, it will first make some judgments. For example, only the master can write, the length of the topic is limited to 256 characters, and the length of the message attribute is limited to 65536 characters.

After the verification is passed, start writing data. Since there may be concurrency problems, you need to apply for a lock every time you write.

After applying for the lock, start writing data to the commitlog. If the log file is first written, there is no file in the commitlog at this time, so a log file with a size of 1G and a name of 00000000000000000000 will be created. If multiple log files already exist, take the last log file directly, because a new log file will be created after the log file is written, and the last log file is the one that needs to be written.

After getting the log file, we need to append the message to this file. At this time, we need to know the write pointer of the current file. If it is a newly created file, the write pointer is 0.

This write pointer cannot be larger than the size of the file. If it exceeds, it means that the file is full, and data cannot be written to it.

Another thing that needs to be judged is whether the current message is enough for this file, that is, the length of the write pointer + the message (mq will reserve other space), whether it will exceed the size of the file, if not, write this file, and if so, a new file is created to write to.

After writing, the above write pointer is updated, that is, the last write pointer is the current write pointer + the size of the message. Finally release the lock to allow other threads to write.

In fact, at this step, the message is not written to the disk, but only appended to the memory, and the process of refreshing the screen will be mentioned later.