RocketMQ -- deletion of expired files

When the messages are continuously sent to the Broker, and the Broker keeps storing the messages in the Commitlog file, ConsumeQueue file, and IndexFile file, the disk will always run out at the moment.

Since the Commitlog file and ConsumeQueue file are written sequentially, that is to say, all write operations are either written to the last file, or a new file is created when it is full, so the previous files will not be updated, then You can delete files from a long time ago. These older files are expired files.

When deleting expired files, it should be noted that the Broker will not judge whether the messages here have been consumed, and the deletion is done anyway.

When the Broker starts, it starts a scheduled task, delays it for 60s, and then starts to determine the release to delete expired files every 10s.

The timing of deletion expiration includes the following three types:

1. The default is 4 am. This is also easier to understand. At this time, there are fewer people using it, and the impact of deletion on the system is minimized.
2. Insufficient disk space. When the disk space is insufficient, it is necessary to delete expired files to provide more space to receive messages.
3. Manual triggering refers to human intervention to delete.

The deleted files are expired files, so which files are expired?

The first is the retention time. The default is 72 hours, which is 3 days. Data that exceeds 3 days needs to be deleted.

If the file is referenced by other threads, it will not be deleted at this time, record the timestamp of the first deletion, exit the task, and after 120s, the reference of the file will be decremented by 1000, and then forced to delete.

If the number of files currently deleted has exceeded the maximum number of batches that can be deleted, quit this task.

During the deletion process, multiple files may be deleted, and there is a time interval between each file. For example, after the first file is deleted, it needs to wait 100ms before deleting the second file.