OpenHarmony littlefs file system storage structure and IO performance optimization analysis

introduction

With the development of science and technology and the advancement of network technology, computer storage space is becoming more and more tight, and the functional requirements of storage space for file systems are increasing. Large-scale data growth has brought new challenges to file storage and unstructured data storage.
For many IoT devices, having a small and resilient file system is critical, and the littlefs file system came into being. The littlefs file system was developed by Christopher Haster in 2017, follows the Apache 2.0 protocol, and is used in ARM's IoT device Mbed operating system. The littlefs file system enables the embedded system to have the basic power-down recovery and wear leveling functions of the file system when the ROM and RAM resources are limited.
littlefs is a minimalist embedded file system suitable for norflash. The file system structure and operation mechanism it adopts makes the storage structure of the file system more compact and consumes less RAM during operation. Its design strategy adopts the strategy of "use time for space", which is completely opposite to the traditional "use space for time". Although it greatly compresses the storage space of the file system, it also increases the consumption of RAM when running. There is a reduction in IO performance during random read and write.
Currently, the OpenAtom OpenHarmony (hereinafter referred to as "OpenHarmony") liteos_m kernel uses littlefs as the default file system. This paper focuses on the storage structure of the littlefs file system, and analyzes the root cause of the random read and write IO performance bottleneck of the littlefs file system based on the analysis of the read and write process, and then proposes some optimization strategies to improve the littlefs random read and write IO performance.

littlefs file system structure

The file system storage structure information basically starts with SuperBlock, then finds the root node of the file system, and then gradually expands into a file system tree structure according to the root node. littlefs is also similar, with SuperBlock and the root directory as the starting point to build a tree-shaped storage structure. The difference is that the root ("/") of the littlefs is attached directly after the SuperBlock and shares a metadata pair with it. The directory or file in littlefs starts from the root node and builds a tree structure similar to other file systems.
The tree storage structure of the littlefs file system is as follows:

 图1 littlefs文件系统树形存储结构示意图

As shown in Figure 1, the structure for storing the metadata of the littlefs file system is a metadata pair, that is, two blocks that rotate with each other and are in each other's tables. The metadata pair that stores the SuperBlock is fixedly stored in block 0 and block 1, and the root directory of the file system is attached to the end of the SuperBlock to share the metadata pair with the SuperBlock. The storage of metadata is stored in the metadata pair in the format of tags. According to the type of metadata, tags are divided into standard files, directories, user data, metadata pair tail pointers and other types. With the help of these different types of tag information, littlefs organizes the littlefs file system into a compact tree-shaped storage structure. For example, a tail type tag can use multiple metadata pairs to store a relatively large directory structure, and use a tail type tag to connect these metadata pairs into a one-way linked list. The tag of the directory type directly points to the metadata pair of the directory. For example, the tag of type "tag: dir_data" points to the metadata pair of the directory "/data", and the metadata pair can contain subdirectories or files (Inline type or outline type).
littlefs directory storage structure
The reference of the littlefs directory is a Tag of type dir in the metadata pair of its parent directory, and its content occupies one or more metadata pairs. The metadata pair of a directory can contain either a Tag referenced by a subdirectory, an Inline-type Tag belonging to a file in the directory, or a CTZ-type Tag pointer pointing to the CTZ jump table of the file. Finally, littlefs forms a tree-shaped storage structure of the file system through layer-by-layer directory or file indexes.
littlefs file storage method
The littlefs file system is a minimalist file system that uses minimal storage overhead and supports both small files (Bytes level) and large files (MB level). For files less than one-eighth the length of a block, the Inline type is used. The file is stored in the way of Outline (CTZ Skip-list).
1.2.1 Inline file storage method
The inline file storage method, as shown in Figure 2, stores the file content and file name together in the metadata pair of its parent directory. One Tag represents its name and one Tag represents its content.

 图2 littlefs Inline文件存储结构

1.2.2 outline file storage method
Outline file storage method, as shown in Figure 3, in the metadata pair (metadata pair) of the parent directory of the file, one Tag represents the file name, and the other Tag is the CTZ type, which points to the linked list header storing the file content.

 图3 littlefs Outline文件存储结构

The special features of the CTZ skip-list linked list are:
(1) The head of the CTZ jump table points to the end of the linked list;
(2) The Block in the CTZ jump table contains more than one jump pointer.
If a conventional linked list is used, and the previous data block of the stored file contains a pointer to the next data block, then when appending or modifying the content of the file, it is necessary to copy all the contents from the starting block of the stored file to the target block to the new block, and update it. A pointer to the next block of data. If the reverse linked list method is used, when appending or modifying the content of the file, it is only necessary to copy all the contents from the target block of the storage file to the block at the end of the linked list into the new block, and then update the corresponding data in the latter data block. Pointer to the previous data block, so that the amount of modification can be reduced for file append mode. In addition, in order to speed up the indexing, the method of skip list is adopted, and the block contains more than one skip pointer. The rule is: if the index value N of a data block in the CTZ skip-list linked list is a number that can be divisible by 2^X, Then he has a pointer to N - 2^X, and the number of pointers is ctz(N)+1. As shown in Table 1, for block 2, it contains 2 pointers, which point to block 0 and block 1 respectively, and the same rules are used for other blocks.

Table 1 The skip-list linked list calculation sample table of littlefs block

littlefs file read and write process

The above chapters analyze the structure of the littlefs file system. Next, we start to discuss the internal operation mechanism of littlefs. Taking the read and write process as an example, analyze the IO performance bottleneck of random read and write of littlefs.
What needs to be understood in advance is that littlefs files have only one cache, which is used as a write cache when writing and a read cache when reading.
littlefs file reading process The following Figure 4 is a flow chart of littlefs reading a file. At the beginning of the reading process, it is detected whether there is a previous write operation to the file, that is, whether the file cache is used as a write cache. If so, the data in the cache is forced to be flushed to the memory, and according to the file type and access location, either directly read from the metadata pair where the file is located, or read from the block in the CTZ jump table that stores the file content, and then Data is copied to the user buffer and prefetched from memory to fill the file buffer. The specific process is as follows:

 图4 littlefs文件系统读过程流程图

littlefs file writing process The following Figure 5 is a flow chart of littlefs writing a file. At the beginning of the writing process, it is detected whether there is a previous read operation to the file, that is, whether the file cache is used as a read cache. If so, clear the data in the cache. If it is an APPEND type of write operation, the write position is directly reduced to the end of the file. If the write position exceeds the file length, indicating that there is a hole between the end of the file and the write position, the hole in the file is filled with 0. Corresponding to the Inline type file, if it is estimated that the file length exceeds the threshold after writing, the file will be converted to the Outline type. For files of Outline type, if you modify the content of the file, you need to apply for a new block, copy all the content before the access position in the target block to the new block, and write the user data in the buffer to the buffer or flush it to the memory.
Note: The commit of the Inline file is not updated immediately after the write, or the CTZ jump table of the Outline file is updated. These operations are delayed when the file is closed or the buffer is used as a read cache again when the file is flushed.

 图5 littlefs文件系统写过程流程图

Random read and write IO performance bottleneck analysis of littlefs files
The littlefs file has only one buffer, which is multiplexed for reading and writing. According to the operation mechanism of littlefs, if the file is read first and then written, it is only necessary to directly clear the data in the buffer, and then apply for a new block to copy the direct data of the access position in the target block to the new block, and then write the data to the new block. . If you write first and then read, you need to refresh the data to the memory and update the CTZ jump table of the file at the same time. In this process, it not only involves flushing data to the memory, but also involves allocating new blocks to replace all blocks after the target block, thereby updating the CTZ jump table, and there are many time-consuming actions of erasing blocks. In the process of random reading and writing, frequent switching of reading and writing occurs, and actions such as applying for new blocks, erasing new blocks (very time-consuming), and data movement occur frequently, which seriously affects the IO performance.

littlefs read and write IO performance optimization strategy

According to the description in "2.3 Random Read/Write IO Performance Bottleneck Analysis of littlefs Files", the main reason that affects the random read/write IO performance of littlefs files is that the file has only one cache and is multiplexed by read and write, resulting in frequent read/write switching. When a file refresh occurs, a new block is applied for, then a time-consuming block erase is performed, and the block content in the block on the CTZ jump table is moved to the new block, and then the CTZ jump table is updated, which seriously affects the performance of random read and write IO.
Therefore, when the RAM space allows, the strategy of "using space for time" can be considered, and the number of file caches can be appropriately increased, so that a file has multiple buffers, and these buffers correspond to the size of a block. Refresh one block at a time under certain conditions to avoid excessive data movement. Also, littlefs' strategy is "time for space", but having one buffer per file is an obvious waste of space. Because only a certain number of files will be read or written in a certain period of time, you can consider establishing a cache pool with a certain number of files to share the cache among files.

 图6 littlefs优化策略

The optimized strategy is shown in Figure 6. The littlefs file cache pool is a doubly linked list fc_pool. The cache pool is extended as the number of open files increases until the maximum limit set by the user; the cache pool gradually increases as the files are closed. reduce.
Each cache is mounted on the doubly linked list of the fc_pool cache pool. When the cache is written or read, the cache is moved to the beginning of the linked list, then the cache at the end of the linked list of the cache pool is the cache to be aged and can be preferentially selected for recycling.
When applying for cache, the free cache is selected from the end of the buffer pool list first; if there is no free cache, the read cache will be preempted; if the cache pool has neither free cache nor read cache, if the length of the cache pool does not reach the optimization limit, Then create a new cache, and then add the new buffer to the head of the linked list; if the cache pool has neither free cache nor read cache, and the length of the cache pool has reached the user limit, then a write cache needs to be preempted from the end of the linked list and forced to occupy it. Cached files are flushed to complete preemption.
When the file is closed or refreshed, the cache is actively released to the buffer pool and mounted at the end of the doubly linked list.
Using the above strategy to optimize the file cache can reduce the memory block erasing action performed by updating the file content to a certain extent, thereby increasing the IO performance of random read and write, and indirectly prolonging the life of NorFlash.

Summarize

Through the explanation of this article, I believe that everyone has a more comprehensive understanding of the littlefs file system. In general, littlefs is a minimalist file system that implements basic data caching, power failure recovery, wear leveling and other functions of the file system. The storage structure adopts the optimization strategy of "using space for time" to improve the IO performance of reading and writing. Learning to effectively use the file system can often achieve multiplier effects with half the effort. I hope developers can effectively apply the knowledge they have learned to future development work, thereby improving the efficiency of development work.