How to use JuiceFS performance tools for file system analysis and tuning

JuiceFS is a high-performance POSIX file system designed for cloud native environments, released under the AGPL v3.0 open source agreement. As a distributed file system on the cloud, any data stored in JuiceFS will be divided into data blocks and stored in object storage (such as Amazon S3) according to certain rules, and the corresponding metadata will be persisted in an independent database. This structure determines that the storage space of JuiceFS can be elastically scaled according to the amount of data, reliably store large-scale data, and support sharing and mounting between multiple hosts, achieving cross-cloud and cross-regional data sharing and migration.

Since the release of v0.13, JuiceFS has added a number of functions related to performance monitoring and analysis. To a certain extent, the development team hopes that JuiceFS can provide excellent performance in large-scale distributed computing scenarios as well as a wide range of Cover more daily use scenarios.

In this article, we start with stand-alone applications to see whether applications that rely on stand-alone file systems can also be used on JuiceFS, and analyze their access characteristics for targeted tuning.

test environment

The next test will be performed on the same Amazon cloud server, and the configuration is as follows:

• server configuration : Amazon c5d.xlarge: 4 vCPUs, 8 GiB memory, 10 Gigabit network, 100 GB SSD
• JuiceFS : Use the local self-built Redis as the metadata engine, and the object storage uses S3 in the same area as the server.
• EXT4 : create directly on the local SSD
• data sample : Use Redis source code as a test sample

Test project one: Git

Commonly used git series commands include clone, status, add, diff, etc. Among them, clone is similar to the compilation operation, and involves writing a large number of small files. Here we mainly test the status command.

Clone the code to the local EXT4 and JuiceFS respectively, and then execute the git status command time-consuming situation as follows:

• Ext4：0m0.005s
• JuiceFS：0m0.091s

It can be seen that there is an order of magnitude difference in time. From the sample of the test environment alone, such performance differences are minimal, and users are almost imperceptible. But if a larger code warehouse is used, the performance gap between the two will gradually become apparent. For example, suppose that the two time-consuming is multiplied by 100 times, the local file system needs about 0.5s, which is still within an acceptable range; but JuiceFS will need about 9.1s, and users can already feel a significant delay. To figure out the main time-consuming points, we can use the profile tool provided by the JuiceFS client:

$ juicefs profile /mnt/jfs

In the analysis process, a more practical way is to record the log first, and then use playback mode :

$ cat /mnt/jfs/.accesslog > a.log
# 另一个会话：git status
# Ctrl-C 结束 cat
$ juicefs profile --interval 0 a.log

The results are as follows:

Here you can clearly see that there are a large number of lookup requests. We can extend the cache time of metadata in the kernel by adjusting the mount parameters of FUSE. For example, use the following parameters to remount the file system:

$ juicefs mount -d --entry-cache 300 localhost /mnt/jfs

Then we use the profile tool to analyze, the results are as follows:

As you can see, lookup requests are reduced a lot, but they are all transformed into getattr requests, so attribute caching is also required:

$ juicefs mount -d --entry-cache 300 --attr-cache 300 localhost /mnt/jfs

At this time, the test found that the time consumption of the status command dropped to 0m0.034s, and the profile tool results are as follows:

It can be seen that the most time-consuming lookup at the beginning has been reduced a lot, and the readdir request has become a new bottleneck. We can also try to set --dir-entry-cache , but the improvement may not be obvious.

Test project two: Make

The compilation time of large projects often needs to be measured in hours, so compile-time performance is more important. Still taking the Redis project as an example, the time-consuming test compilation is:

• Ext4：0m29.348s
• JuiceFS：2m47.335s

We tried to increase the metadata cache parameters, and the overall time was reduced by about 10s. The analysis results through the profile tool are as follows:

Obviously the data read and write here is the key to performance, we can use the stats tool for further analysis:

$ juicefs stats /mnt/jfs

One of the results is as follows:

From the above figure, it can be seen that the ops of fuse is close to meta, but the average latency is much larger than meta, so it can be judged that the main bottleneck is on the object storage side. It is not difficult to imagine that a large number of temporary files are generated in the early stage of compilation, and these files will be read in the later stages of compilation. It is difficult to directly meet the requirements with the performance of normal object storage. Fortunately, JuiceFS provides a data writeback mode, which can first establish a write cache on the local disk, which is suitable for compilation scenarios:

$ juicefs mount -d --entry-cache 300 --attr-cache 300 --writeback localhost /mnt/jfs

At this point, the total compilation time has dropped to 0m38.308s, which is very close to the local disk. The monitoring results of the stats tool in the later stage are as follows:

It can be seen that basically all read requests are hit in the blockcache, and there is no need to access the object storage; the ops statistics on the fuse and meta sides have also been greatly improved, which is in line with expectations.

Summarize

This article uses Git warehouse management and Make compilation tasks that the local file system is better at as the entry point, evaluates the performance of these tasks on JuiceFS storage, and uses the profile and stats tools that come with JuiceFS to analyze, and adjust the file system mounting parameters Do targeted optimization.

Undoubtedly, there are natural characteristic differences between local file systems and distributed file systems such as JuiceFS, and the application scenarios for the two are also completely different. This article chooses two special application scenarios, just to introduce how to do performance tuning for JuiceFS in a clearly different situation, and aims to draw a lesson from others. I hope everyone learns from one another. If you have relevant ideas or experience, welcome to share and discuss in the JuiceFS forum or user group.

Recommended reading
Know x JuiceFS: Use JuiceFS to accelerate Flink container startup

project address : Github ( https://github.com/juicedata/juicefs ) If you have any help, welcome to follow us! (0ᴗ0✿)