JuiceFS Performance Evaluation Guide

JuiceFS is a high-performance POSIX file system designed for cloud native environments. Any data stored in JuiceFS will be split into data blocks according to certain rules and stored in object storage (such as Amazon S3), and the corresponding metadata is persisted in In a separate database. This structure determines that the storage space of JuiceFS can be elastically scaled according to the amount of data, reliably store large-scale data, and support sharing and mounting between multiple hosts, achieving cross-cloud and cross-regional data sharing and migration.

During the operation of JuiceFS, the actual performance may be different due to differences in hardware and software, system configuration, file size, etc. Previously shared [How to use JuiceFS performance tools for analysis and tuning] In this article, we will further introduce how to perform accurate performance evaluation on JuiceFS, hoping to help everyone.

Preface

Before performing a performance test, it is best to write down a general description of the usage scenario, including:

1. What is the docking application? For example, Apache Spark, PyTorch, or programs written by yourself, etc.
2. Resource configuration for application operation, including CPU, memory, network, and node size
3. Estimated data size, including file number and capacity
4. File size and access mode (large file or small file, sequential read and write or random read and write)
5. Performance requirements, such as the amount of data to be written or read per second, QPS of access, or operation delay, etc.

The clearer and more detailed the above content, the easier it is to formulate a suitable test plan and performance indicators that need to be paid attention to to determine the application's requirements for all aspects of the storage system, including JuiceFS metadata configuration, network bandwidth requirements, configuration parameters, etc. Of course, it is not easy to write all the above clearly at the beginning. Some of the content can be gradually clarified during the test, but at the end of a complete test, the above usage scenario description and the corresponding test method and test The data and test results should be complete .

If the above content is not clear, it does not matter. The built-in testing tool of JuiceFS can get the core indicators of stand-alone benchmark performance with one line of command. At the same time, this article will also introduce two built-in performance analysis tools of JuiceFS. When doing more complex tests, these two tools can help you analyze the reasons behind the performance of JuiceFS simply and clearly.

Get started quickly with performance testing

The following example introduces the basic usage of the bench tool built in JuiceFS.

Environment configuration

• Test host: one Amazon EC2 c5.xlarge
• Operating system: Ubuntu 20.04.1 LTS (Kernel 5.4.0-1029-aws)
• Metadata engine: Redis 6.2.3, storage (dir) is configured in the system disk
• Object storage: Amazon S3
• JuiceFS version：0.17-dev (2021-09-23 2ec2badf)

JuiceFS Bench

The JuiceFS bench command can help you quickly complete the stand-alone performance test, and judge whether the environment configuration and performance are normal through the test results. Assuming that you have mounted /mnt/jfs location 061b050fadfac2 of the test machine (if you need help in initializing and mounting JuiceFS, please refer to Quick Start Guide , and execute the following commands (it is recommended -p parameter 061b050fadfac3 is set to the number of CPU cores of the test machine) ):

$ juicefs bench /mnt/jfs -p 4

The test results will display various performance indicators as green, yellow or red. If there are red indicators in your results, please check the relevant configuration first. If you need help, you can describe your problem in detail in GitHub Discussions.

JuiceFS bench specific process benchmarks are as follows (its implementation logic is very simple, interested in knowing the details can be seen directly Source :

1. N concurrently write a large file of 1 GiB each, and the IO size is 1 MiB
2. N concurrently reads a large file of 1 GiB previously written, and the IO size is 1 MiB
3. N concurrently write 100 small files of 128 KiB each, the IO size is 128 KiB
4. N concurrently read 100 previously written small files of 128 KiB, the IO size is 128 KiB
5. N concurrently each stat 100 previously written small files of 128 KiB
6. Clean up the temporary directory for testing

I.e., the value of N by the number of concurrent bench command -p specified parameters.

Here is a performance comparison with several commonly used storage types provided by AWS:

• When EFS 1TiB capacity, read 150MiB/s, write 50MiB/s, the price is $0.08/GB-month
• EBS st1 is a throughput optimized HDD, with a maximum throughput of 500MiB/s, a maximum IOPS (1MiB I/O) of 500, a maximum capacity of 16TiB, and the price is $0.045/GB-month
• EBS gp2 is a general-purpose SSD, the maximum throughput is 250MiB/s, the maximum IOPS (16KiB I/O) is 16000, the maximum capacity is 16TiB, and the price is $0.10/GB-month

It is not difficult to see that in the above test, the sequential read and write ability of JuiceFS is significantly better than AWS EFS, and the throughput capacity also exceeds the commonly used EBS. However, the speed of writing small files is not fast, because every time you write a file, you need to persist the data in S3, and calling the object storage API usually has a fixed overhead of 10~30ms.

Note 1: The performance and capacity of Amazon EFS are linearly related to the reference ( AWS official document ), so it is not suitable for small data volume and high throughput scenarios.

Note 2: The price refers to AWS US East (US East, Ohio Region) , there are slight differences in prices in different regions.

Note 3: The above data comes from AWS official document . The performance index is the maximum value. The actual performance of EBS is related to the volume capacity and the mounted EC2 instance type. Generally speaking, the larger the capacity, the higher the configuration of EC2, you can get The better the EBS performance, but not exceeding the maximum value mentioned above.

Performance observation and analysis tools

Next, we will introduce two performance observation and analysis tools, which are essential tools in the process of testing, using, and tuning JuiceFS.

JuiceFS Stats

JuiceFS stats is a tool for real-time statistics of JuiceFS performance indicators, similar to the dstat command of the Linux system, which can display the indicator changes of the JuiceFS client in real time (for detailed description and usage, performance monitoring document ). When executing juicefs bench , execute the following command in another session:

$ juicefs stats /mnt/jfs --verbosity 1

The results are as follows, which can be compared with the above benchmark test process to make it easier to understand:

The specific meaning of each indicator is as follows:

• usage

  • cpu: CPU consumed by the JuiceFS process
  • mem: physical memory occupied by JuiceFS process
  • buf: The size of the read and write buffer inside the JuiceFS process, limited by the mount item --buffer-size
  • cache: internal indicators, don’t pay attention

• fuse

  • ops/lat: The number of requests processed by the FUSE interface per second and their average delay (unit: milliseconds, the same below)
  • read/write: The bandwidth value of FUSE interface processing read and write requests per second

• meta

  • ops/lat: The number of requests processed by the metadata engine per second and their average latency (please note that some requests that can be directly processed in the cache are not included in the statistics to better reflect the interaction between the client and the metadata engine. Time)
  • write transactions processed per second by the metadata engine and their average latency (read-only requests such as getattr only counted in ops, not txn)
  • retry: The number of times the metadata engine retries write transaction

• blockcache

  • read/write: read and write traffic per second of the client's local data cache

• object

  • read request processed by the object storage per second, the number of requests and their average delay
  • write request processed by the object storage per second, the number of requests and their average delay
  • del_c/lat: The number and average latency of delete request processed by object storage per second

JuiceFS Profile

JuiceFS profile used to output all the access logs of JuiceFS client in real time, including the information of each request. At the same time, it can also be used to replay and count JuiceFS access logs, so that users can intuitively understand the operation of JuiceFS (for detailed instructions and usage, see performance diagnosis document ). When executing juicefs bench , execute the following command in another session:

$ cat /mnt/jfs/.accesslog > access.log

Among them, .accesslog is a virtual file, it usually does not generate any data, only when it is read (such as executing cat ) will there be JuiceFS access log output. After the end, use Ctrl-C end the cat command, and run:

$ juicefs profile access.log --interval 0

The --interval parameter sets the sampling interval of the access log. When set to 0, it is used to quickly replay a specified log file and generate statistics, as shown in the following figure:

From the description of the previous benchmark test process, a total of (1 + 100) * 4 = 404 files were created in this test process, and each file has gone through "Create → Write → Close → Open → Read → Close → Delete" The process, so there is a total of:

• 404 create, open and unlink requests
• 808 flush requests: flush is called automatically every time the file is closed
• 33168 write/read requests: 1024 1 MiB IOs are written to each large file, and the default maximum value requested at the FUSE layer is 128 KiB, which means that each application IO will be split into 8 FUSE requests. So there are a total of (1024 8 + 100) 4 = 33168 requests. The read IO is similar, and the count is the same.

The above values can correspond exactly to the results of profile In addition, the results also show that the average delay of write is very small (45 microseconds), and the main time-consuming point is flush. This is because JuiceFS write writes to the memory buffer first by default, and then calls flush to upload data to the object storage when the file is closed, which is in line with expectations.

Other test tool configuration examples

Fio stand-alone performance test

Fio is a commonly used performance testing tool in the industry. After completing JuiceFS bench, you can use it to do more complex performance testing.

Environment configuration

It is consistent with the JuiceFS Bench test environment.

Test task

Perform the following four Fio tasks to perform sequential write, sequential read, random write, and random read tests:

# Sequential
$ fio --name=jfs-test --directory=/mnt/jfs --ioengine=libaio --rw=write --bs=1m --size=1g --numjobs=4 --direct=1 --group_reporting
$ fio --name=jfs-test --directory=/mnt/jfs --ioengine=libaio --rw=read --bs=1m --size=1g --numjobs=4 --direct=1 --group_reporting

# Random
$ fio --name=jfs-test --directory=/mnt/jfs --ioengine=libaio --rw=randwrite --bs=1m --size=1g --numjobs=4 --direct=1 --group_reporting
$ fio --name=jfs-test --directory=/mnt/jfs --ioengine=libaio --rw=randread --bs=1m --size=1g --numjobs=4 --direct=1 --group_reporting

Parameter Description:

• --name : The test name specified by the user will affect the test file name
• --directory : Test catalog
• --ioengine : The way to issue IO during testing; usually libaio is enough
• --rw : commonly used are read, write, randread, randwrite, which represent sequential read and write and random read and write respectively
• --bs : The size of each IO
• --size : The total IO size of each thread; usually equal to the size of the test file
• --numjobs : Test the number of concurrent threads; by default, each thread runs a separate test file
• --direct O_DIRECT mark bit when opening the file, without system buffering, which can make the test result more stable and accurate

The results are as follows:

# Sequential
WRITE: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=4096MiB (4295MB), run=5825-5825msec
READ: bw=817MiB/s (856MB/s), 817MiB/s-817MiB/s (856MB/s-856MB/s), io=4096MiB (4295MB), run=5015-5015msec

# Random
WRITE: bw=285MiB/s (298MB/s), 285MiB/s-285MiB/s (298MB/s-298MB/s), io=4096MiB (4295MB), run=14395-14395msec
READ: bw=93.6MiB/s (98.1MB/s), 93.6MiB/s-93.6MiB/s (98.1MB/s-98.1MB/s), io=4096MiB (4295MB), run=43773-43773msec

Vdbench multi-machine performance test

Vdbench is also a common file system evaluation tool in the industry, and it supports multi-machine concurrent testing well.

test environment

It is similar to the JuiceFS Bench test environment, except that two more hosts with the same configuration are opened, for a total of three.

Ready to work

You need to install vdbench under the same path on each node:

1. official website download 50406 version
2. Install Java: apt-get install openjdk-8-jre
3. Test vdbench installation is successful: ./vdbench -t

Then, assuming that the names of the three nodes are node0, node1, and node2, you need to create a configuration file on node0, as follows (testing a large number of small files to read and write):

$ cat jfs-test
hd=default,vdbench=/root/vdbench50406,user=root
hd=h0,system=node0
hd=h1,system=node1
hd=h2,system=node2

fsd=fsd1,anchor=/mnt/jfs/vdbench,depth=1,width=100,files=3000,size=128k,shared=yes

fwd=default,fsd=fsd1,operation=read,xfersize=128k,fileio=random,fileselect=random,threads=4
fwd=fwd1,host=h0
fwd=fwd2,host=h1
fwd=fwd3,host=h2

rd=rd1,fwd=fwd*,fwdrate=max,format=yes,elapsed=300,interval=1

Parameter Description:

• vdbench=/root/vdbench50406 : Specifies the installation path of the vdbench tool
• anchor=/mnt/jfs/vdbench : Specify the path to run the test task on each node
• depth=1,width=100,files=3000,size=128k : Defines the test task file tree structure, that is, another 100 directories are created under the test directory, each directory contains 3000 128 KiB files, a total of 300,000 files
• 061b050fae0ce4: Defines the actual test task, that is, randomly selects operation=read,xfersize=128k,fileio=random,fileselect=random

The results are as follows:

FILE_CREATES        Files created:                              300,000        498/sec
READ_OPENS          Files opened for read activity:             188,317        627/sec

The overall system creation speed of 128 KiB files is 498 per second, and the speed of reading files is 627 per second.

Summarize

This article shares the overall process of performance evaluation of JuiceFS from the perspective of environment configuration, tool introduction, and example testing. Accurate performance evaluation can help optimize JuiceFS to better suit your application scenarios. Finally, everyone is welcome to actively record and share their testing process and results to the JuiceFS forum or user group.

Recommended reading: know almost x JuiceFS: using JuiceFS Flink container to start to accelerate

If you have any help, please pay attention to our project Juicedata/JuiceFS ! (0ᴗ0✿)