fio recipes for shared-FS benchmarks

Generate ready-to-run fio commands tuned for an NFS-style or DPU-fronted parallel filesystem.

These are the eight runs we usually fire off when validating a freshly mounted shared filesystem (Weka, VAST, Lustre). Together they give a reasonable picture of throughput, IOPS, and tail latency without taking all afternoon — each runs for 60 seconds.

Always do the setup run first so the file exists at full size; otherwise the seq-read jobs spend most of their time waiting on writes to allocate.

Test file path

setupPre-create test file (64 GB)one-shot

fio --name=create --rw=write --bs=1m --size=64g \
  --numjobs=1 --end_fsync=1 \
  --filename=/mnt/shared/fio-testfile

Run once before the read tests. NFS clients hate writing into a sparse file mid-benchmark.

seqSequential read · 1M block8 jobs · iodepth 32

fio --name=seq-1m --ioengine=libaio --direct=1 \
  --rw=read --bs=1m --numjobs=8 --iodepth=32 \
  --size=64g --runtime=60 --time_based --group_reporting \
  --filename=/mnt/shared/fio-testfile

Throughput-oriented. Expect bandwidth close to NIC line rate on a healthy mount.

seqSequential read · 256K block8 jobs · iodepth 32

fio --name=seq-256k --ioengine=libaio --direct=1 \
  --rw=read --bs=256k --numjobs=8 --iodepth=32 \
  --size=64g --runtime=60 --time_based --group_reporting \
  --filename=/mnt/shared/fio-testfile

randRandom read · 4K · deep queue8 jobs · iodepth 64

fio --name=rand-4k-deep --ioengine=libaio --direct=1 \
  --rw=randread --bs=4k --numjobs=8 --iodepth=64 \
  --size=64g --runtime=60 --time_based --group_reporting \
  --filename=/mnt/shared/fio-testfile

IOPS test. Useful for "small-file inference" workloads.

randRandom read · 4K · shallow queue4 jobs · iodepth 4

fio --name=rand-4k-shallow --ioengine=libaio --direct=1 \
  --rw=randread --bs=4k --numjobs=4 --iodepth=4 \
  --size=64g --runtime=60 --time_based --group_reporting \
  --filename=/mnt/shared/fio-testfile

latLatency probe · iodepth 1p99 / p99.9

fio --name=lat-serial --ioengine=libaio --direct=1 \
  --rw=randread --bs=4k --numjobs=1 --iodepth=1 \
  --size=64g --runtime=60 --time_based --group_reporting --lat_percentiles=1 \
  --filename=/mnt/shared/fio-testfile

Single in-flight IO. The p99 / p99.9 numbers tell you about tail latency under no contention.

mixMixed 70 % read / 30 % write8 jobs · 256K

fio --name=mix-70r --ioengine=libaio --direct=1 \
  --rw=randrw --bs=256k --numjobs=8 --iodepth=16 \
  --size=64g --runtime=60 --time_based --group_reporting --rwmixread=70 --lat_percentiles=1 \
  --filename=/mnt/shared/fio-testfile

mixMixed 90 % read / 10 % write16 jobs · 64K

fio --name=mix-90r --ioengine=libaio --direct=1 \
  --rw=randrw --bs=64k --numjobs=16 --iodepth=16 \
  --size=64g --runtime=60 --time_based --group_reporting --rwmixread=90 --lat_percentiles=1 \
  --filename=/mnt/shared/fio-testfile

What to look for in the output

BW=… — aggregate bandwidth. For a 100GbE-fronted pool on a single client, the seq-1m run should comfortably exceed 5 GB/s.
IOPS=… — aggregate IOPS. The rand-4k-deep run is the one that stresses metadata + small-IO paths.
clat percentiles — latency distribution. p99.9 in the single milliseconds is good; tens of milliseconds means the storage tier or the network is queuing.
On NFSv3 mounts you almost always need --ioengine=libaio and --direct=1 to stop the kernel from caching everything in page cache.