I/O scheduler tuning for NVMe + Weka + scratch

The Linux block layer has an I/O scheduler that decides what order to dispatch requests to a device. The right choice matters less than it used to — modern NVMe doesn't really need scheduling, and Weka is its own world — but the wrong choice can chop bandwidth in half or invent latency that wasn't in the hardware.

This page is the operator's view: which scheduler for which class of device, how to sanity-check the choice, and the tuning that actually changes throughput on a fresh node.

Schedulers available

cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfq
#  ^^^^                                    <-- bracketed = active

# All your block devices
for d in /sys/block/*/queue/scheduler; do
  echo "$d: $(cat $d)"
done

For a GPU/HPC node, the answer is almost always:

• NVMe → none (or mq-deadline if your kernel insists on having scheduling for fairness)
• HDD (RARE on modern HPC) → mq-deadline
• Network-attached (Weka, NFS, Lustre) → N/A (the block scheduler doesn't see them)

Set persistently via udev so it survives reboot:

# /etc/udev/rules.d/60-ioschedulers.rules

# NVMe — no software scheduling
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"

# SATA/SAS SSDs — mq-deadline (rotational=0)
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", \
  ATTR{queue/scheduler}="mq-deadline"

# Real spinning disks — mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", \
  ATTR{queue/scheduler}="mq-deadline"

sudo udevadm control --reload
sudo udevadm trigger
# Verify
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfq

Why NVMe wants none

NVMe devices have multiple hardware submission queues (typically 64-128) and the firmware is doing its own scheduling internally. Adding mq-deadline on top means the kernel maintains an in-memory queue of requests, decides ordering, then submits — adding software latency for no benefit, since the firmware will reorder as it pleases anyway.

You can see the queue count:

ls /sys/block/nvme0n1/mq/
# 0  1  2  3  4  5  6  7  8  9  10  11 ... 63

cat /sys/block/nvme0n1/queue/nr_hw_queues
# 64

# Per-queue depth
cat /sys/block/nvme0n1/queue/nr_requests
# 1023                                  <-- per software queue (with mq, per hw queue)

With none, requests go through the per-CPU software queue and into the matching hardware queue with minimum overhead. This gives you the full IOPS the device can do (often 1M+ for an enterprise NVMe).

The exception: if you have noisy-neighbor concerns (multiple processes on the same node hammering the same NVMe), mq-deadline does provide some fairness via its in-memory queue. On dedicated HPC nodes that's not a real concern.

Queue depth and request counts

Two things to read:

# Per-queue request count — the in-flight budget
cat /sys/block/nvme0n1/queue/nr_requests
# 1023

# Max IO size in one request
cat /sys/block/nvme0n1/queue/max_sectors_kb
# 1280                                  <-- 1.25 MiB per req

# What the device claims it can do
cat /sys/block/nvme0n1/queue/max_hw_sectors_kb
# 32767                                 <-- ~32 MiB cap from firmware

# Read-ahead (sequential prefetch into pagecache)
cat /sys/block/nvme0n1/queue/read_ahead_kb
# 128

For HPC, where you're typically reading large dataset files in big sequential chunks:

# /etc/udev/rules.d/61-nvme-tuning.rules
ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", \
  ATTR{queue/nr_requests}="2048", \
  ATTR{queue/read_ahead_kb}="2048", \
  ATTR{queue/max_sectors_kb}="2048"

read_ahead_kb=2048 means the kernel prefetches 2 MiB ahead on sequential reads — much better cache hit rate for streaming dataset access. For a database workload (random reads), set read_ahead_kb=0 or 32.

nr_requests=2048 doubles the in-flight queue depth. Useful for fio benchmarks; mild gains on real workloads.

Weka client tuning

Weka mounts as /mnt/weka (or wherever) via FUSE-with-bypass — the kernel block layer never sees the I/O. So traditional NVMe tuning doesn't apply. What does apply:

Mount options

# /etc/fstab
default /mnt/weka wekafs net=eth1,num_cores=8,nofail 0 0

# Or in a systemd .mount unit
[Mount]
What=default
Where=/mnt/weka
Type=wekafs
Options=net=eth1,num_cores=8,nofail

The num_cores parameter is important: it pins how many cores Weka's frontend spends on client-side I/O. Too low → bottleneck. Too high → steals cores from the workload. Common values: 4-12 for an 8-GPU compute node.

Read-ahead and prefetch

Weka has its own client cache. Configure via weka CLI:

# Show current client config
weka local status

# Set the client read-ahead (in 4KB pages — 256 = 1 MiB ahead)
weka cluster client default-mount-options --options readahead-pages=256

# Restart the client process to pick up
sudo systemctl restart weka-agent

For training-data sequential reads, readahead-pages=512 or 1024 (2-4 MiB ahead) gives the best throughput. For random-access workloads, drop it to 32.

Per-mount stats

weka stats client realtime
# Aggregated stats per mount point — IOPS, throughput, latency p50/p99
weka stats client mount-points

Or read at the kernel level:

# /proc/<pid> stats for a process inside the container
cat /proc/<pid>/io
# rchar: 12345678
# wchar: 89012345
# read_bytes: 1234567890       <-- actual read from storage
# write_bytes: 234567890

Filesystem mount options

Independent of the scheduler, mount options change behavior more than most operators realize.

noatime and nodiratime

mount -o remount,noatime,nodiratime /scratch

# Or in fstab
/dev/nvme0n1p1 /scratch ext4 defaults,noatime,nodiratime 0 2

noatime disables updating the access time (atime) on every read. Default is relatime which updates atime once per day at most — already most of the way there, but noatime is even cheaper. For HPC scratch storage that reads tens of millions of files, the savings are real (no metadata write per read).

nodiratime is the same for directory atime.

lazytime

mount -o remount,lazytime /scratch

Defer atime/mtime/ctime updates to memory until either:

• The inode is evicted from cache
• 24 hours pass
• The filesystem is sync'd

Saves a write per touched inode. Pair with noatime for maximum effect on metadata-heavy workloads.

discard vs fstrim

For SSDs, discard mount option issues TRIM on every delete — but with overhead per transaction. Better:

# Disable inline discard
mount -o remount,nodiscard /scratch

# Run weekly fstrim instead
sudo systemctl enable --now fstrim.timer
sudo systemctl list-timers | grep fstrim
# fstrim.timer ... weekly

fstrim runs once per week, batches all the freed extents, and issues big TRIM commands — way less overhead than per-transaction.

data=writeback (ext4)

# /etc/fstab
/dev/nvme0n1p1 /scratch ext4 defaults,data=writeback,noatime 0 2

Default data=ordered writes data before metadata. data=writeback doesn't, which is faster but means after a crash you can have files with garbage contents (correctly-sized, but not your data). Acceptable for scratch (the dataset is regenerable). NEVER for /home or real user data.

fio recipes for NVMe characterization

Always characterize a fresh node before declaring it healthy. fio is the tool.

Sequential read (single thread, max bandwidth)

sudo fio --filename=/dev/nvme0n1 --direct=1 --rw=read \
  --bs=1M --iodepth=32 --numjobs=1 --runtime=30 --time_based \
  --group_reporting --name=seqread
# READ: bw=6500MiB/s (6.81GB/s), 6500MiB/s-6500MiB/s

Sequential write

sudo fio --filename=/dev/nvme0n1 --direct=1 --rw=write \
  --bs=1M --iodepth=32 --numjobs=1 --runtime=30 --time_based \
  --group_reporting --name=seqwrite
# WRITE: bw=4500MiB/s

Use --filename=/path/to/file --size=10G if you don't want to scribble on the device directly. Add --direct=1 to bypass pagecache and see the real device performance.

Queue-depth sweep — find the saturation point

for qd in 1 2 4 8 16 32 64 128; do
  echo "=== iodepth=$qd ==="
  sudo fio --filename=/dev/nvme0n1 --direct=1 --rw=randread \
    --bs=4k --iodepth=$qd --numjobs=1 --runtime=10 --time_based \
    --group_reporting --name=qd$qd 2>&1 | grep -E 'IOPS|lat \(usec\):'
done
# === iodepth=1 ===
#   read: IOPS=12.3k, ...
#   lat (usec): 80.0
# === iodepth=8 ===
#   read: IOPS=85.4k, ...
#   lat (usec): 90.0
# === iodepth=64 ===
#   read: IOPS=850k, ...
#   lat (usec): 75.0
# === iodepth=128 ===
#   read: IOPS=860k, ...    <-- saturated
#   lat (usec): 145.0       <-- latency starts climbing

The "knee" (where IOPS plateaus and latency climbs) tells you the optimal queue depth for your application. For PyTorch dataset reads, iodepth=16-32 hits a sweet spot.

Concurrent-job sweep — find the multi-process scaling

for nj in 1 2 4 8 16; do
  echo "=== numjobs=$nj ==="
  sudo fio --filename=/dev/nvme0n1 --direct=1 --rw=randread \
    --bs=4k --iodepth=32 --numjobs=$nj --runtime=10 --time_based \
    --group_reporting --name=nj$nj 2>&1 | grep -E 'IOPS|cpu \s'
done

If IOPS doesn't scale linearly with numjobs, you've hit a software bottleneck (kernel block layer contention, or the device's hardware queue count is < numjobs).

Mixed read/write (real-world simulation)

sudo fio --filename=/dev/nvme0n1 --direct=1 --rw=randrw --rwmixread=70 \
  --bs=64k --iodepth=64 --numjobs=4 --runtime=60 --time_based \
  --group_reporting --name=mixrw

70% reads / 30% writes is a typical mixed workload. Real HPC datasets are more like 99% read, but you should measure both.

Diagnosis with iostat / blktrace

iostat — first look

iostat -xz 1 5
# Device   r/s    rkB/s   w/s    wkB/s   ... await   %util
# nvme0n1  4523.0  234567  12.0   8000   ...  0.45    98.0
# nvme1n1   123.0    8000  45.0   2000   ...  1.20    35.0

Read column-by-column:

• r/s, w/s: IOPS
• rkB/s, wkB/s: bandwidth
• await: average request latency in ms (read + queue wait)
• r_await / w_await: separate read/write
• aqu-sz: average queue depth (in flight)
• %util: time the device was busy

%util near 100% with low aqu-sz (1-2) means the workload is single-threaded and saturating the device. %util near 100% with high aqu-sz (32+) means the device is genuinely saturated. %util near 100% on mq-deadline with high await (>10 ms) is a classic sign of scheduler thrash — switch to none.

blktrace + blkparse — request-level

When iostat says "high latency" and you want to know why:

sudo blktrace -d /dev/nvme0n1 -o nvme0n1 &
# ... run workload ...
sudo kill %1

blkparse -i nvme0n1 -o blkparse.txt | head -30
#   8,0    1   0     0.000000000  1234  Q   R 1024 + 8 [process]
#   8,0    1   1     0.000012000  1234  G   R 1024 + 8 [process]
#   8,0    1   2     0.000045000     0  D   R 1024 + 8 [process]
#   8,0    1   3     0.000234000     0  C   R 1024 + 8 [process]

The columns: Q (queued), G (got request), D (dispatched to driver), C (completed). The time between Q and D is queue wait; between D and C is hardware service time. If D→C is 2 ms but Q→D is 50 ms, the scheduler is the bottleneck.

btt (part of blktrace) summarizes:

btt -i nvme0n1 -o nvme0n1.btt
# ==================== All Devices ====================
#             ALL           MIN         AVG           MAX        N
# --------------- ------------- ------------- ------------- ---------
# Q2Q             0.000000064   0.000123456   0.012345678   12345
# Q2G             0.000000128   0.000003456   0.000234567   12345
# G2I             0.000000256   0.000001234   0.000123456   12345
# I2D             0.000000128   0.000005678   0.000456789   12345
# D2C             0.000045678   0.000234567   0.001234567   12345  <-- hardware
# Q2C             0.000045890   0.000245678   0.001345678   12345  <-- end-to-end

Q2C = queue-to-completion is the user-visible latency. D2C is the hardware-only part. The difference is software overhead (scheduler, queue plugging).

Filesystem-level diagnosis

iotop — which process is the cause

sudo iotop -oP
# Total DISK READ:    234.5 M/s | Total DISK WRITE:    12.3 M/s
# Current DISK READ:  234.5 M/s | Current DISK WRITE:  12.3 M/s
#   PID  PRIO  USER     DISK READ    DISK WRITE   SWAPIN     IO  COMMAND
# 12345  be/4  user     230.0 M/s     0.0 K/s    0.00 % 78.5 % python train.py
#   234  be/3  root       4.5 M/s     8.5 M/s    0.00 %  3.2 % kworker/u:0

-o shows only processes doing I/O. -P shows process-level (vs thread). IO column is iowait %.

pidstat -d — per-process I/O history

pidstat -d 1 5
# 14:32:11    UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
# 14:32:12   1000     12345    234567       0         0      45    python
# 14:32:13   1000     12345    240000       0         0      52    python

iodelay is the time the process spent stuck in iowait (in clock ticks). Rising = the process is waiting on storage.

Common failure modes

100% iowait, near-zero throughput

top
# %Cpu(s):  2.3 us,  1.0 sy,  0.0 ni, 0.0 id, 96.7 wa, 0.0 hi, 0.0 si

iostat -xz 1
# nvme0n1  234.0  ... await=450 ms %util=100

await of 450 ms on an NVMe is broken. Check:

# 1. Is the SSD wearing out?
sudo smartctl -a /dev/nvme0n1 | grep -i percent
# Percentage Used:                    78%        <-- close to end of life
# Available Spare:                    8%         <-- low

# 2. Is the drive thermal-throttling?
sudo smartctl -a /dev/nvme0n1 | grep -i temp
# Temperature:                        85 Celsius  <-- hot
# Warning  Comp. Temperature Time:    1234        <-- accumulated time over warning

# 3. Is the link degraded?
sudo lspci -vvv -s $(lspci | grep -i nvme | awk '{print $1}') | grep LnkSta
# LnkSta: Speed 8GT/s (downgraded), Width x4 (downgraded)  <-- bad
# Should be: Speed 16GT/s, Width x4 (Gen4) or higher

Slow boot due to fsck

journalctl -b 0 -u systemd-fsck@.service
# fsck on /dev/nvme0n1p2 took 4 min 23 s

If fsck runs every boot, the FS isn't being unmounted cleanly:

sudo tune2fs -l /dev/nvme0n1p2 | grep -E 'Mount count|Maximum'
# Mount count:              1
# Maximum mount count:      -1
# Last write:               ...

# Check shutdown ordering
journalctl -u <whatever_holds_the_fs> | grep -i 'umount\|stop'

Common cause: a systemd unit holds the mount open at shutdown (kubelet, monitoring agents, GPU operator pods). Add Before=local-fs.target and Conflicts=shutdown.target correctly.

NVMe journal replay on every boot

journalctl -b 0 | grep -i 'recovery'
# EXT4-fs (nvme0n1p2): recovery complete

This is normal after a crash, but if you see it every boot, the FS isn't sync'ing on shutdown. Force a clean shutdown sequence:

sudo systemctl set-default multi-user.target
sudo systemctl reboot --force
# Or for graceful: sudo shutdown -h now

Read-ahead too aggressive — random workload tanking

# Symptom: random-read benchmark gives 1/3 the IOPS of sequential
cat /sys/block/nvme0n1/queue/read_ahead_kb
# 4096                              <-- 4 MiB read-ahead, terrible for random

Drop to 32 or 64 for random workloads:

sudo blockdev --setra 64 /dev/nvme0n1
# or
echo 32 | sudo tee /sys/block/nvme0n1/queue/read_ahead_kb

Persistent via udev rule.

"Stalled" task warnings under heavy I/O

dmesg -T | grep -i 'blocked for'
# [time] INFO: task python:12345 blocked for more than 120 seconds.
# [time]       Tainted: G           OE     5.15.0
# [time] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
# [time] task:python state:D stack:    0 pid:12345 ppid: 1234

The task is in D (uninterruptible sleep), waiting for I/O. Check what:

cat /proc/12345/stack
# [<0000>] folio_wait_bit_common+0x123/0x456
# [<0000>] filemap_fault+0x123/0x456
# [<0000>] ...

filemap_fault = it's waiting for a pagecache page to be filled from disk. If the storage is truly slow, the only fix is faster storage. If iostat shows the device idle, the wait is on something else (a remote NFS, a stalled CIFS, a Weka client recovery).

data=writeback corruption after crash

If you mounted scratch with data=writeback and the box crashes mid-write, files written in the last few seconds may have garbage content. Always clean up:

# After unexpected reboot
sudo find /scratch -newer /var/log/last-clean-shutdown -type f -delete
# Or per-job: cleanup at the start of every job

Mark this in the operations runbook so people don't trust scratch contents after a hard reboot.

See also

• Sysctl tuning — vm.dirty_* interacts directly with block I/O
• GRUB cmdline — nvme_core.io_timeout, elevator= (deprecated)
• Hugepages — DAX-mode storage tuning
• Kernel troubleshooting playbook — broader diagnostics

External:

• Documentation/admin-guide/blockio.rst (Linux kernel)
• man fio, man iostat, man blktrace
• Weka docs — Linux client tuning
• Linux block/blk-mq.c (read-only — for the truly curious)