NFS for GPU clusters — when it works, when it does not, and how to make it not break

NFS is on almost every GPU cluster that was set up by engineers who knew how to deploy NFS and needed something running quickly. It works. For the right workload and the right cluster size, it keeps working. For the wrong workload or a cluster that grew past its sizing assumptions, it becomes the loudest problem in the operations queue. This page is the decision guide.

Why NFS at all

The case for NFS in a GPU cluster is simple: every Linux node can mount it, nothing needs to be installed, and a single well-configured NFS server can deliver 10–20 GB/s to clients. For a 32-GPU cluster doing LLM token pretraining (where the dataloader reads are modest), that's sufficient and costs a fraction of a Weka or Ceph deployment.

The honest limitations:

For clusters ≤ 32 GPUs or workloads that are compute-bound (LLM tokenized pretraining), NFS is a reasonable choice. For clusters ≥ 128 GPUs or video/vision workloads with high per-GPU read bandwidth, NFS almost always becomes the bottleneck.

pNFS and NFSv4.2

What pNFS adds

pNFS (parallel NFS, standardized in NFSv4.1) splits the metadata path from the data path. The NFS server handles metadata (open, stat, namespace). Data is served directly from storage backends — potentially multiple, in parallel.

In theory: pNFS with multiple data servers delivers linear bandwidth scaling. In practice:

• Vendor support is uneven: pNFS block layout (pNFS/block) is reasonably mature. pNFS/file (parallel data servers over standard NFS) has fewer production deployments. pNFS/object is largely theoretical.
• Clients need pNFS-aware mount: Linux kernel NFSv4.1 client supports pNFS file and block layouts since kernel 3.0+. Verify with mount | grep pnfs or cat /proc/fs/nfsfs/volumes.
• Server must export with pNFS enabled: NetApp ONTAP, Pure FlashBlade, and some commodity Linux NFS stacks support pNFS.

# Check if a mounted NFS filesystem is using pNFS
$ cat /proc/fs/nfsfs/volumes
NV SERVER   PORT DEV          FSID                              FSC
v4 10.0.1.5 2049 0:47         7bb6...                           no
# pNFS: look for "pNFS: YES" or check nfsstat -m output
$ nfsstat -m | grep -i pnfs

NFSv4.2 additions

NFSv4.2 (kernel 4.9+) adds:

• Server-side copy: copy_file_range() — file copies happen on the server without passing data through the client.
• Sparse files: efficient seek over holes.
• Labeled NFS: mandatory access control attributes.
• READ_PLUS: returns sparse file data more efficiently.

For GPU clusters, server-side copy is the most relevant — dataset preparation scripts that copy files between directories benefit from it.

# Force NFSv4.2 mount
mount -t nfs -o vers=4.2 <server>:/<export> /mnt/nfs
# Verify version negotiated
mount | grep nfs   # Should show "nfs4" with 4.2 in options
nfsstat -m | grep vers   # Should show "vers=4.2"

Server sizing and options

ZFS-on-Linux

For commodity NFS servers, ZFS-on-Linux is the standard choice. It provides checksumming, snapshots, compression, and no RAID needed (ZFS handles it).

Typical build for a high-performance NFS server:

• 8–16× NVMe SSD in a ZFS RAID-Z2 or striped mirror pool
• 2× 25 GbE or 1× 100 GbE NIC (bonded for NFS)
• 256–512 GB RAM (ZFS ARC cache is critical — size RAM generously)
• NFS server: Linux kernel NFS server (nfs-kernel-server) or NFS Ganesha (user-space)

# ZFS pool: striped mirrors for sequential BW (best for training data)
zpool create training-data mirror nvme0 nvme1 mirror nvme2 nvme3 mirror nvme4 nvme5

# ZFS tuning for NFS workloads
zfs set recordsize=1M training-data         # large record for sequential IO
zfs set compression=lz4 training-data       # lz4 is fast, good for compressed datasets
zfs set atime=off training-data             # disable atime updates (eliminates write per read)
zfs set xattr=sa training-data             # store xattrs in SA for better metadata perf

# NFS Ganesha (user-space) can deliver better throughput than kernel NFS for large files
# but requires more tuning; kernel NFS is simpler and well-tested

Expected performance: a ZFS-on-Linux box with 4× NVMe in RAID-Z1 and 100GbE NIC delivers approximately 8–15 GB/s sequential read (depending on NVMe model, ZFS recordsize, and NFS rsize). Two or more such servers with DNS round-robin gives 16–30 GB/s aggregate, but that's not truly parallel — the client chooses one server per mount.

NetApp ONTAP

ONTAP is a production-grade NFS server with mature NFS-over-RDMA support, pNFS, and enterprise HA. It's significantly more expensive than commodity Linux but delivers consistent throughput and support contracts that some organizations require.

ONTAP FAS/AFF series with NFS-over-RDMA (NFS/RDMA or "NFS-oRDMA") provides:

• Sub-100 µs read latency (vs. 200–500 µs for TCP-based NFS)
• Near-line-rate throughput up to the cluster's aggregate interface BW

Pure FlashBlade

Pure FlashBlade is a purpose-built scale-out NAS. It supports pNFS natively and delivers consistent throughput scaling with blade count. Relevant for operators who want NFS semantics with parallel scaling — essentially pNFS done by the appliance, transparent to the client.

FlashBlade S-series: up to 64 blades × 17.5 GB/s per blade = 1.1 TB/s aggregate (vendor specification). Real-world training workload performance is typically 60–70% of peak spec.

NFS-over-RDMA setup

NFS-over-RDMA (RFC 5667) bypasses the TCP stack for NFS data, using RDMA verbs directly. This reduces CPU overhead and latency significantly.

Requirements

• Linux kernel ≥ 5.3 for stable NFS-over-RDMA client
• RDMA-capable NICs on both client and server (ConnectX-7 in RoCE or IB mode)
• NFSv4.1 or later (pNFS or standard v4.1)

Server setup

# Server: load RDMA transport module
modprobe svcrdma
echo rdma 20049 > /proc/fs/nfsd/portlist   # Register RDMA on port 20049

# Verify RDMA transport is listening
rpcinfo -p localhost | grep 20049
# Should show nfs service on port 20049 proto rdma

Client mount

# Client: mount using RDMA transport
mount -t nfs -o vers=4.1,proto=rdma,port=20049 <server-ip>:/<export> /mnt/nfs

# Verify RDMA is in use
cat /proc/mounts | grep nfs   # Should show proto=rdma
nfsstat -m | grep proto        # Should show rdma

With NFS-over-RDMA on a ConnectX-7 100G NIC:

• Read latency drops from ~200 µs (TCP) to ~80–120 µs (RDMA)
• CPU overhead on the NFS server drops by 40–60% at high bandwidth
• Maximum throughput increases by 10–20% vs. TCP at 100G

Client mount options that matter

# Full recommended mount options for GPU cluster clients on NFSv4.2
mount -t nfs \
  -o vers=4.2,\
     rsize=1048576,\
     wsize=1048576,\
     hard,\
     timeo=600,\
     retrans=5,\
     nconnect=16,\
     noresvport,\
     noatime \
  <server>:/<export> /mnt/nfs

Option explanation:

nconnect — the most impactful option

A single TCP connection to an NFS server is rate-limited by the TCP window and CPU processing on one core. nconnect=16 opens 16 parallel TCP connections, each managed by a different CPU core on both ends. This is the single most impactful option for training dataloaders on commodity NFS.

Before nconnect (introduced in Linux 5.3):

Single connection: ~2–4 GB/s from a 25 GbE NFS server

With nconnect=16:

16 connections: ~10–18 GB/s from the same 25 GbE NFS server
(limited by server NIC, not by TCP connection overhead)

Verify nconnect is working:

# Check that multiple TCP connections are established to the server
ss -tn dst <server-ip>:2049 | wc -l
# Should show 16 (or nconnect value) connections

hard vs soft

Always use hard for training workloads. A soft mount returns EIO to the application when the server doesn't respond within timeo. In a training job, EIO in the dataloader causes the entire training process to crash, not just a timeout. hard retries indefinitely, which is correct for transient network events.

The downside of hard: a permanently failed server causes the mount to hang. Processes blocked on hard NFS operations become unkillable (D-state). Recovery requires either the server coming back or a forceful unmount.

Workload fit analysis

Training dataset broadcast — works

When all GPU nodes read the same training dataset files (or different shards from the same directory), NFS reads scale reasonably because:

• The NFS server and clients cache the data in their page caches.
• After the first epoch, most reads are served from the NFS server's RAM (ZFS ARC, or server page cache).
• Sequential read patterns are NFS-friendly.

At what scale does it break? The NFS server becomes a bottleneck when aggregate client read bandwidth exceeds server I/O bandwidth. For a ZFS server with 100 GbE:

• Single server read cap: ~12–15 GB/s sustained
• At 32 GPUs × 150 MB/s = 4.8 GB/s aggregate: fine
• At 128 GPUs × 150 MB/s = 19.2 GB/s aggregate: beyond single-server capacity

Shared model checkpoint storage — works with care

Checkpointing over NFS is viable if:

• Checkpoints are written by one rank (rank-0) and other ranks write locally, then rank-0 aggregates.
• Checkpoint files are large (hundreds of MB to GB) — sequential write is NFS-friendly.
• You don't have more than ~16 concurrent checkpoint writers per NFS server.

It breaks when:

• All 1024 GPU processes write their own shard files simultaneously (concurrent small writes to many files = metadata storm on the NFS server).
• Checkpoint frequency is high (every 5 steps at 1 step/second = 200 MB/s average write from a large cluster — rapidly saturates NFS).

Distributed checkpoint write — wrong choice

Distributed checkpointing (PyTorch DCP, Megatron-LM checkpoint sharding) where every GPU rank writes its own shard file in parallel: do not do this to NFS at scale. At 1024 GPUs:

• 1024 simultaneous open() + write() + close() operations hit the NFS server's metadata stack simultaneously.
• NLM contention (if any file-level locking is in use) causes queuing.
• The result: checkpointing that should take 30 seconds takes 5–10 minutes, or fails entirely.

For distributed checkpointing: use Weka, Ceph, or a parallel filesystem. NFS is not the right tool.

Operational pitfalls

ESTALE storms

An ESTALE error means the client holds a filehandle that the server no longer recognizes. This happens after:

• NFS server restart (the filehandle cache is rebuilt and old handles are invalid)
• ZFS pool export/import (new pool generates new filehandles)
• NFS export path changed on the server

At scale, an ESTALE storm hits all clients simultaneously post-restart. Symptoms:

# Clients see errors like:
NFS: server <ip> OK
stale file handle: <path>
# Or in Python:
OSError: [Errno 116] Stale file handle: '/mnt/nfs/dataset/shard_0.bin'

Recovery: remount the NFS filesystem on all clients. With 1024 clients, this requires an orchestrated remount (Ansible, pdsh, or similar).

Prevention:

• NFSv4 has better state recovery than v3. Use vers=4.2.
• For ZFS, never export/import the pool while NFS is serving — suspend NFS first.
• Run NFS server HA: two servers with shared ZFS pool (iSCSI-backed) and floating IP. Client sees a single IP; failover is transparent if the new server picks up filehandles from the replicated state.

Lock manager flapping

NFSv3 NLM (Network Lock Manager) can enter a state where the lock daemon (statd) on clients and servers repeatedly connect and disconnect. Symptoms:

• Huge number of rpcsvc_send_response errors in server dmesg
• Client processes stuck in D state waiting for lock
• rpc.statd consuming high CPU on server

Fix for most cases:

# Restart NFS services on server
systemctl restart nfs-server
# Restart statd on affected clients
systemctl restart rpc-statd
# Or: avoid the problem entirely by using NFSv4 (no NLM, uses built-in locking)
# Mount option: vers=4.2 (NLM replaced by NFSv4 lock protocol)

Stuck mounts post-network blip

With a hard mount, a network partition causes all I/O to the NFS server to hang. Processes block in the kernel waiting for the NFS response. After the network recovers, operations resume — but during the outage, nothing progresses.

In training jobs: the job appears hung. If the network blip is longer than the job scheduler's health-check timeout (Slurm: HealthCheckInterval), the node may be drained and the job cancelled.

# Check for stuck NFS operations
$ cat /proc/sys/sunrpc/nfs_tcp_retrans   # Number of retransmits before considering server dead
$ nfsiostat 1   # Watch per-mount read/write ops — zero ops indicates blocked
$ ps aux | grep D   # D-state processes = blocked in uninterruptible sleep (often NFS)

# Forceful unmount if stuck (requires the mount to be not busy):
$ umount -l /mnt/nfs   # Lazy unmount — detaches filesystem but allows current operations to finish
$ umount -f /mnt/nfs   # Force unmount — more aggressive, may kill stuck operations

Migration paths

V1 of a cluster: NFS as the starting point

Starting with NFS is a legitimate pattern for a new cluster. The reasons:

• Fast to deploy — no new tooling, every team knows it.
• Sufficient for early workloads (30–64 GPU development jobs).
• Lets the team learn the cluster's actual storage access patterns before committing to a parallel filesystem architecture.

Planning for growth:

• Namespace design: structure the NFS export paths so that the directory hierarchy is portable. /mnt/nfs/datasets/<project>/<dataset> should work whether the backend is NFS, Weka, or CephFS.
• Mount point abstraction: use symlinks or automounter maps so that changing from NFS to Weka requires only updating the symlink target, not updating all training scripts.
• Checkpoint directory: keep checkpoints on a separate mount from training data from day one. This lets you migrate the checkpoint store to a parallel filesystem without touching the dataset mount.

Growing from 32 to 256 GPUs

Stage 1 (≤32 GPUs):
  Storage: single ZFS-on-Linux NFS server, 4× NVMe, 100 GbE
  Throughput: ~12 GB/s aggregate
  Suitable for: LLM token pretraining, fine-tuning

Stage 2 (33–128 GPUs):
  Option A: add a second NFS server + load balancer (DNS round-robin)
            → two independent NFS namespaces; migration complexity
  Option B: add a small Weka cluster (4–6 backend hosts)
            → single namespace, parallel reads, same client mount point
            → Weka client replaces NFS client on GPU nodes
  
  Trigger: NFS server CPU or NIC at >70% sustained during a training job

Stage 3 (128–1024 GPUs):
  NFS is untenable for write-intensive checkpoint workloads.
  Weka or Ceph (CephFS) is the right answer.
  Migration: use rsync to copy dataset from NFS to new filesystem,
             then flip the mount point symlink.

Worked example: 32-GPU lab vs. 256-GPU production

32-GPU lab — lives happily on ZFS NFS

Cluster: 4× 8-GPU H100 servers (32 GPUs total)
Workload: LLM fine-tuning, 2 TB dataset, 1 team
Target read BW: 32 × 120 MB/s = 3.8 GB/s
Target write (checkpoints): 1 GB/s burst

Server: 1× commodity 2U server
  - 4× 3.84 TB NVMe in ZFS mirror pairs
  - 1× 25 GbE NIC
  - 256 GB RAM (ZFS ARC)
  - nfs-kernel-server, NFSv4.2

Capacity: 4× 3.84 TB × 0.5 (mirror) = 7.68 TB usable
Read BW: ~8 GB/s sustained (2× mirror pairs, striped)
Client options: vers=4.2, rsize=1M, nconnect=8, hard

Result: 3.8 GB/s read target << 8 GB/s server capacity — works fine.
Cost: ~$15K server + 4× NVMe drives ≈ $25–35K total. 
Weka for this cluster would add ~$40–80K licensing on top. Not justified.

Same workload at 256 GPUs — NFS breaks

Cluster: 32× 8-GPU H100 servers (256 GPUs total)
Same workload but 8× scale
Target read BW: 256 × 120 MB/s = 30.7 GB/s
Target checkpoint write burst: 8 GB/s (all nodes write simultaneously)

NFS server options:
  - Single 100 GbE server: max 12 GB/s → 30.7 GB/s target is 2.5× over capacity → fails
  - Two 100 GbE servers: max 24 GB/s → still below target, and namespace is split
  - Three 100 GbE servers: ~36 GB/s → possible, but 3 separate namespaces, no atomic consistency
  
Issues at 256 GPUs:
  - Concurrent checkpoint writes from 256 processes: NFS metadata storm at job boundaries
  - Client count × nconnect = 256 × 16 = 4096 TCP connections per server — server TCP stack overhead
  - Single point of failure per server — if any server goes down during training, job fails

Right answer at 256 GPUs: Weka (4–6 backend hosts) or CephFS (6–8 OSD hosts).
NFS is no longer the right tool.