NVMe over Network (NVMe-oF) for GPU clusters — benefits, drivers, tuning, and when not to use it

NVMe-oF (NVMe over Fabrics) removes the assumption that NVMe storage must be inside the server. It exports the NVMe command queue directly over a network transport, so the initiator (client) sends IO operations to a remote NVMe namespace almost as if the drive were local. The "almost" matters a great deal in GPU cluster design, and this page is the honest accounting of where that gap closes and where it does not.

The short version before the detail: NVMe-oF is the right answer for a narrow set of GPU workloads — primarily single-stream low-latency reads (inference weight loading) and single-tenant dedicated checkpoint targets. For the wider training-pipeline use case, parallel filesystems (Weka, Ceph) still win on multi-tenant aggregate bandwidth. Read both before deciding.

Why NVMe-oF at all

The latency floor that parallel filesystems cannot match

A POSIX-compliant distributed filesystem (Weka, CephFS, Lustre, pNFS) introduces a metadata layer. Every open(), stat(), and close() has to traverse the metadata server path. Even in Weka's distributed metadata implementation, a cold open() typically costs 200–500 µs. CephFS MDS p50 is in the 1–3 ms range. That overhead is per-file, and it matters when a workload reads many small files or when startup latency for the first IO of a job is on the critical path.

NVMe-oF has no metadata server. The initiator has a namespace handle and sends read or write commands directly to the target's NVMe drive queue. The practical latency floor:

NVMe-oF / RoCE v2 (ConnectX-7 NDR, Gen5 NVMe):
  4 KB read, single-queue-depth-1: ~50–80 µs end-to-end (wire + NVMe)

  Breakdown (approximate):
    NVMe drive local read latency:        ~15–30 µs (Gen5 enterprise NVMe)
    RoCE v2 wire latency (NDR at 400G):   ~2–5 µs
    NVMe-oF target CPU / offload:         ~5–20 µs (ConnectX-7 with offload)
    NVMe-oF initiator stack:              ~10–20 µs
    Total:                                ~50–80 µs

NVMe-oF / TCP (100 GbE, same Gen5 NVMe):
  4 KB read, single-queue-depth-1: ~150–300 µs end-to-end

Weka tier-1 (NVMe behind DPDK parallel filesystem, same drives, same fabric):
  4 KB read p50: ~100–200 µs
  4 KB read p99: ~500 µs–1 ms (metadata contention under concurrency)

These are approximate operator rules of thumb. The NVMe-oF latency floor is real and measurable. The question is whether your workload is latency-bound at this granularity.

When the latency floor matters for GPU work

Three scenarios where the NVMe-oF latency advantage translates to real application gains:

1. Inference weight loading at startup. A 70B parameter BF16 model weighs roughly 140 GB. Loading it into GPU HBM at job start is a single-stream sequential read. The lower the read latency floor, the faster the replica boots. With GDS (GPUDirect Storage), the read path is GPU → NIC → wire → target NIC → NVMe, with zero CPU copies.

2. Checkpoint reload after a failure. After a training run crashes at step N, the job restarts from the last checkpoint. With a single dominant tenant on dedicated NVMe-oF storage, the checkpoint reload is a single-stream sequential read per host. NVMe-oF with RDMA gets it off disk faster than a shared parallel filesystem can.

3. Single-tenant dedicated I/O lane. If one tenant has exclusive use of a set of NVMe drives and does not need to share namespace with others, raw NVMe-oF eliminates the overhead of the distributed metadata layer entirely. The tradeoff is that the operator now owns what the parallel filesystem was providing: sharding logic, quota enforcement, rebalancing, and erasure-coded durability.

When the latency floor does NOT matter for GPU work

Most training pipeline I/O is bandwidth-bound, not latency-bound. A dataloader reading shuffled tokenized sequences from a shared corpus accesses hundreds of files concurrently across dozens of nodes. The throughput matters; the per-read latency at sub-millisecond granularity does not. Weka's parallel read path scales with backend host count and delivers 20–30 GB/s per backend host. A 6-host Weka cluster can sustain 100+ GB/s aggregate read bandwidth, which is what a 512-GPU vision pretraining job actually needs.

NVMe-oF's latency advantage evaporates under concurrency: when many initiators compete for the same target, the target's NVMe queue depth fills up, and the effective latency climbs toward the same ballpark as a well-tuned parallel filesystem. The architectural difference is that NVMe-oF has no mechanism for horizontal scaling across drives from multiple hosts unless the client-side software explicitly shards across multiple subsystems. A parallel filesystem provides that by default.

The honest summary:

Transports compared

NVMe/TCP

NVMe/TCP encapsulates NVMe command capsules inside TCP segments. No special hardware required beyond any Ethernet NIC. Linux kernel support landed in 5.0 (initiator) and 5.4+ for the target (nvmet-tcp).

Latency: 150–300 µs add latency compared to local NVMe. TCP's flow control and per-packet CPU processing are the dominant cost. Under high queue depths the throughput can approach line rate, but single-outstanding-IO latency is substantially worse than RDMA transports.

CPU overhead: significant. TCP processing on the initiator consumes CPU cores proportional to bandwidth. At 100 GbE line rate (roughly 12 GB/s), expect 2–4 CPU cores fully committed to NVMe/TCP receive processing on the initiator, depending on whether interrupt coalescing and polling mode are tuned.

When to pick NVMe/TCP:

• Heterogeneous networks: some nodes have RDMA NICs, some do not. TCP transport works everywhere.
• Small-scale clusters or dev environments where RDMA fabric configuration overhead is not justified.
• Clusters where the storage admin team is not familiar with RoCE DCB tuning.
• When the workload is bandwidth-tolerant (sequential read, not latency-sensitive).

When not to pick NVMe/TCP: latency-sensitive inference weight loading where the extra 100–200 µs per IO matters across hundreds of thousands of IOs at startup.

NVMe/RoCE v2 (NVMe-oF over RDMA)

NVMe/RoCE v2 carries NVMe commands over RDMA verbs on a RoCEv2 (RDMA over Converged Ethernet version 2) fabric. The transport is rdma in the NVMe-oF connection string.

Latency: sub-100 µs add latency when the network is properly configured. The RDMA path bypasses the TCP stack entirely; the NVMe command is DMA'd directly from the initiator's memory into the target's NVMe queue, and the completion is DMA'd back.

Requirements: DCB (Data Center Bridging), PFC (Priority Flow Control), and ECN (Explicit Congestion Notification) must be configured on every switch and NIC on the path. Without PFC, RoCE is lossy and performance degrades severely under congestion. This is the same DCB requirement as NCCL RoCE — the fabric setup is identical. If the cluster already runs NCCL over RoCE v2, the fabric configuration is already there.

Hardware: Mellanox ConnectX-6 Dx and newer for both initiator and target. ConnectX-5 works but lacks target-side NVMe-oF offload; all command processing is on the host CPU. ConnectX-6 Dx introduces hardware offload for the NVMe-oF target, and ConnectX-7 extends it to full offload (see §3).

When to pick NVMe/RoCE v2:

• Clusters that already operate RoCE v2 for NCCL (the DCB tuning is already done).
• Production HPC deployments where sub-100 µs IO latency is a real requirement.
• Inference serving clusters loading large model weights at startup.

When not to pick NVMe/RoCE v2: clusters with no existing RDMA fabric and no budget or expertise to configure DCB/PFC/ECN. The DCB configuration surface area is real and mistakes cause silent performance degradation that is hard to diagnose.

NVMe/IB (RDMA over InfiniBand)

NVMe/IB is NVMe-oF over the InfiniBand transport. The latency is comparable to RoCE v2 (sometimes marginally better due to InfiniBand's native credit-based flow control), but the DCB/PFC configuration requirement disappears — InfiniBand manages congestion natively and does not need Ethernet-level flow control overlays.

When to pick NVMe/IB: clusters that already run InfiniBand for compute traffic. The storage target speaks the same IB verbs as the compute fabric, and there is no separate configuration layer for lossless behavior. Operationally simpler than adding RoCE configuration.

When not to pick NVMe/IB: clusters that use Ethernet for storage and IB for compute. Mixing transports means dual fabric management overhead. Unless there is a strong operational reason, align storage and compute on the same fabric type.

NVMe/FC

NVMe over Fibre Channel is the enterprise SAN extension of NVMe-oF. It carries NVMe commands over an FC fabric using NVMe FC Login Service (NVME-LS) and the FC-NVMe protocol. Latency is competitive with RDMA (~50–100 µs end-to-end with a modern all-flash array).

For GPU fleets, FC is rarely the right answer:

• FC infrastructure (HBAs, switches, zone management, SAN fabric) is expensive, operationally separate from the Ethernet/IB compute fabric, and requires dedicated SAN expertise.
• GPU cluster storage traffic is LAN-native; adding a separate FC SAN doubles the fabric management surface area without a latency advantage over RoCE.
• GPU servers (DGX, HGX, whitebox) do not ship with FC HBAs; adding them competes for PCIe slots that are already constrained.

NVMe/FC is mentioned here for completeness. If your organization already operates an FC SAN for other workloads and wants to expose NVMe namespaces to GPU nodes, it works. But greenfield GPU cluster storage does not deploy FC.

Transport comparison

Compatible NICs, SmartNICs, and DPUs

Mellanox / NVIDIA ConnectX series

ConnectX-5 (25G/100G):

• Works as NVMe-oF initiator and target with kernel nvmet or SPDK.
• No hardware offload for NVMe-oF target; all command processing runs on the host CPU.
• Minimum firmware: 16.28.x or newer for stable nvme_rdma module pairing.
• MLNX_OFED 5.4+ for full kmod coverage. ConnectX-5 is functional but should not be deployed as a new target in 2025–2026 — CPU offload absence makes it the bottleneck at high queue depths.

ConnectX-6 Dx (100G):

• NVMe-oF target offload begins here. The NIC can process NVMe capsules in hardware, removing them from the host CPU data path.
• Minimum firmware: 22.34.x for stable NVMe-oF target offload.
• MLNX_OFED 5.8+ or OFED 23.10+ for full target offload kmod support.
• Viable for production targets but ConnectX-7 is preferred for new deployments.

ConnectX-7 NDR (400G single-port or 2× 200G dual-port):

• Full NVMe-oF target and initiator offload. Recommended for production deployments in 2025–2026.
• Minimum firmware: 28.36.x for NVMe-oF target offload stability.
• MLNX_OFED 23.10+ covers NVMe-oF initiator; OFED 24.01+ adds improvements to target offload scheduling.
• PCIe Gen5 x16 required for full NDR throughput — same slot requirement as compute NICs (see §7 on PCIe lane budget).

ConnectX-8 XDR (800G):

• Full offload with XDR fabric support. For NVMe-oF targets where the storage fabric runs at XDR speed.
• Minimum firmware: 28.40.x or newer (as of 2025–2026 timeframe).
• OFED 24.x required for XDR kmod coverage.
• Primarily relevant for next-generation deployments; most production NVMe-oF clusters as of 2025 run ConnectX-7 NDR.

BlueField-2 and BlueField-3 DPUs

BlueField DPUs combine an RDMA NIC with an embedded ARM processor. For NVMe-oF, they are most useful on the target side — the ARM cores on the DPU can run the NVMe-oF target software (SPDK or nvmet) entirely offloaded from the host CPU. The host server then has its CPU free for other work.

• BlueField-2: supports NVMe-oF target in DPU mode. Run SPDK on the ARM cores; the DPU's NIC presents the target to the fabric while the host NVMe drives appear as PCIe-attached devices to the DPU. Firmware: BFB 3.9.x or newer.
• BlueField-3: same architecture, more ARM cores, higher NIC speed (400G). Better suited for dense target configurations. BFB 4.x+.

On the initiator side, BlueField is typically overkill. A standard ConnectX-7 initiating NVMe-oF commands is sufficient; the DPU's ARM offload adds complexity without benefit for the initiator path.

Broadcom Stingray

Broadcom's Stingray SmartNIC (PS series) supports NVMe-oF initiator and target in firmware. Less common in GPU HPC environments; the operational toolchain and driver ecosystem are thinner than Mellanox/NVIDIA. Mention it here because some server platforms include Stingray-based OCP cards, but for GPU-specific NVMe-oF deployments, ConnectX-7 is the operational default.

Intel IPU (E2000)

Intel's Infrastructure Processing Unit (E2000, "Mount Evans") supports NVMe-oF offload via the IDPF/IPsec/NVMe-oF firmware pipeline. Niche deployment in GPU clusters; driver support in upstream Linux (5.20+) is present but the operational surface area is narrower than Mellanox. If an OEM server bundle includes the E2000 IPU, it is a viable option for NVMe/TCP — for RDMA-based NVMe-oF, prefer ConnectX-7.

Compatible drives (target side)

PCIe Gen4 enterprise NVMe

The current production workhorse tier. Representative drives:

These are vendor-published datasheet numbers; actual numbers under NVMe-oF load depend on queue depth, block size, and target-side CPU/offload configuration.

For checkpoint write paths with moderate DWPD requirements, 1 DWPD is typically adequate for training workloads that checkpoint every few hundred steps. If checkpoint frequency is very high (every step on a large model), verify DWPD budget explicitly.

PCIe Gen5 enterprise NVMe

The latency improvement from Gen4 to Gen5 at the NVMe level is modest for sequential reads (the bottleneck shifts to the flash itself, not the PCIe interface). The latency advantage shows up for random 4 KB reads where PCIe transfer overhead is a larger fraction of total latency:

At the application layer, the latency improvement over Gen4 is visible for workloads that are genuinely latency-bound (queue depth 1, random reads). For high-queue-depth sequential reads (the dominant NVMe-oF workload for model weight loading), Gen4 and Gen5 converge — the bandwidth doubles but the per-IO latency difference narrows. Use Gen5 when your target server platform supports it and the workload is latency-sensitive; otherwise Gen4 is the cost-effective choice.

Form factors: U.2, U.3, E1.S, E3.S

For NVMe-oF target servers, U.2 is the most operationally common form factor due to broad drive model availability. E1.S is preferred in JBOF or storage-dense 2U targets where physical drive count matters more than per-drive capacity.

QLC vs TLC — explicit recommendation

TLC (triple-level cell): use for all checkpoint write paths and model-weight rotation workloads. Write endurance is 1–3 DWPD, which covers checkpoint-heavy training pipelines.

QLC (quad-level cell): acceptable for read-heavy training data tiers — the static dataset that is read many times but written rarely. QLC's lower write endurance (~0.1–0.3 DWPD) is adequate when the drive is written once (dataset ingest) and read many times (training epochs).

Explicit recommendation: never use QLC for the checkpoint write path or model-weight rotation. Checkpoint writes at high frequency saturate QLC endurance budgets in months, not years. A 70B model checkpoint at 504 GB, written every 200 steps at one step per second, writes ~2.5 GB/s averaged over the checkpoint window. Over a 12-month training campaign, that is approximately 78 PB of writes — far exceeding QLC endurance on any drive in a reasonably-sized target.

Driver and kernel prerequisites

Minimum kernel versions

NVMe/TCP initiator: kernel 5.0+ (merged with the nvme-tcp module). Functional but 5.14+ is recommended for stable multipath and error recovery behavior. For production, 5.15 LTS is the minimum supported baseline.

NVMe/RoCE v2 initiator (nvme_rdma module): present since kernel 4.19, stable for production at 5.4+. Kernel 5.15 LTS is the recommended floor; 6.1 LTS adds improvements to RDMA queue pair error recovery.

NVMe target (nvmet and nvmet-tcp): kernel 5.0+ for nvmet-tcp; nvmet over RDMA at 5.0+ with OFED overlay or 5.4+ native kernel. For production target deployments, 5.15 LTS minimum.

NVMe Native Multipath: introduced at kernel 5.0 (CONFIG_NVME_MULTIPATH=y). This is the preferred multipath mechanism for NVMe-oF (handles ANA groups natively). DM-Multipath can also work but adds configuration complexity; NVMe Native Multipath with ANA is simpler and preferred for new deployments.

MLNX_OFED version matrix

For clusters running ConnectX-7 with OFED 23.10+, NVMe-oF target offload is stable. The 24.x series adds XDR support and improves target CPU utilization under high fan-out (many initiators connecting to one target).

nvme-cli version

Minimum nvme-cli 2.6 for nvme connect-all discovery improvements. Earlier versions have known bugs with:

• ANA state transitions during failover (nvme connect-all not reconnecting to recovered namespaces).
• nvme list-subsys output format changes that break older parsing scripts.

Install or build from source if the distro package is below 2.6:

# Check installed version
nvme version
# Should output: nvme version 2.6 or higher

Distro matrix

Ubuntu 24.04 is the cleanest new-deployment target for NVMe-oF: the kernel version, OFED support, and nvme-cli version all land in a good place without additional source builds.

irqbalance on the NVMe-oF data path

irqbalance will move RDMA queue IRQs to balance CPU load across cores. On an NVMe-oF target or initiator, this interferes with NUMA pinning and can significantly degrade throughput by moving interrupt processing to cores that are not NUMA-local to the NIC.

Recommendation: disable irqbalance on NVMe-oF nodes and pin IRQs manually, or use tuned-adm profile network-latency which sets irqbalance to managed mode with topology hints.

# Disable irqbalance (preferred for NVMe-oF production targets)
systemctl stop irqbalance
systemctl disable irqbalance

# Then pin NIC IRQs to NUMA-local cores (example for mlx5_0, NUMA node 0)
# Get IRQ numbers for the NIC
cat /sys/class/infiniband/mlx5_0/device/local_cpulist
# Outputs: 0-31 (cores on NUMA node 0 for this NIC)

# Set CPU affinity for each queue IRQ
for irq in $(grep mlx5 /proc/interrupts | awk '{print $1}' | tr -d ':'); do
  echo 0x0000FFFF > /proc/irq/${irq}/smp_affinity  # cores 0-15, NUMA 0
done

Alternatively, use the tuned profile which handles IRQ affinity as part of the profile:

tuned-adm profile network-latency

Target setup — minimal viable production config

Reference hardware spec

Two NVMe-oF target hosts, each:

• 16× PCIe Gen4 U.2 NVMe (7.68 TB each, TLC) = 122.88 TB raw per host
• 2× ConnectX-7 NDR 400G (dual-rail, active/active)
• 2× 10G out-of-band management NIC
• 2-socket CPU (AMD EPYC or Intel Xeon), ≥512 GB RAM
• Ubuntu 24.04 or RHEL 9.4+, OFED 23.10+, nvme-cli ≥ 2.6

Kernel target with nvmet (LIO-based)

The nvmet kernel target is the simplest path. It uses the Linux target framework (LIO) via the NVMe target subsystem (nvmet, nvmet-tcp, nvmet-rdma). Config is exposed via /sys/kernel/config/nvmet/.

# Load modules (add to /etc/modules-load.d/nvmet.conf for persistence)
modprobe nvmet
modprobe nvmet-rdma    # for RoCE v2 / IB transport
modprobe nvmet-tcp     # for TCP transport

# Create a subsystem (one per target group)
mkdir /sys/kernel/config/nvmet/subsystems/nqn.2025-01.com.example:tenant-foo-01
cd /sys/kernel/config/nvmet/subsystems/nqn.2025-01.com.example:tenant-foo-01

# Allow any host to connect (for testing; use allowed_hosts for production)
echo 1 > attr_allow_any_host

# Add a namespace (maps a block device or file to a namespace ID)
mkdir namespaces/1
echo /dev/nvme0n1 > namespaces/1/device_path
echo 1 > namespaces/1/enable

mkdir namespaces/2
echo /dev/nvme1n1 > namespaces/2/device_path
echo 1 > namespaces/2/enable

# Create a port and link the subsystem
mkdir /sys/kernel/config/nvmet/ports/1
echo 10.0.10.1 > /sys/kernel/config/nvmet/ports/1/addr_traddr  # storage IP
echo rdma       > /sys/kernel/config/nvmet/ports/1/addr_trtype
echo 4420        > /sys/kernel/config/nvmet/ports/1/addr_trsvcid

ln -s /sys/kernel/config/nvmet/subsystems/nqn.2025-01.com.example:tenant-foo-01 \
      /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2025-01.com.example:tenant-foo-01

For persistence across reboots, use nvmetcli to dump and restore the configuration:

# Install nvmetcli (not to be confused with nvme-cli)
apt install nvmetcli   # Ubuntu
nvmetcli save /etc/nvmet/nvmet-config.json
# Restore at boot: nvmetcli restore /etc/nvmet/nvmet-config.json

What kernel nvmet gives: simple configuration, no userspace daemon, direct kernel path. Well-understood for development and small-scale production. Bottleneck at high fan-out (many initiators): the host CPU handles command scheduling when there is no hardware offload.

SPDK target

SPDK (Storage Performance Development Kit) runs in userspace with DPDK, bypassing the kernel storage stack entirely. It polls the NVMe device queues (no interrupts) and manages NVMe-oF command processing with zero kernel involvement.

# Configure SPDK NVMe-oF target (simplified)
# Bind NVMe devices to vfio-pci for SPDK ownership
./scripts/setup.sh   # SPDK scripts rebind devices to vfio-pci

# Start the SPDK target daemon
./build/bin/nvmf_tgt -m 0x00FF   # pin to cores 0-7

# Configure via RPC (in a separate shell)
./scripts/rpc.py nvmf_create_transport -t RDMA -u 131072
./scripts/rpc.py bdev_nvme_attach_controller -b NVMe0 -t pcie -a 0000:01:00.0
./scripts/rpc.py nvmf_create_subsystem nqn.2025-01.com.example:tenant-foo-01 -a
./scripts/rpc.py nvmf_subsystem_add_ns nqn.2025-01.com.example:tenant-foo-01 NVMe0n1
./scripts/rpc.py nvmf_subsystem_add_listener nqn.2025-01.com.example:tenant-foo-01 \
    -t rdma -a 10.0.10.1 -s 4420

What SPDK gains over kernel nvmet: significantly higher IOPS at the same CPU count (polling vs interrupt), lower latency at low queue depths, and the ability to drive NVMe drives at close to their rated IOPS. For high-performance production NVMe-oF targets, SPDK is typically 2–4× higher throughput per CPU core than kernel nvmet. The tradeoff: operationally more complex, requires dedicated CPU cores for polling, and requires the drives to be unbound from the kernel.

ANA groups and HA topology

ANA (Asymmetric Namespace Access) is the NVMe-oF mechanism for HA and multipath. Each namespace has access states on different controllers (optimized, non-optimized, inaccessible). The initiator's multipath driver uses ANA state to route IO.

ANA HA TOPOLOGY (2 target hosts, dual-port each)

  Target-A (stor-01)                    Target-B (stor-02)
  ┌────────────────────┐                ┌────────────────────┐
  │ CX-7 port 0        │                │ CX-7 port 0        │
  │ 10.0.10.1:4420     │                │ 10.0.10.3:4420     │
  │                    │                │                    │
  │ CX-7 port 1        │                │ CX-7 port 1        │
  │ 10.0.10.2:4420     │                │ 10.0.10.4:4420     │
  │                    │                │                    │
  │ NVMe[0..15]        │                │ NVMe[0..15]        │
  │ (ns 1-16)          │                │ (ns 17-32)         │
  └─────────┬──────────┘                └────────┬───────────┘
            │  ANA state: optimized               │  ANA state: non-optimized
            │  for ns 1-16 on Target-A            │  for ns 1-16 on Target-B
            └────────────────────────────────────┘
                    Initiator sees 4 paths per namespace
                    NVMe Native Multipath routes to optimized path

Configuration for ANA in nvmet:

# On Target-A: set ANA state for namespace 1 as optimized on this controller
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.2025-01.com.example:tenant-foo-01/\
          namespaces/1/ana_grpid   # assign to ANA group 1

# ANA groups are defined per-port and per-subsystem
# Group 1 = optimized on Target-A, non-optimized on Target-B
# Group 2 = optimized on Target-B, non-optimized on Target-A

What ANA HA survives and what it does not:

Survives:

• Single port failure on a target host (remaining port continues serving)
• Target host reboot with ctrl_loss_tmo long enough for ANA failover (see §7)
• NIC failure on one path

Does not survive:

• Both hosts rebooting simultaneously
• Drive failure (NVMe-oF HA is network HA; drive-level durability requires RAID or erasure coding on the target)
• Network partition that isolates the initiator from both targets

For drive-level durability, configure RAID-1 or RAID-5 on the target before presenting the namespace. A raw NVMe-oF namespace without drive-level protection means a single drive failure loses the namespace permanently.

Client (initiator) setup and tuning

nvme connect examples

# TCP transport
nvme connect -t tcp -n nqn.2025-01.com.example:tenant-foo-01 \
  -a 10.0.10.1 -s 4420

# RoCE v2 / RDMA transport
nvme connect -t rdma -n nqn.2025-01.com.example:tenant-foo-01 \
  -a 10.0.10.1 -s 4420

# InfiniBand transport (same rdma keyword; IB is auto-detected)
nvme connect -t rdma -n nqn.2025-01.com.example:tenant-foo-01 \
  -a 10.0.10.1 -s 4420

# Connect to all targets in a discovery log (nvme-cli ≥ 2.6)
nvme connect-all -t rdma -a 10.0.10.1 -s 8009

# With explicit queue count and keep-alive tuning
nvme connect -t rdma -n nqn.2025-01.com.example:tenant-foo-01 \
  -a 10.0.10.1 -s 4420 \
  -i 16 \          # nr_io_queues: 16 IO queues per controller
  -l 120 \         # ctrl_loss_tmo: 120 seconds before giving up on reconnect
  -k 60            # keep_alive_tmo: 60 seconds keepalive interval

Kernel, sysctl, and queue knobs

# IO scheduler: always none for NVMe (no mechanical seek; scheduler adds overhead)
echo none > /sys/block/nvme0n1/queue/scheduler

# Merge requests: disable for NVMe-oF (merges add latency, NVMe prefers queue depth)
echo 2 > /sys/block/nvme0n1/queue/nomerges

# Queue depth: NVMe-oF can sustain high queue depth; set nr_requests to match target's queue depth
echo 1024 > /sys/block/nvme0n1/queue/nr_requests

# For NVMe/TCP — reduce TCP polling latency
sysctl -w net.core.busy_read=50    # microseconds to busy-poll on receive
sysctl -w net.core.busy_poll=50    # microseconds to busy-poll on send
sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728
sysctl -w net.core.rmem_default=134217728
sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"

# MTU for RoCE v2 (jumbo frames required for RDMA efficiency)
ip link set <storage-nic> mtu 9216
# 9216 is preferred over 9000 for RoCE v2 to align with 9000-byte RDMA payload + headers

# NIC coalescing for TCP (reduce interrupt rate, increase throughput at modest latency cost)
ethtool -C <nic> rx-usecs 50 tx-usecs 50

# For RDMA: disable coalescing for latency-sensitive workloads
ethtool -C <storage-rdma-nic> rx-usecs 0 tx-usecs 0 adaptive-rx off adaptive-tx off

nr_io_queues tuning

Each NVMe-oF controller connection can have multiple IO queues. The optimal value depends on the initiator's CPU count and the expected IO concurrency:

Rule of thumb: nr_io_queues = min(cpu_count_on_numa_node, 32)

For a 2-socket host with 32 cores per NUMA node and 1 storage NIC on NUMA 0:
  nr_io_queues = 16–32 (use 16 as a safe starting point)

At queue depth 1 (latency-focused): even 1 queue is fine; the bottleneck is NVMe latency
At queue depth 128 (bandwidth-focused): use nr_io_queues >= 8 to spread across CPU cores

PCIe lane budget for the initiator

Each ConnectX-7 NDR requires PCIe Gen5 x16 (same requirement as compute NICs). This matters for GPU servers acting as NVMe-oF initiators: the server needs a spare PCIe slot for the storage NIC. In DGX H100 servers, a 9th NIC slot is available. In some OEM HGX designs with 8 NIC slots, the storage NIC competes with compute NIC slots.

See NIC sizing per GPU for the full PCIe lane budget discussion. The relevant constraint: do not put the storage NIC and a compute NIC in the same PCIe root complex unless PCIe bandwidth is verified to be non-bottlenecking for both simultaneously.

NUMA pinning for initiator queues

NVMe-oF IO queues should be scheduled on CPU cores that are NUMA-local to the storage NIC. When IO queues are scheduled on a remote NUMA node, every IO completion crosses a QPI/UPI hop, adding ~80–200 ns and consuming cross-NUMA memory bandwidth.

# Find NUMA node of the storage NIC
cat /sys/bus/pci/devices/<nic-pci-address>/numa_node
# Example: 0 (NUMA node 0)

# Pin IO submission to NUMA-local cores using taskset or cpuset
# For training jobs that read from NVMe-oF, ensure DataLoader workers run on NUMA 0:
numactl --cpunodebind=0 --membind=0 python3 train.py

# Verify IO queue CPU affinity (for connected namespace /dev/nvme0n1):
cat /sys/class/block/nvme0n1/device/queue_count
# Check which CPUs are handling queue completions:
cat /proc/interrupts | grep nvme

Recommended for GPU workloads? — direct, opinionated answer

Direct GDR over NVMe-oF (GPUDirect Storage to remote NVMe-oF target)

Yes, when the storage tier must be remote and the workload is read-dominated weight loading or short checkpoint reads.

GDS (GPUDirect Storage) with NVMe-oF enables the data path: GPU HBM ← NIC ← wire ← target NIC ← NVMe drive, with no CPU copy. The latency budget:

GPU to remote NVMe-oF latency (GDS path, RoCE v2, ConnectX-7 NDR, Gen5 NVMe):

  NVMe drive local read:       ~60–80 µs
  Target NIC (CX-7 offload):   ~5–15 µs
  RoCE v2 wire:                ~2–5 µs (NDR fabric)
  Initiator NIC (CX-7):        ~5–15 µs
  GDS DMA to GPU HBM:          ~10–30 µs (PCIe Gen5)
  Total:                       ~80–145 µs

Weka tier-1 (same drives, same fabric, for comparison):
  CephFS/Weka read p50:        ~100–200 µs (parallel FS metadata + IO path)
  CephFS/Weka read p99:        ~500 µs–1 ms (under multi-tenant concurrency)

For a single-tenant single-stream read — loading a 140 GB model from a dedicated namespace at job start — NVMe-oF with GDS is competitive with or faster than Weka tier-1, with simpler semantics (no metadata server involved, no filesystem FUSE or POSIX layer in the path).

For many concurrent tenants reading shared datasets, Weka tier-1 wins on aggregate read bandwidth because the parallel-FS metadata layer scales horizontally. A 6-host Weka cluster delivers 100+ GB/s aggregate to many clients. Six NVMe-oF targets each with 16 drives would need client-side sharding logic to match that, and you would have implemented a simplified parallel filesystem.

NVMe-oF for training datasets

Usually no.

Training jobs read shuffled samples from shared corpora with high concurrency from many nodes. That is exactly the workload that Weka, Ceph parallel pools, and pNFS were designed for. Going to raw NVMe-oF for this purpose means rolling your own:

• Client-side sharding (which drive holds which shard?)
• Access pattern balancing across namespaces
• Quota enforcement
• Per-tenant namespace isolation

Worth attempting only if: single dominant tenant on dedicated hardware, and the p99 latency of the parallel filesystem is measurable in epoch startup time. In that narrow case, dedicated NVMe-oF namespaces per training job can eliminate the metadata latency that parallel-FS clients pay.

For multi-tenant production training: use Weka or Ceph.

NVMe-oF for checkpoint write

Situational.

A single large write to a single namespace at end of a training step is well-matched to NVMe-oF. The operation is sequential, large, and does not require distributed metadata. If the target has adequate drive endurance and the write bandwidth matches the checkpoint burst rate, NVMe-oF checkpoint works.

Weka tier-1 with its erasure-coded write path is more durable (data is protected across multiple hosts and drives from the first byte written) and self-balancing across the cluster. Use NVMe-oF checkpoint only when:

• Regulatory data placement requires the checkpoint to land on specific hardware.
• Single-tenant burst and the NVMe-oF target is dedicated to that tenant's checkpoint stream.
• You have explicit RAID or erasure coding on the target drives (raw NVMe without this is a single-drive-failure-from-lost-checkpoint configuration).

NVMe-oF for model serving / inference

Yes, often the best answer for per-replica weight loading.

Loading a large model's weights from a remote NVMe-oF namespace at startup is an excellent NVMe-oF pattern:

• Read-only for the duration of the replica's lifetime.
• Single-stream-per-replica (each inference server loads its own copy).
• Latency-sensitive — the replica starts serving requests only after the weights are loaded.
• No distributed writes, no metadata updates, no concurrency across the namespace.

The replica connects at startup with nvme connect, then opens the weights file and mmap-reads it into GPU memory via GDS. When the replica is replaced (rolling deployment, model update), it disconnects and a new replica connects and loads the updated namespace.

For large-fleet inference (many replicas loading the same model version simultaneously), the aggregate bandwidth can exceed a single NVMe-oF target's capability. In that case, either distribute across multiple namespaces/targets or use an object-backed parallel path (Weka tier-1, FlashBlade S3 via GDS).

NVMe-oF for VAST or similar appliance backends

VAST Data's system exposes NVMe-oF over its DBox enclosures. Treat as a managed NVMe-oF solution: the operator surface is primarily target-side configuration (VAST element manager) and the recommendations in this document apply to the initiator-side (client config, nr_io_queues, NUMA pinning, keep-alive timeouts). The VAST-specific distinction: VAST implements its own erasure coding and metadata at the appliance layer, so the raw-NVMe durability concerns in this document are handled for you.

Tuning the driver — concrete settings

nvme_rdma module parameters

# Load nvme_rdma with parameters (or set in /etc/modprobe.d/nvme_rdma.conf)
modprobe nvme_rdma

# Verify current parameters
cat /sys/module/nvme_rdma/parameters/register_always
# register_always=0: MR registration is cached (default, better for steady-state)
# register_always=1: re-register MR on each IO (avoids stale MR issues, higher overhead)
# Recommendation: leave at 0 (default) unless troubleshooting MR errors

# srq_count: shared receive queue count per device
# Default is typically 0 (disabled); enabling reduces per-QP RX buffer overhead
# echo 256 > /sys/module/nvme_rdma/parameters/srq_count
# Recommendation: leave at default unless running very high QP counts (>256 subsystems)

nvme_tcp module parameters

# io_work_init_delay: delay (in jiffies) before starting IO work
# Default: 1; set to 0 for lower latency at cost of slightly higher CPU wakeup rate
echo 0 > /sys/module/nvme_tcp/parameters/io_work_init_delay

# recv_pdu_first: process receive PDU before other work
# Default: 1 (enabled); keep enabled for latency
cat /sys/module/nvme_tcp/parameters/recv_pdu_first

# NVMe/TCP polling mode (requires kernel 5.18+ for poll queue support)
# Connect with poll queues for CPU-bound latency-critical workloads:
nvme connect -t tcp -n <nqn> -a <addr> -s 4420 --nr-poll-queues 4

Sysctl reference block

# Network buffers (TCP transport)
sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728
sysctl -w net.core.rmem_default=134217728
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"
sysctl -w net.ipv4.tcp_congestion_control=bbr   # BBR better for NVMe/TCP throughput

# TCP busy-polling (reduces interrupt latency for NVMe/TCP)
sysctl -w net.core.busy_read=50
sysctl -w net.core.busy_poll=50

# Persist in /etc/sysctl.d/99-nvmeof.conf

ethtool NIC settings

# For NVMe/TCP: moderate coalescing (balance throughput and latency)
ethtool -C <nic> rx-usecs 50 tx-usecs 50 adaptive-rx off adaptive-tx off

# For NVMe/RoCE v2: zero coalescing (RDMA latency-sensitive)
ethtool -C <rdma-nic> rx-usecs 0 tx-usecs 0 adaptive-rx off adaptive-tx off

# Ring buffer sizes (important: small rings cause drops at high BW)
ethtool -G <nic> rx 8192 tx 8192

# Channel count: match to NVMe queue count
# If nr_io_queues = 16, set 16 combined channels
ethtool -L <nic> combined 16

# Verify current settings
ethtool -c <nic>   # coalescing
ethtool -g <nic>   # ring sizes
ethtool -l <nic>   # channels

Target-side settings (nvmet)

# Target queue depth per subsystem: set in nvmet configuration
# Default is 128; increase for high fan-out (many initiators)
echo 256 > /sys/kernel/config/nvmet/subsystems/<nqn>/attr_qid_max

# TCP-only: kernel poll delay (kpoll_delay, in microseconds)
# Lower values reduce latency; increase if CPU overhead is too high
# Set via configfs if exposed (kernel version dependent):
# echo 100 > /sys/kernel/config/nvmet/ports/<port>/param_kpoll_delay

# ANA group count: one per active/standby pair
mkdir /sys/kernel/config/nvmet/subsystems/<nqn>/ana_groups/1
mkdir /sys/kernel/config/nvmet/subsystems/<nqn>/ana_groups/2

Known-good baseline tuned profile plus explicit settings

# Step 1: apply network-latency tuned profile as the baseline
tuned-adm profile network-latency

# Step 2: overlay NVMe-oF specific settings (these are not in the tuned profile)
# Place in /etc/tuned/nvmeof-overlay/tuned.conf or apply manually after profile

# Sysctl overrides
sysctl -w net.core.busy_read=50
sysctl -w net.core.busy_poll=50
sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728

# IO scheduler for all NVMe-oF namespaces
for dev in /sys/block/nvme*; do echo none > ${dev}/queue/scheduler; done

# IRQ affinity (after tuned-adm, re-pin NIC IRQs to NUMA-local cores)
for irq in $(grep mlx5 /proc/interrupts | awk '{print $1}' | tr -d ':'); do
  echo ff > /proc/irq/${irq}/smp_affinity
done

# ethtool coalescing (RDMA path)
ethtool -C <rdma-nic> rx-usecs 0 tx-usecs 0 adaptive-rx off adaptive-tx off
ethtool -G <rdma-nic> rx 8192 tx 8192
ethtool -L <rdma-nic> combined 16

# MTU
ip link set <rdma-nic> mtu 9216

Verification and benchmarking

Connectivity verification

# List connected NVMe-oF subsystems
nvme list-subsys
# Example output shows subsystem NQN, transport, address, state

# List all NVMe namespaces (local and remote NVMe-oF)
nvme list
# Remote NVMe-oF namespaces appear alongside local NVMe drives

# For NVMe Native Multipath: check ANA state of each namespace path
nvme list-subsys -o json | python3 -m json.tool | grep -E '"ana_state"|"transport"'
# Expected: at least one path in "optimized" state per namespace

# For DM-Multipath (if used instead of NVMe Native Multipath)
multipath -ll
# Should show NVMe-oF paths under the multipath device map

fio benchmark recipes

# Latency floor: 4 KB random read, queue depth 1
fio --name=nvmeof-lat \
    --filename=/dev/nvme0n1 \
    --ioengine=io_uring \
    --iodepth=1 \
    --bs=4k \
    --rw=randread \
    --numjobs=1 \
    --runtime=60 \
    --time_based \
    --group_reporting

# Expected (NVMe/RoCE v2, CX-7 NDR, Gen5 PM1743, queue depth 1):
#   lat (usec): min=55, avg=70-90, max=200
#   IOPS: ~10,000-14,000 (limited by QD=1 latency)

# Bandwidth ceiling: 1 MB sequential read, queue depth 128
fio --name=nvmeof-bw \
    --filename=/dev/nvme0n1 \
    --ioengine=io_uring \
    --iodepth=128 \
    --bs=1m \
    --rw=read \
    --numjobs=4 \
    --runtime=60 \
    --time_based \
    --group_reporting

# Expected (NVMe/RoCE v2, CX-7 NDR, Gen5 PM1743, QD=128):
#   bw: 10,000-12,000 MiB/s (approaching Gen5 NVMe sequential limit of ~12-13 GB/s)
#   Note: this is per-namespace; aggregate across multiple namespaces scales linearly

When the numbers are off by 2x — diagnostic flow

Symptom: latency 2× expected.

1. Check NIC ring size: ethtool -g <nic> — if RX: 256 (too small), increase to 8192.
2. Check NUMA: is the fio process on the same NUMA node as the NIC? numactl --hardware, compare with cat /sys/bus/pci/devices/<nic>/numa_node.
3. Check IRQ pinning: watch -n1 'cat /proc/interrupts | grep mlx5' — verify IRQs are not bouncing across NUMA nodes.

Symptom: bandwidth 2× below expected.

1. Check link rate: ethtool <nic> | grep Speed — should be 400000Mb/s for NDR. If lower, check cable, transceiver, and switch port negotiation.
2. Check PCIe speed: lspci -vvv -s <nic-pci-addr> | grep LnkSta — should be Gen5 x16. Gen4 x16 is adequate for NDR; Gen4 x8 will bottleneck.
3. Check PCIe ASPM: Active State Power Management can reduce PCIe link speed under low load. Disable for storage paths: echo performance > /sys/bus/pci/devices/<nic>/power/control.
4. Check drive-side wear: nvme smart-log /dev/nvme0n1 | grep percentage_used — drives at high wear may throttle sustained write bandwidth. Less common for read-heavy benchmarks.
5. Check multipath misconfiguration: if the namespace has 4 paths but 2 are in non-optimized state and traffic is not preferring optimized, bandwidth is split across the wrong paths. Verify with nvme list-subsys -o json.

Operations and gotchas

Live cable yank

What NVMe-oF does: if a cable is yanked on the initiator side, the NVMe-oF controller enters a reconnect loop. The ctrl_loss_tmo parameter controls how long the initiator waits before returning errors to the application. Default ctrl_loss_tmo is 600 seconds (10 minutes) — the initiator will hang IO for up to 10 minutes waiting for reconnect.

For GPU training jobs, a 10-minute IO hang is a failed job. Set ctrl_loss_tmo to a value aligned with the job scheduler's health-check timeout:

# Reduce ctrl_loss_tmo to 120 seconds (adjust to match Slurm HealthCheckInterval)
nvme connect ... --ctrl-loss-tmo 120

If multipath is configured and a second path is available, ANA failover will redirect IO to the surviving path within seconds. Cable yanks on one path without multipath are much more disruptive.

Target reboot during a job

With ANA failover:

• Initiator detects path failure (keepalive timeout, default 60 seconds).
• IO is redirected to the non-optimized (surviving) path on the other target.
• Latency increases briefly during failover (IO queues drain, re-target).
• After target reboot, the path reconnects and ANA re-optimizes.
• Total impact: 60–120 seconds of degraded performance; job continues.

Without multipath:

• Initiator waits for ctrl_loss_tmo, then returns errors.
• Job fails.

Always deploy multipath for production NVMe-oF.

Multipath misconfiguration that silently halves bandwidth

A common operator mistake: both paths in a multipath configuration are active, but both are routing IO through the same physical NIC port on the target side due to misconfigured ANA groups or duplicate port addresses. The result: apparent throughput is half of expected because the "two paths" are actually the same physical path with double the CPU overhead.

Diagnose:

# Check that paths use different physical ports (different IP addresses)
nvme list-subsys -o json | grep traddr
# Should show: 10.0.10.1 and 10.0.10.2 (different IPs), not duplicates

# Check ANA state per path
nvme list-subsys -o json | grep -E "traddr|ana_state"
# At least one path per namespace should be "optimized"; the rest "non-optimized"

Sharing a fabric between RoCE storage and RoCE compute (NCCL)

Both NCCL allreduce traffic and NVMe-oF storage traffic can run on the same RoCE v2 fabric, but they require separate DCB priority class assignment to avoid head-of-line blocking.

Priority class layout (example):
  PCP 3 → DSCP 26 → NVMe-oF storage traffic (mapped to PFC priority 3)
  PCP 4 → DSCP 34 → NCCL compute traffic (mapped to PFC priority 4)

  Switch configuration: separate egress queues per priority class
  PFC enabled on both queues: each queue has independent pause/resume

  NIC ethtool (storage NIC, RoCE for storage):
  mlnx_qos -i <storage-nic> --pfc 0,0,0,1,0,0,0,0   # PFC on priority 3 only
  tc qdisc and ip rule to mark storage NVMe-oF traffic with DSCP 26

Failure to separate: checkpoint bursts (all nodes writing simultaneously) generate large RoCE frames that fill switch egress queues, triggering PFC on the NCCL queue as well. NCCL allreduce operations pause. GPU compute stalls. This is observable as sudden training throughput drops at checkpoint boundaries.

Security: TLS and authentication

NVMe/TCP: TLS 1.3 for in-transit encryption is available since kernel 6.7. Enable with:

# Connect with TLS (requires key exchange setup; see nvme-cli ≥ 2.6 tls support)
nvme connect -t tcp -n <nqn> -a <addr> -s 4420 --tls

The pre-shared key (PSK) handshake mechanism (keyring:nvme-psk) is the standard path. TLS for NVMe/TCP is not yet universally deployed but is the right direction for multi-tenant storage.

NVMe/RoCE v2: RoCEv2 has no native per-connection authentication mechanism. Rely on network isolation: put NVMe-oF storage traffic in a dedicated VLAN or VRF, separate from tenant compute VLANs. IP-based allow-lists in nvmet (via allowed_hosts per subsystem NQN) provide coarse-grained access control.

Practical recommendation: for a GPU cluster where the storage VLAN is restricted to known node IPs and the management VLAN is separate, RoCEv2 without TLS is an acceptable risk. For any multi-tenant environment where a compromised compute node could sniff storage traffic, require TLS on NVMe/TCP or enforce strict VLAN segmentation with per-subsystem NQN allow-lists.

When to skip NVMe-oF entirely

Tiny clusters (fewer than 8 nodes)

A single ZFS-on-Linux NFS server with 4–8 NVMe drives delivers 10–20 GB/s at a fraction of the configuration overhead. For clusters where the aggregate read bandwidth requirement is under 15 GB/s and there is no latency-sensitive single-stream workload, NFS is simpler, better-understood, and operationally less brittle. See NFS for GPU clusters for sizing guidance.

Weka or VAST already sized for the workload

If the cluster already has a Weka or VAST deployment sized to the training workload, adding a parallel NVMe-oF path creates a second storage system to operate. Weka tier-1 delivers competitive latency (100–200 µs p50) for most GPU workloads. VAST's DBox NVMe-oF is already managed. The incremental benefit of adding raw NVMe-oF on top of a working parallel filesystem is rarely worth the added operational surface area.

Exception: if Weka or VAST is sized for the training data read path but a specific tenant needs a dedicated low-latency checkpoint namespace with defined drive exclusivity, adding a targeted NVMe-oF namespace for that checkpoint stream is a reasonable design.

No RDMA NICs and no budget

NVMe/TCP works but the latency win over NFS-over-RDMA or Weka is small when the fabric is Ethernet-only with no RDMA. For bandwidth-focused workloads (training dataset reads), NVMe/TCP's CPU overhead and higher latency eat into the advantage. If the cluster has no ConnectX-6 Dx or newer NICs and there is no budget to add them, NFS-over-RDMA is simpler than NVMe/TCP for most training workloads.

Single-tenant ML on local NVMe

Direct-attached NVMe is faster than NVMe-oF by definition — no wire latency, no NIC overhead, no protocol stack. For a single-node job (large fine-tuning run on a single 8-GPU server with local NVMe), local storage is simpler and faster. NVMe-oF is a network storage protocol; it only makes sense when the storage needs to be remote.

Summary: skip NVMe-oF when