nvidia-peermem: GPUDirect RDMA, the silent perf killer when missing

GPUDirect RDMA (GDR) is what makes multi-node GPU training scale. Without it, a tensor leaving GPU 0 on node A heading to GPU 0 on node B has to: copy from GPU VRAM into host RAM (PCIe up), DMA from host RAM out the NIC (PCIe down), receive on the remote NIC into host RAM (PCIe up), and finally copy into the remote GPU's VRAM (PCIe down). Four hops, four PCIe traversals, plus CPU involvement and host memory pressure.

With GDR, the NIC DMAs directly into GPU VRAM and back. Two hops, no CPU on the data path. Empirically: GDR off vs on is the difference between ~25 GB/s and ~180 GB/s per NIC on a healthy NDR/HDR400 IB cluster, ~5-10× speedup in NCCL all-reduce wall time, with no error message either way. The job will run; it will just be uselessly slow.

nvidia-peermem is the kernel glue that makes GDR work. This page explains what it does, the load-order trap, how to verify it's actually doing its job, and the silent-failure mode you'll spend a day debugging if you don't know about it.

What problem the module solves

The InfiniBand verbs API requires that any memory region the NIC reads/writes from is registered — pinned in physical memory and known to the NIC's address translation cache. Verbs ibv_reg_mr() works for system RAM out of the box; the kernel pins the pages and gives the NIC a virtual-to-physical map.

GPU VRAM is not host memory. The NIC has no way to know where GPU pages are physically located, and they don't exist in the host's pagetables at all. We need a kernel-side bridge that:

1. Lets userspace pass GPU memory pointers to ibv_reg_mr().
2. Asks the NVIDIA driver "where is the physical PCIe address of these GPU pages?".
3. Hands those addresses to the IB stack so the NIC HCA can DMA against them.
4. Holds page references so the GPU doesn't move/free them while the NIC has outstanding DMAs.

That bridge is nvidia-peermem. It registers a peer_memory_client callback into the IB core (ib_register_peer_memory_client()) and the IB stack uses it whenever a memory region register call references a non-host pointer.

The history: nv_peer_mem → nvidia-peermem

For years (~2015-2021) the canonical module was nv_peer_mem, a separate out-of-tree module shipped by Mellanox/NVIDIA in MLNX_OFED. It was DKMS-built, version-coupled to OFED, and a reliable source of confusion.

Starting with CUDA 11.4 / driver 470, NVIDIA replaced it with nvidia-peermem, which is bundled with the NVIDIA driver itself. The new module:

• Ships with nvidia-driver-XXX — no extra package to install.
• Is version-coupled to the driver, not OFED — fixes a whole class of "driver upgrade broke RDMA" bugs.
• Lives in /lib/modules/$(uname -r)/updates/dkms/nvidia-peermem.ko (DKMS-built) or /lib/modules/$(uname -r)/kernel/drivers/video/nvidia-peermem.ko (precompiled).

If you find a node with both modules installed, you have an upgrade path mistake — nv_peer_mem should be uninstalled, and the new path goes through nvidia-peermem. Two peer memory clients trying to register against the same IB stack causes confusion and at least one will fail.

Load order — the trap

Here's the trap that catches most operators on a fresh image:

nvidia-peermem requires both the NVIDIA driver (nvidia.ko) and the IB core (ib_core) to be loaded before it can register itself. If nvidia-peermem is loaded first (or auto-loaded by kmod from a stale modules.dep), the registration silently fails and the module sits there doing nothing.

The default systemd systemd-modules-load.service loads modules in alphabetical order from /etc/modules-load.d/*.conf. There is no dependency declaration — order is whatever the filesystem returns, which on most distros is name-sorted. So i*.conf (IB stuff) comes before n*.conf (nvidia stuff) — which means MLNX_OFED's mlnx_ofed.conf loads first, but if you have a separate nvidia-peermem.conf, that may load before nvidia.ko is up if the latter is auto-loaded by udev based on PCI device probe order.

In practice, on a healthy node you want this layering:

1. mlx5_core             (loaded by udev when Mellanox NIC PCI device probes)
2. mlx5_ib               (depends on mlx5_core)
3. ib_core               (auto-pulled in by mlx5_ib)
4. nvidia                (loaded by udev when NVIDIA GPU PCI device probes)
5. nvidia_uvm            (loaded on first CUDA call)
6. nvidia_peermem        (loaded explicitly via /etc/modules-load.d after the above)

The robust pattern is to not rely on auto-loading peermem at all. Instead, list it in /etc/modules-load.d/nvidia-peermem.conf so systemd loads it after mlx5_* and nvidia are already up:

$ cat /etc/modules-load.d/nvidia-peermem.conf
nvidia_peermem

For complete OFED hardening, the canonical 9-module list goes in a separate file — see OFED.

DKMS rebuild on kernel upgrade

When you bump the kernel, both nvidia.ko and nvidia-peermem.ko need to be rebuilt against the new headers. DKMS handles this automatically if linux-headers-$(target-kernel) is installed. Verify before reboot:

$ dkms status
nvidia/570.124.06, 6.8.0-45-generic, x86_64: installed
nvidia/570.124.06, 6.8.0-46-generic, x86_64: installed   # newly built ✓

If only the running kernel shows up but not the one you're about to boot into, peermem will not load on next boot. Run:

$ sudo apt install linux-headers-$(ls /boot/vmlinuz-* | tail -1 | sed 's|.*vmlinuz-||')
$ sudo dkms autoinstall

How to verify peermem is doing its job

There are four levels of verification, from cheapest to most authoritative.

Level 1: module loaded?

$ lsmod | grep peermem
nvidia_peermem         16384  0
nvidia                 56012800 1 nvidia_peermem
ib_core                573440 12 rdma_cm,ib_ipoib,...,nvidia_peermem

Crucially ib_core should have nvidia_peermem in its dependents list (the right-most column on ib_core's row). If nvidia_peermem shows in lsmod but is not listed as a dependent of ib_core, the registration failed. This is subtle and easy to miss — re-read the output above carefully.

Level 2: peer_mem registration sysfs

$ ls /sys/kernel/mm/memory_peers/
nv_mem
$ cat /sys/kernel/mm/memory_peers/nv_mem/version
1.0
$ cat /sys/kernel/mm/memory_peers/nv_mem/num_alloc_mrs
0          # number of currently-registered GDR regions
$ cat /sys/kernel/mm/memory_peers/nv_mem/num_dereg_mrs
0

The presence of nv_mem in /sys/kernel/mm/memory_peers/ is the strongest in-kernel signal that registration succeeded. If this directory is missing, peermem is not registered, period.

(Older nv_peer_mem showed /proc/driver/nv_peer_mem/clients instead — that path is dead on new builds.)

Level 3: dmesg

$ dmesg | grep -iE "peermem|gpu.?direct|peer.mem"
[   28.451233] nvidia-peermem nvidia_peermem: nvidia-peermem 570.124.06 loaded
[   28.452991] nvidia-peermem nvidia_peermem: registered as peer memory client to ib_core

Two messages, both informational. Errors here ("failed to register peer memory client", "ib_register_peer_memory_client returned -EEXIST") indicate either a load-order problem or an nv_peer_mem collision.

Level 4: end-to-end NCCL test

# Multi-node all-reduce, with GDR debug on — the only authoritative check
$ NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=NET,GRAPH \
  NCCL_IB_GID_INDEX=3 \
  /opt/nccl-tests/build/all_reduce_perf -b 1G -e 1G -f 1 -g 8 2>&1 | grep -i gdr

  NCCL INFO Using network IB
  NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB ... GDR-direct
  NCCL INFO Channel 00 : 0[1b000] -> 8[1b000] [send] via NET/IB/0/GDRDMA
  NCCL INFO Channel 00 : 8[1b000] -> 0[1b000] [recv] via NET/IB/0/GDRDMA

Look for GDR-direct (NCCL detected GDR support) and GDRDMA in the channel logs (NCCL is actually using it for this transfer). If you see via NET/IB/0/0 instead of GDRDMA, NCCL fell back to host-staging. The result will be 1/5th to 1/10th of the throughput.

If the all_reduce_perf bus bandwidth at 1 GiB looks like ~25 GB/s instead of ~180 GB/s on healthy NDR fabric, GDR is off. Period.

The silent failure: how it usually breaks

Imagine a node where peermem is missing. NCCL prints these informational lines:

NCCL INFO Using network IB
NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ...
NCCL INFO Channel 00 : 0[1b000] -> 8[1b000] [send] via NET/IB/0/0

There is no error. There is no warning. The job runs. Loss curves look normal. The only signal is that throughput is ~5-10× too low — and you only know that's "too low" if you have a healthy reference cluster to compare against.

In production this typically surfaces as:

• "Why is our 256-GPU H100 cluster getting only 1.2× the throughput of our 32-GPU baseline?"
• "Training takes 11 days instead of 2."
• "Customer reports the cluster is slow but everything looks healthy."

Always run an all_reduce_perf baseline at 1 GiB messages on every new node and bake it into your acceptance test. If busbw is more than ~10% below the cluster norm, suspect peermem first.

Common failure modes

Peermem missing because OFED packaging clashed

MLNX_OFED installer asks "do you want to enable GPUDirect support?" — if you said yes pre-CUDA 11.4, it installed nv_peer_mem. Post-CUDA 11.4 you should say no and let the NVIDIA driver provide nvidia-peermem. Mixed installs leave both lying around. Fix:

sudo apt-get purge nv-peer-memory nv-peer-memory-dkms
# or, on RHEL:
sudo yum remove nvidia-peer-memory

sudo apt-get install --reinstall nvidia-driver-570    # ensures nvidia-peermem.ko is in place
sudo modprobe -r nv_peer_mem 2>/dev/null
sudo modprobe nvidia_peermem

modprobe nvidia_peermem returns "ERROR: could not insert: Invalid argument"

This is symbol-version mismatch. Two common cases:

1. In-tree vs DKMS clash on Ubuntu HWE kernel. The HWE kernel ships an in-tree IB stack that DKMS-built MLNX_OFED conflicts with. dmesg will show module: disagrees about version of symbol ib_register_peer_memory_client. Fix: rebuild OFED DKMS modules against the running kernel, or pin to a non-HWE kernel.
2. NVIDIA driver was upgraded but DKMS didn't rebuild peermem. dkms status should show peermem against the running kernel. If not, dkms install nvidia/<ver>.

In both cases the recovery is the same: rebuild against current kernel, reload nvidia first then nvidia_peermem.

nv_mem not in /sys/kernel/mm/memory_peers/ despite module loaded

Almost always load order. Reload in correct sequence:

sudo rmmod nvidia_peermem 2>/dev/null
# ensure ib_core and mlx5_ib are loaded
lsmod | grep -E '^ib_core|^mlx5_ib'
# reload nvidia first if needed
sudo modprobe nvidia
# then peermem
sudo modprobe nvidia_peermem
ls /sys/kernel/mm/memory_peers/

If you still don't see nv_mem, dmesg will tell you why.

NCCL says GDR not supported despite all module checks passing

Two more things to check:

1. ACS (Access Control Services) on the PCIe root complex. ACS isolates devices from each other and blocks peer-to-peer DMA. lspci -vvv | grep -A 4 "Access Control Services" — if ACSCtl is non-zero on the upstream port between GPU and NIC, P2P is blocked. See ACS / IOMMU.
2. NIC and GPU on different NUMA nodes / non-peer PCIe domains. GDR works only when the NIC HCA and GPU share a PCIe root or are connected through a PCIe switch that supports peer-to-peer. nvidia-smi topo -m shows the topology — anything SYS or NODE between GPU and NIC means GDR will not work for that pair.

See also

• NVIDIA driver stack — peermem ships in this package
• Mellanox OFED — the IB stack peermem registers into
• GPUDirect / GDR — protocol-level explanation
• NCCL multi-node — the workload that exercises peermem in anger
• ACS / IOMMU — the other thing that silently kills GDR