NVIDIA driver stack: kernel module, userspace, and the full picture

If you have ever stared at a server that booted fine, has 8 H100s plugged in, lights blinking — and nvidia-smi returns "Failed to initialize NVML: Driver/library version mismatch", this page is the why-and-how reference you wish you had open at that moment. The NVIDIA driver is not a single thing; it is at least five kernel modules and a stack of userspace libraries, all of which must agree on a version. Most production GPU pain comes from one of those pieces being out of step.

Why a "GPU driver" is actually five drivers

When you apt install nvidia-driver-550 (or run the .run installer, or let the GPU Operator do it), you are getting these five kernel modules plus userspace:

The first four come with the driver package and load automatically. nvidia_peermem is the one operators forget — it ships with the driver but is not auto-loaded, and silently missing it is the #1 cause of "training works but is 5x slower than it should be" (see nvidia-peermem).

Load order matters

# nominal order on a healthy node
$ lsmod | awk '/nvidia|mlx5|rdma|^ib_/{print $1}'
nvidia_peermem
nvidia_uvm
nvidia_drm
nvidia_modeset
nvidia
mlx5_ib
ib_uverbs
ib_core
mlx5_core

nvidia_peermem depends on both nvidia and ib_core being loaded first — if Mellanox OFED (or whatever provides ib_core) loads after nvidia, peermem cannot register itself with the IB stack and silently won't be functional. This is why on freshly imaged HGX nodes you will sometimes see lsmod | grep peermem return nothing even though the package is installed.

The fix is persistent module loading via /etc/modules-load.d/ so the order is deterministic across reboots. See the OFED page for the full canonical 9-module list.

DKMS vs precompiled vs Open Kernel Modules

There are three ways to get the kernel module on a node, and the choice has real operational consequences:

Precompiled (nvidia-driver-XXX from NVIDIA's apt/yum repo)

Ships a .ko built against a specific kernel ABI. Fast install, no build dependencies on the node.

Pro: deterministic, no compiler / kernel-headers needed at install time. Con: pinned to one kernel. If the host runs apt upgrade and gets a new kernel, the driver does not rebuild and the next reboot has no GPU. You'll see nvidia-smi fail with "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver".

DKMS (nvidia-dkms-XXX)

Ships source. DKMS rebuilds the module against whatever kernel you boot into.

Pro: survives kernel upgrades transparently. Con: needs gcc, make, linux-headers-$(uname -r) available at upgrade time. If headers aren't present (very common on minimal images / Talos / RKE2 air-gapped), DKMS fails silently in apt post-install and you find out at next reboot.

# verify DKMS health
$ dkms status
nvidia/550.144.03, 6.8.0-45-generic, x86_64: installed
nvidia/550.144.03, 6.8.0-46-generic, x86_64: installed

# If you see a kernel listed under "WARNING! Diff between built and installed module"
# the module is stale and needs `dkms autoinstall` before reboot.

Always run dkms status before rebooting a node after a kernel upgrade. Cheap insurance.

Open Kernel Modules (since 515; default since 560)

NVIDIA started shipping the kernel-side of the driver as open-source GPL/MIT-licensed in 2022. The userspace (libcuda, libnvidia-ml, etc.) is still proprietary. On Hopper (H100/H200) and Blackwell (B100/B200/B300), Open Kernel Modules are the recommended path — for Turing/Ampere they are still optional and the proprietary nvidia.ko works fine.

# tell which flavor you have
$ modinfo nvidia | grep -E "license|version"
license:        Dual MIT/GPL          # ← Open
version:        550.144.03

$ modinfo nvidia | grep -E "license|version"
license:        NVIDIA                # ← proprietary
version:        535.183.01

On 560+ on Hopper/Blackwell, the .run installer defaults to Open. If you specifically want proprietary you have to pass --kernel-module-type=proprietary. For Open you need:

• Kernel ≥ 5.4 (everywhere in practice).
• GSP firmware files in /lib/firmware/nvidia/<driver-version>/gsp_*.bin. The driver loads the GPU System Processor firmware on init; missing/corrupt GSP firmware = nvidia-smi returns garbage or errors. This is much more visible on Open since it relies more heavily on GSP for initialization.

Driver branches: TRD, Production, Long-Term Support

NVIDIA publishes three parallel driver lineages. Picking the wrong one is a slow-burn problem — your stack works for a while, then a CUDA upgrade breaks because the driver minimum was bumped.

In a real cluster:

• H100/H200 fleet, doing LLM training — Production Branch. 535 → 550 → 570 → 580 are all valid; pick whatever supports your CUDA toolkit minimum (see compat table below) and sit on it for a release cycle.
• B200/B300 fleet — minimum 550 for B100/B200; 580+ recommended for B200; 600 series for B300. NFB/TRD often required at launch since Production Branch lags arch by ~6 months.
• Old A100 fleet that nobody touches — 535 LTS, do not upgrade unless you have a reason.

Verify your branch

$ nvidia-smi --query-gpu=driver_version --format=csv,noheader
570.86.16

NVIDIA's release notes specify branch type: https://docs.nvidia.com/datacenter/tesla/drivers/index.html.

CUDA toolkit / cuDNN / NCCL compatibility

The version game has four layers:

your application           (PyTorch 2.4, JAX 0.4.x, TF 2.18, …)
      |
      v
NCCL                      (e.g. 2.21, 2.23) — bundled or system
cuDNN                     (e.g. 9.x for CUDA 12) — usually bundled in pip wheel
CUDA toolkit              (12.4, 12.6, 12.8, 13.0)
      |  ↑ minimum driver requirement
      v
NVIDIA driver             (535, 550, 570, 580, …)
      |
      v
GPU silicon               (Hopper / Blackwell)

Each CUDA major version requires a minimum driver version. Newer drivers are almost always backward-compatible with older CUDA toolkits (CUDA Forward Compatibility), so the safe rule is driver newer than or equal to the toolkit's minimum.

NCCL compatibility is mostly forward — NCCL 2.20 wheel will run on driver 535 against H100, but new collectives (e.g. NVLS for SHARP, FP8 reductions) need newer drivers. In practice: pin a single driver version across the fleet, then ship newer NCCL via container without touching the host.

Example pinning for a stable LLM training fleet (Hopper)

# concrete versions that play together as of 2025
host:
  kernel: 5.15.0-LTS or 6.8.x
  nvidia driver: 570.124.06   # PB
  fabric-manager: 570.124.06  # MUST match driver minor
  CUDA driver libs: 570.x

container (CUDA 12.6):
  cuda toolkit: 12.6.3
  cudnn: 9.5.x
  nccl: 2.23.x
  pytorch: 2.5.x

Cross-check NVIDIA's official table at https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html.

persistence mode — and why it matters for ephemeral pods

When the first process opens /dev/nvidia*, the driver does several seconds of init work: query EEPROM, train PCIe link, load GSP firmware, bring NVLink up, etc. When the last process closes it, by default the driver tears all that down. Next process pays the init cost again.

For interactive workstations this is invisible. For Kubernetes, where pods come and go on second timescales, every short-lived pod that touches the GPU pays a 2-10 second startup tax. Worse, between the last-close and the next-open, NVLink and clock state can flap.

The fix is persistence mode: keep a kernel-side reference open so init state survives between client processes.

# enable on every GPU
$ sudo nvidia-smi -pm 1
Persistence mode: Enabled

# verify
$ nvidia-smi --query-gpu=persistence_mode --format=csv
persistence_mode
Enabled

The modern way is the nvidia-persistenced daemon (since driver 396):

$ sudo systemctl enable --now nvidia-persistenced
$ systemctl status nvidia-persistenced --no-pager
● nvidia-persistenced.service - NVIDIA Persistence Daemon
     Loaded: loaded (/lib/systemd/system/nvidia-persistenced.service; enabled)
     Active: active (running) since Wed 2025-09-10 14:22:11 UTC; 4 weeks ago

If you don't run the daemon, set nvidia-smi -pm 1 in /etc/rc.local or a one-shot systemd unit run after nvidia.ko loads.

Failure mode: on Hopper without persistence, you'll see NVSwitch/Fabric Manager log "fabric not initialized" warnings on every cold pod start, plus 5-10 second extra latency. The GPU Operator turns this on by default; hand-installed nodes often don't.

nvidia-smi -q deep dive

nvidia-smi (no args) is a summary. nvidia-smi -q dumps every counter the driver tracks. The fields you actually need to know:

ECC

ECC Mode
    Current                           : Enabled
    Pending                           : Enabled
ECC Errors
    Volatile
        SRAM Correctable              : 0
        SRAM Uncorrectable            : 0
        DRAM Correctable              : 12
        DRAM Uncorrectable            : 0
    Aggregate
        SRAM Correctable              : 0
        SRAM Uncorrectable            : 0
        DRAM Correctable              : 184
        DRAM Uncorrectable            : 1
Retired Pages
    Single Bit ECC                    : 0
    Double Bit ECC                    : 1

• Correctable errors are bit-flips the ECC engine fixed in flight. Some baseline rate is normal (low single digits per week per GPU). Steep growth = thermal or aging memory.
• Uncorrectable (especially DRAM) means a memory location is bad. The driver will "row-remap" — H100/H200/B-series do this dynamically without reboot. Watch Remapped Rows and Pending count.
• Aggregate counters survive reboots; Volatile clears on driver reload. On a node with concerning aggregate counts, nvidia-smi -q | grep -A 5 "Remapped Rows" shows whether remap pool is full (= RMA the GPU).

PCIe link

PCI
    Bus                               : 0x1B
    GPU Link Info
        PCIe Generation
            Max                       : 5
            Current                   : 5
        Link Width
            Max                       : 16x
            Current                   : 16x

H100 SXM5 = PCIe Gen5 x16 (each direction). If Current shows Gen3 / Gen4 / x8, the PCIe link is degraded — usually means a reseat is needed, sometimes a faulty riser. NCCL throughput will look fine on NVLink-only collectives but tank on host-staged paths.

Throttle reasons

Clocks Throttle Reasons
    Idle                              : Not Active
    Applications Clocks Setting       : Not Active
    SW Power Cap                      : Not Active
    HW Slowdown                       : Not Active
    HW Thermal Slowdown               : Not Active
    HW Power Brake Slowdown           : Not Active
    Sync Boost                        : Not Active
    SW Thermal Slowdown               : Not Active

Any of HW *Slowdown Active = an alarm. HW Thermal Slowdown = the GPU has hit ~88-95°C and is dropping clocks to survive. HW Power Brake = upstream PSU asserted PWR_BRAKE (often a tripped power budget at the rack PDU). Both are visible during NCCL all-reduce stress tests.

Programmatic queries (better for monitoring)

# CSV one-liner per GPU
$ nvidia-smi --query-gpu=index,name,pstate,temperature.gpu,utilization.gpu,memory.used,ecc.errors.uncorrected.aggregate.total --format=csv,noheader

0, NVIDIA H100 80GB HBM3, P0, 41, 0 %, 312 MiB, 0
1, NVIDIA H100 80GB HBM3, P0, 39, 0 %, 312 MiB, 0

For real fleet monitoring use DCGM — it has stable metrics names and Prometheus integration. nvidia-smi is for ad-hoc human use.

Troubleshooting playbook

"Failed to initialize NVML: Driver/library version mismatch"

$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 570.124

The kernel module loaded is one version, the userspace libnvidia-ml.so is a different version. Almost always caused by an apt upgrade nvidia-driver-* that updated userspace but couldn't unload the running kernel module (because something has /dev/nvidia* open).

Fixes in order of preference:

1. Reboot. Brutal but reliable. Schedule it.
2. Kill all GPU users, unload modules, reload:

   sudo lsof /dev/nvidia* 2>/dev/null
   # kill those PIDs (or stop the workloads gracefully)
   sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia_peermem nvidia
   sudo modprobe nvidia        # auto-cascades the rest
   nvidia-smi
   

3. If you can't unload because something keeps re-opening (nvidia-persistenced, container runtime, kubelet, MIG manager): stop those services first.

"no devices were found" / missing /dev/nvidia*

$ ls -la /dev/nvidia*
ls: cannot access '/dev/nvidia*': No such file or directory

Root causes, ordered by probability:

1. Kernel module didn't load. dmesg | grep -i nvidia — look for "modules verification failed" (Secure Boot signing), "no compatible GPU" (wrong driver branch for your arch), or "NVRM: failed to copy vbios" (often a hardware reseat / PCIe error).
2. nvidia-modprobe couldn't create device nodes. Caused by misconfigured cgroup namespace or missing the SUID bit on /usr/bin/nvidia-modprobe.
3. GPU not seen by PCIe. lspci | grep -i nvidia shows nothing → it's a hardware/BIOS-level problem, not a driver problem. Check dmesg | grep -i pcie and BMC/iDRAC logs.

"NVRM: GPU has fallen off the bus"

NVRM: Xid (PCI:0000:1b:00): 79, GPU has fallen off the bus.

Catastrophic — the GPU dropped off PCIe entirely. Driver can't recover until reboot. Causes: hardware fault, severe thermal event, NVLink fabric error cascading back, marginal power. Once you see this, the node is hard-down for that GPU until reboot, and the GPU is suspect for RMA if it recurs.

Kernel module signing / Secure Boot

On UEFI systems with Secure Boot enabled, an unsigned nvidia.ko won't load:

modprobe: ERROR: could not insert 'nvidia': Operation not permitted

dmesg will show "Lockdown: insmod: unsigned module loading is restricted". Either disable Secure Boot, or sign the modules with a Machine Owner Key (MOK). Most production HPC fleets just disable Secure Boot — it interacts poorly with DKMS rebuilds.

nvidia-peermem not loaded → no GDR

$ lsmod | grep peermem
$                              # silence is the failure

Multi-node NCCL will work but will fall back to staging through host RAM, a 5-10x throughput hit with no error message. See nvidia-peermem for the full story.

See also

• fabric-manager — required for any NVSwitch-equipped HGX/DGX node
• nvidia-peermem (GDR) — how the IB stack DMAs into GPU memory
• Container Toolkit — running CUDA in containers
• Mellanox OFED — the IB/RoCE stack that pairs with the GPU stack
• DCGM monitoring — what to scrape into Prometheus once the driver is healthy
• GPU generations — pick the right driver branch for your silicon