Driver / firmware mismatch matrix

Half of the production GPU failures I've debugged trace back to "someone upgraded one component and didn't upgrade the matched component". The NVIDIA + Mellanox stack has at least seven version-coupling relationships, none of which the package manager enforces. apt upgrade will happily pull a kernel that breaks nvidia.ko, an OFED that breaks GDR, or a driver that mismatches the fabric-manager — and the failure won't show up until your training job hangs at NCCL_COMM_INIT.

This page is the matrix. Walk through it before touching any node, especially before upgrading anything in production.

The matrix

Three rules:

1. Plan upgrades top-down: kernel → OFED → NVIDIA driver → fabric-manager → peermem → workload images.
2. Pin everything in your base image / OS image that you don't want surprises in.
3. Validate before rolling: bring up one node with the new versions, run the bring-up validation script, then propagate.

NVIDIA driver vs CUDA toolkit (in container image)

The NVIDIA driver exposes a CUDA driver API (libcuda.so) at a specific version. The CUDA toolkit (in your container) has a runtime API (libcudart.so) compiled against a target driver API. Forward compatibility exists but only one way: a newer driver supports older CUDA toolkits.

Compatibility quick reference (driver → minimum CUDA toolkit it supports):

If your training container ships CUDA 12.8 and the host driver is 535 (CUDA driver API 12.2), the workload will fail at startup:

CUDA driver version is insufficient for CUDA runtime version

How to check on a host:

$ nvidia-smi | head -3
| NVIDIA-SMI 550.144.03    Driver Version: 550.144.03    CUDA Version: 12.4   |

$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  550.144.03  Wed Mar 19 ...

How to check inside a container:

$ nvcc --version
Cuda compilation tools, release 12.4, V12.4.131
$ ldconfig -p | grep libcudart
libcudart.so.12 => /usr/local/cuda/lib64/libcudart.so.12

The host driver's CUDA version (nvidia-smi's top line) must be ≥ the container's CUDA toolkit version.

NVIDIA driver vs nvidia-fabricmanager

On HGX / DGX systems with NVSwitch (H100/H200/B200), nvidia-fabricmanager configures the switch fabric so GPUs can talk over NVLink. It must match the driver minor version exactly.

$ nvidia-smi | grep "Driver Version"
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03   ...

$ dpkg -l | grep nvidia-fabricmanager
ii  nvidia-fabricmanager-550   550.144.03-1   amd64   Fabric Manager for NVSwitch ...

Both at 550.144.03. If apt upgrade bumps the driver to 550.146 but holds back fabric-manager (or vice versa), fabric-manager refuses to start:

$ systemctl status nvidia-fabricmanager
Active: failed (Result: exit-code) since ...

$ tail /var/log/fabricmanager.log
fabric manager NVIDIA driver interface version 550.146 doesn't match with the expected version 550.144

GPUs are visible to nvidia-smi but every multi-GPU workload hangs.

Fix: align the versions.

# pin both packages
apt install --allow-downgrades \
    nvidia-driver-550=550.144.03-1 \
    nvidia-fabricmanager-550=550.144.03-1
apt-mark hold nvidia-driver-550 nvidia-fabricmanager-550

See fabric-manager for the full picture.

NVIDIA driver vs nvidia-peermem

nvidia-peermem.ko is shipped inside the NVIDIA driver package, so they always share a version. The trap is that kernel upgrades trigger a DKMS rebuild of both modules — and if the rebuild fails or is skipped, nvidia-peermem won't load on the new kernel even though nvidia.ko might.

Verify both load on the running kernel:

$ uname -r
6.8.0-45-generic

$ modinfo nvidia        | grep -E "^(filename|version|vermagic)"
filename:       /lib/modules/6.8.0-45-generic/updates/dkms/nvidia.ko.zst
version:        550.144.03
vermagic:       6.8.0-45-generic SMP preempt mod_unload modversions

$ modinfo nvidia_peermem | grep -E "^(filename|version|vermagic)"
filename:       /lib/modules/6.8.0-45-generic/updates/dkms/nvidia-peermem.ko
version:        550.144.03
vermagic:       6.8.0-45-generic SMP preempt mod_unload modversions

vermagic must match uname -r. If the file is /lib/modules/6.8.0-43-generic/... you booted into a different kernel than the one DKMS built modules for — expect failures.

Re-run DKMS:

dkms autoinstall                          # rebuild for current kernel
# or specifically:
dkms install nvidia/550.144.03 -k $(uname -r)

See peermem.

MLNX_OFED vs ConnectX firmware

Mellanox publishes a firmware compatibility table per OFED release. Each OFED supports a range of firmware per HCA model. Outside that range = undefined.

How to check current firmware:

$ ibstat | grep "Firmware version"
        Firmware version: 28.41.1000

$ flint -d /dev/mst/mt4129_pciconf0 q | head -10
Image type:            FS4
FW Version:            28.41.1000
FW Release Date:       19.3.2024
...

How to check current OFED:

$ ofed_info -s
MLNX_OFED_LINUX-24.10-1.1.4.0:

Then cross-reference:

(Always check the actual release notes for the version you're on; this table will drift.)

If firmware is below the supported range, the most common symptom is link instability (frequent ibstat flaps) or vendor errors at random times under load. Above the supported range works most of the time but isn't a configuration NVIDIA tests.

Updating firmware:

# fetch FW from the nvidia download page or use mlxup
$ wget https://www.mellanox.com/downloads/firmware/fw-ConnectX7-rel-28_41_1000.bin
$ flint -d /dev/mst/mt4129_pciconf0 -i fw-ConnectX7-rel-28_41_1000.bin burn
$ mlxfwreset -d /dev/mst/mt4129_pciconf0 -y reset

The reset is needed to make new firmware active without a full reboot. See OFED.

K8s GPU Operator chart vs NVIDIA driver vs CUDA toolkit

The GPU Operator chart pins a default driver image version in values.yaml. When you update the chart, the driver version moves too. If you also have user containers with a fixed CUDA version, the matrix gets tight.

# values.yaml
driver:
  enabled: true
  version: "550.90.07"     # this is what the operator deploys as nvidia-driver-daemonset
  repository: "nvcr.io/nvidia"
  image: "driver"
toolkit:
  enabled: true
  version: "v1.16.2-ubuntu20.04"

If your tenant containers are CUDA 12.8 (driver API requirement = 570) and the operator deploys driver 550 (CUDA API = 12.4), tenants will hit "driver version too old". You either pin the chart to a newer driver, or ask tenants to use a CUDA 12.4-compatible image.

Workflow:

1. Pick the workload's CUDA target.
2. Find the minimum driver that supports it (table above).
3. Find the GPU Operator chart that ships ≥ that driver.
4. Pin everything; document.

See GPU Operator.

Linux kernel vs nvidia.ko (and the Open Kernel Modules question)

NVIDIA ships two flavors of kernel modules:

• Closed source (nvidia-driver-XXX, traditional). Works on every supported kernel; DKMS-rebuilds against headers.
• Open Kernel Modules (open-gpu-kernel-modules, since R515). MIT/GPL dual-licensed, statically build-able, but only supports Turing+ (no Pascal/Volta) and requires kernel ≥ 4.18.

Open is the future and is required for some features (e.g., GSP RPC offload on newer GPUs is more reliable on open). But there are kernel-specific regressions that bite. Real example: kernel 6.8.111 introduced a change to pin_user_pages behavior that broke nvidia-peermem GDR setup on driver 535. Symptoms: lsmod shows peermem loaded, but every ibv_reg_mr(ptr_to_gpu_mem) returns ENOMEM.

Detection:

$ uname -r
6.8.111-generic

$ dmesg | grep -i peermem
[  XXX] nvidia-peermem: Loaded NVIDIA-peermem module successfully
[  XXX] nvidia-peermem: failed to register peer client: -22

-22 = EINVAL = kernel API surface changed under us.

Rollback: pin the kernel to the last known-good version.

# ubuntu
$ apt install linux-image-6.8.110-generic linux-headers-6.8.110-generic
$ apt remove linux-image-6.8.111-generic linux-headers-6.8.111-generic
$ update-grub
$ reboot

# pin going forward
$ apt-mark hold linux-image-6.8.110-generic linux-headers-6.8.110-generic

Forward fix: wait for an NVIDIA driver release that's been re-validated against the new kernel, or for the kernel to revert. The takeaway is that we always validate the matrix on one node before propagating a kernel upgrade.

Linux kernel vs MLNX_OFED upstream-included drivers

Modern kernels ship mlx5_core and friends inbox. If you install MLNX_OFED, it replaces these with its own DKMS-built versions. The DKMS rebuild requires:

• Matching kernel headers installed (linux-headers-$(uname -r)).
• gcc / make / kernel build infrastructure.
• A kernel version supported by the OFED release.

OFED 24.10 supports up to ~6.8 stable; if you upgrade to 6.10, OFED won't build until they release a matching version. Symptoms: mlx5_core not loading, dmesg shows DKMS rebuild errors.

$ dkms status
mlnx-ofed-kernel/24.10.OFED.24.10.4.1.4.1, 6.8.0-45-generic, x86_64: installed
mlnx-ofed-kernel/24.10.OFED.24.10.4.1.4.1, 6.10.0-9-generic, x86_64: failed (build_log)

$ cat /var/lib/dkms/mlnx-ofed-kernel/24.10.OFED.24.10.4.1.4.1/build/make.log
... unknown type 'foo_bar', driver/net/ethernet/mellanox/mlx5/...

Fix: stay on a kernel version OFED supports, OR fall back to the inbox rdma-core (uninstall OFED) until OFED catches up.

How to find what's installed (cheat sheet)

# NVIDIA driver
nvidia-smi | head -3                                    # driver + CUDA driver API
cat /proc/driver/nvidia/version                         # exact NVRM build string
modinfo nvidia | grep -E '^(version|vermagic)'          # module file's reported version

# fabric-manager
systemctl status nvidia-fabricmanager
dpkg -l | grep nvidia-fabricmanager                     # package version
nv-fabricmanager --version 2>/dev/null

# peermem
lsmod | grep nvidia_peermem
modinfo nvidia_peermem | grep -E '^(version|vermagic|filename)'

# OFED
ofed_info -s                                            # short OFED version
ofed_info -n                                            # numeric only
dpkg -l | grep -E '^ii.*(mlnx-ofed|doca-ofed)'

# ConnectX firmware
ibstat | grep "Firmware version"                        # per-HCA
flint -d /dev/mst/mt4129_pciconf0 q | head             # detailed
mlxlink -d mlx5_0 -m -e                                # signal-quality counters

# kernel
uname -r
cat /proc/version

# kernel headers (DKMS needs these)
dpkg -l | grep "linux-headers-$(uname -r)"

# K8s GPU Operator
helm list -n gpu-operator
helm get values nvidia-gpu-operator -n gpu-operator | grep -E "(version|driver|toolkit)"

# CUDA in containers
kubectl exec -it <pod> -- bash -c 'nvcc --version; ldconfig -p | grep libcudart'

Safe upgrade order

When upgrading any one of these, others may need to move with it. Use this order:

1. Plan: read each component's release notes. Identify dependencies.
2. Cordon one node: kubectl cordon <node>; kubectl drain <node> --ignore-daemonsets.
3. Upgrade kernel (if needed). Reboot.
4. Validate kernel headers + DKMS for current modules: dkms status should show all installed for the new kernel.
5. Upgrade OFED (if needed). The OFED installer (mlnxofedinstall) handles its own DKMS.
6. Upgrade NVIDIA driver (and peermem rides along). Reboot if driver was running.
7. Upgrade fabric-manager to match driver. Restart service.
8. Validate: run nvidia-smi, ibstat, lsmod, dcgmi diag -r 1, then the bring-up validation script.
9. Run multi-node NCCL between this node and a known-healthy node before un-cordoning.
10. Un-cordon, watch for issues for at least one full workload run before moving on.

Skipping the cordon-and-validate step on the first node has cost me weeks. Don't skip it.

Real-world rollback example

Scenario: in production, kernel 6.8.111 was rolled out by routine apt-upgrade across the GPU pool. NCCL training jobs started failing with vendor err 88 (LOCAL_PROTECTION_ERR). lsmod showed nvidia_peermem loaded; dmesg showed nvidia-peermem: failed to register peer client: -22.

Diagnosis:

# compare a healthy node and a failing node
healthy$ uname -r
6.8.110-generic

failing$ uname -r
6.8.111-generic

# ⇒ kernel is the only difference.

Rollback:

# on each affected node
$ apt install --allow-downgrades \
    linux-image-6.8.110-generic linux-headers-6.8.110-generic
$ apt remove linux-image-6.8.111-generic linux-headers-6.8.111-generic
$ update-grub
$ reboot

# verify after reboot
$ uname -r
6.8.110-generic
$ lsmod | grep peermem
nvidia_peermem  16384  3
$ dmesg | grep -i peermem
[  XXX] nvidia-peermem: Loaded NVIDIA-peermem module successfully

# pin
$ apt-mark hold linux-image-6.8.110-generic linux-headers-6.8.110-generic linux-image-generic linux-headers-generic

Prevention: cluster-wide policy that kernels are pinned in the OS image and only roll forward through validated builds. Routine apt upgrade does not touch kernel.

# /etc/apt/preferences.d/no-kernel-upgrades
Package: linux-image-* linux-headers-* linux-generic linux-image-generic linux-headers-generic
Pin: release *
Pin-Priority: -1

See also

• NVIDIA driver
• fabric-manager
• peermem
• Mellanox OFED
• GPU Operator
• Triage decision tree

External:

• NVIDIA driver release notes: docs.nvidia.com/datacenter/tesla/tesla-release-notes
• CUDA compatibility matrix: docs.nvidia.com/deploy/cuda-compatibility/
• MLNX_OFED matrix: network.nvidia.com/support/mlnx-ofed-matrix/
• Open GPU Kernel Modules: github.com/NVIDIA/open-gpu-kernel-modules