Mellanox OFED: the RDMA stack underneath every multi-node GPU job

If your cluster has Mellanox/NVIDIA ConnectX (CX-5/6/7) or BlueField NICs and you run multi-node NCCL, you live or die by the RDMA stack on each host. There are three flavors — upstream inbox rdma-core, MLNX_OFED (the historical full-stack package), and the newer DOCA-OFED — and picking the right one matters. This page is the operator's view: which to use, what the components actually are, and the persistent module-loading pattern that prevents the most common silent failure.

What MLNX_OFED actually is

OFED stands for OpenFabrics Enterprise Distribution — originally an upstream, vendor-agnostic RDMA stack maintained by the OpenFabrics Alliance. MLNX_OFED is Mellanox/NVIDIA's downstream of that, repackaged as a giant tarball that contains:

1. Custom kernel modules: Mellanox-patched versions of mlx5_core, mlx5_ib, ib_core, ib_uverbs, ib_cm, rdma_cm, rdma_ucm, ib_umad, ib_ipoib, ib_isert and friends, often ahead of upstream by months.
2. Userspace libraries: libibverbs, librdmacm, libibumad, the Mellanox provider plugin libmlx5, plus libnvme-rdma, libfabric for OFI workloads.
3. Firmware tools: mft package — mst, flint, mlxconfig, mlxlink, mlxcables — for everything from flashing firmware to staring at SerDes eye diagrams.
4. Diagnostics: ibstat, ibstatus, ibv_devinfo, ibping, ib_send_bw, ib_write_bw, ib_read_bw, iblinkinfo, ibnetdiscover, ibtracert.
5. OpenSM: optional userspace InfiniBand subnet manager (only needed if your IB fabric isn't using switch-managed SM).
6. DAPL / SDP shims: legacy. You won't touch these on a modern training cluster.

The tarball ships as MLNX_OFED_LINUX-<version>-<distro>-x86_64.tgz and you install it with ./mlnxofedinstall. It blows away the inbox stack and replaces it with its own.

Why it exists

For years, upstream rdma-core lagged Mellanox hardware by 12-18 months. ConnectX-6 launched with features (e.g. dynamically connected transport, hardware steering offloads) that took a year or more to land in mainline. If you wanted production-grade performance from new HCAs, MLNX_OFED was effectively mandatory.

In 2024-2025 the gap has narrowed — modern kernels (6.x) carry ConnectX-7 support out of the box, and rdma-core has caught up on the userspace side. But MLNX_OFED is still the safe choice for clusters where: you have CX-7/BlueField, you need GPUDirect, you need the full Mellanox firmware tool suite, or you need SHARP for in-network reductions.

Inbox vs MLNX_OFED vs DOCA-OFED

DOCA-OFED is gradually replacing MLNX_OFED as NVIDIA consolidates their networking software around the DOCA SDK. As of 2025, MLNX_OFED 24.x and DOCA-OFED 2.x are both supported. New greenfield deployments should evaluate DOCA-OFED first; existing MLNX_OFED clusters should stay on MLNX_OFED until there's a reason to migrate.

Common shorthand: people say "OFED" without specifying. Confirm by looking at what's installed:

$ ofed_info -s
MLNX_OFED_LINUX-24.10-1.1.4.0:

$ dpkg -l | grep -E '^ii.*(mlnx-ofed|doca-ofed)' | head -3
ii  mlnx-ofed-kernel-dkms   24.10.OFED.24.10.4.1.4.1  ...
ii  mlnx-ofed-kernel-utils  24.10.OFED.24.10.4.1.4.1  ...

The kernel modules — and why all 9 belong in /etc/modules-load.d

The IB stack is modular: each piece (CM, ucm, umad, IPoIB) is its own kernel module, and not all of them auto-load from a fresh boot on every distro. Reproducible failure pattern: a node boots, the GPUs are healthy, ibstat shows the IB ports Active, but your training workload fails with "could not open RDMA device" or NCCL falls back to TCP.

The fix that has solved this on every cluster I've worked on: explicitly list all 9 RDMA-related modules in /etc/modules-load.d/ so systemd loads them at boot, regardless of whether something else would have lazily loaded them.

What each module does

Plus optionally ib_isert (iSCSI extensions), ib_iser (initiator side), nfsrdma for NFS-over-RDMA — only enable these if you're using those workloads.

Persistent module loading (the universal fix)

# /etc/modules-load.d/rdma.conf
mlx5_core
mlx5_ib
ib_core
ib_uverbs
ib_cm
rdma_cm
rdma_ucm
ib_umad
ib_ipoib

Apply it by running systemctl restart systemd-modules-load (or just reboot). Verify:

$ lsmod | grep -E '^(mlx5|ib_|rdma_)'
ib_isert               45056  0
iscsi_target_mod      483328  1 ib_isert
target_core_mod       569344  3 iscsi_target_mod,ib_isert
ib_ipoib              163840  0
ib_umad                32768  4
rdma_ucm               36864  0
rdma_cm                86016  2 ib_iser,rdma_ucm
ib_iser                57344  0
ib_cm                 131072  2 rdma_cm,ib_ipoib
ib_uverbs             192512  2 rdma_ucm,mlx5_ib
mlx5_ib               524288  0
ib_core               573440 11 rdma_cm,ib_ipoib,iscsi_iser,ib_iser,...,ib_uverbs,...,nvidia_peermem
mlx5_core            2367488  1 mlx5_ib

What you want to see:

• All 9 modules listed.
• ib_core has lots of dependents — including nvidia_peermem if you've also loaded that. That dependency is the strongest signal that GDR registration succeeded.
• /dev/infiniband/ populated:

$ ls /dev/infiniband/
issm0   issm2   issm4   issm6   issm8    rdma_cm   uverbs1  uverbs3  uverbs5  uverbs7
issm1   issm3   issm5   issm7   issm9              uverbs0  uverbs2  uverbs4  uverbs6
umad0   umad2   umad4   umad6   umad8              uverbs9
umad1   umad3   umad5   umad7   umad9

You should see uverbsN, umadN, issmN for each port, plus a single rdma_cm. Missing umadN means ib_umad didn't load (and ibstat will fail). Missing rdma_cm means rdma_ucm didn't load (and NCCL will fail to use IB).

Why "modules redundantly listed" is fine

You will see (especially on freshly imaged DGX or deeply-customized HGX) that some modules are already loaded by the time systemd-modules-load runs — for example, mlx5_core typically auto-loads via udev when the PCI device probes. Listing it in rdma.conf is harmless: modprobe is idempotent. Belt and braces is the right policy here.

Firmware version pinning per OFED

The MLNX_OFED package version is paired with a recommended HCA firmware version. Mismatch causes a wide variety of warnings and the occasional outright failure:

$ ofed_info -s
MLNX_OFED_LINUX-24.10-1.1.4.0

$ ibv_devinfo | grep -E "^\s*hca_id|fw_ver"
hca_id: mlx5_0
        fw_ver: 28.41.1000

$ flint -d /dev/mst/mt4129_pciconf0 query | head -5
Image type:            FS4
FW Version:            28.41.1000
FW Release Date:        9.10.2024
Product Version:       28.41.1000
Rom Info:              type=UEFI version=14.34.12 cpu=AMD64,AARCH64

NVIDIA publishes a compat matrix per OFED release. Roughly:

Mixed-firmware fleets are a real source of grief. A 24-node cluster where 21 nodes are on 28.41 and 3 nodes are on 28.39 can show subtle PFC mis-handling on RoCE, or NCCL all-reduce hangs only when those 3 nodes are part of the ring. Standardize firmware before chasing software bugs.

To upgrade firmware:

# enumerate HCAs
$ mst start
$ mst status -v
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET
ConnectX7(rev:0)        /dev/mst/mt4129_pciconf0      1b:00.0   mlx5_0          net-ibp27s0
ConnectX7(rev:0)        /dev/mst/mt4129_pciconf0.1    1b:00.1   mlx5_1          net-ibp27s0d1
...

# burn firmware (offline; reset-required afterwards)
$ flint -d /dev/mst/mt4129_pciconf0 -i fw-ConnectX7-rel-28_42_1000.bin burn
$ mlxfwreset -d /dev/mst/mt4129_pciconf0 -l 3 reset

Test on one node, then push fleet-wide via Ansible / Salt / your config tool of choice.

mlxconfig: firmware-level settings you'll actually touch

mlxconfig writes settings to NVRAM on the HCA. These persist across reboots and affect how the card behaves before the OS-level driver gets involved. The handful you'll actually touch on a GPU host:

$ mlxconfig -d /dev/mst/mt4129_pciconf0 query | head -40
Device #1:
----------
Device type:        ConnectX7
Name:               MCX755106AS-HEAT_Ax
Description:        NVIDIA ConnectX-7 VPI adapter card; HDR IB and 200GbE; dual-port QSFP56;
Device:             /dev/mst/mt4129_pciconf0

Configurations:                                      Next Boot
        ...
        LINK_TYPE_P1                                IB(1)
        LINK_TYPE_P2                                IB(1)
        ROCE_CC_PRIO_MASK_P1                        255
        ROCE_CC_PRIO_MASK_P2                        255
        CNP_DSCP_P1                                 48
        CNP_DSCP_P2                                 48
        ...

Setting + reset:

$ mlxconfig -d /dev/mst/mt4129_pciconf0 set PCI_WR_ORDERING=1
Device #1:
----------
Device type:    ConnectX7
Configurations:                              Next Boot       New
        PCI_WR_ORDERING                     per_mkey(0)     force_relax(1)
 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

$ mlxfwreset -d /dev/mst/mt4129_pciconf0 -l 3 reset    # or just reboot

mft / mst / flint / mlxlink — the diagnostics suite

You'll touch these tools whenever the link layer feels off:

mlxlink — physical layer health

The single most useful command for "why is this link slow / flapping":

$ mlxlink -d /dev/mst/mt4129_pciconf0 -p 1 -m -c -e
Operational Info
----------------
State                           : Active
Physical state                  : LinkUp
Speed                           : NDR
Width                           : 4x
FEC                             : Standard_RS-FEC - (528,514+2DC+2AD)
Loopback Mode                   : No Loopback
Auto Negotiation                : ON

Supported Info
--------------
Enabled Link Speed              : 0x00010000 (NDR)
Supported Cable Speed           : 0x00010000 (NDR)

Troubleshooting Info
--------------------
Status Opcode                   : 0
Group Opcode                    : N/A
Recommendation                  : No issue was observed.

Tool Information
----------------
Firmware Version                : 28.41.1000
amBER Version                   : 4.0
MFT Version                     : mft 4.30.1-23

Physical Counters and BER Info
------------------------------
Time Since Last Clear           : 4 days 11:24:18
Effective Physical Errors       : 0
Effective Physical BER          : 1E-255
Raw Physical Errors Per Lane    : 0,0,0,0
Raw Physical BER                : 1E-255

What to look for:

• State: Active + Physical state: LinkUp — port is good.
• Effective Physical BER at 1E-255 is the placeholder for "essentially zero". Anything in 1E-12 or worse is a flapping cable / dirty connector.
• Status Opcode non-zero = firmware reports a specific link issue. The "Recommendation" line spells it out.

flint — firmware ops

flint -d /dev/mst/mt4129_pciconf0 query     # version + dates
flint -d /dev/mst/mt4129_pciconf0 verify    # verify firmware checksums
flint -d /dev/mst/mt4129_pciconf0 burn -i <fw.bin>

ibstat / ibv_devinfo — port view

$ ibstat mlx5_0
CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.41.1000
        Hardware version: 0
        Node GUID: 0xa088c20300c8de4a
        System image GUID: 0xa088c20300c8de4a
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 41
                LMC: 0
                SM lid: 1
                Capability mask: 0xa651e848
                Port GUID: 0xa288c2fffec8de4a
                Link layer: InfiniBand

State: Active + Physical state: LinkUp + Rate: 400 (NDR) means the port is operationally healthy and connected to a subnet manager. Anything else (Initializing, Polling) means the port can't reach the SM — usually a cabling or switch-side problem.

Troubleshooting

/dev/infiniband/ empty or partial

$ ls /dev/infiniband/
ls: cannot access '/dev/infiniband/': No such file or directory

ib_uverbs didn't load. Either OFED isn't installed, the modules aren't in /etc/modules-load.d/rdma.conf, or there's a kernel symbol mismatch (see below). Check:

$ lsmod | grep -E '^(mlx5_core|ib_uverbs|ib_umad)'
$ dmesg | grep -iE "mlx5|ib_core" | tail -20

dmesg is the source of truth for module load errors.

modprobe: ERROR: could not insert 'rdma_ucm': Invalid argument

Symbol-version mismatch. Two flavors:

1. MLNX_OFED DKMS modules trying to load on a kernel they weren't built against. Often after a linux-image-* upgrade. Fix:

   sudo dkms autoinstall
   # or fully reinstall:
   sudo /usr/sbin/mlnxofedinstall --kernel $(uname -r) --kernel-only --add-kernel-support
   

2. Mixed in-tree + DKMS state. Distro inbox rdma-core modules are loaded; their symbols don't match the DKMS-built modules now trying to layer on top. dmesg will say module: disagrees about version of symbol ib_register_peer_memory_client or similar. Fix: rmmod the inbox modules in dependency order, then modprobe the OFED ones; or (more reliably) reboot.

ibstat says "ibwarn: [...] umad_get_ca: can't open UMAD port"

ib_umad not loaded. Either it wasn't in your modules-load.d or it failed to load.

NCCL multi-node fails with "Failed to set up RDMA QP"

NCCL INFO Channel 00 : 0[1b000] -> 8[1b000] [send] via NET/IB/0/0
... time passes ...
NCCL ERROR: failed to set up RDMA queue pair: Cannot allocate memory

Check three things:

1. MTU mismatch on RoCE. All NICs in the path must agree on MTU. ibv_devinfo -v | grep active_mtu — should be 4096 for IB or whatever the RoCE network is configured for.
2. ib_uverbs not loaded on one node — silent partial outage.
3. Memory locking limits. ulimit -l must be unlimited for the user/process running NCCL. Most images don't set this. Add to systemd service / pod spec:

   securityContext:
     capabilities:
       add: ["IPC_LOCK"]
   

   See ulimits.

peermem GDR not registering despite IB ports Active

$ ls /sys/kernel/mm/memory_peers/
ls: cannot access '/sys/kernel/mm/memory_peers/': No such file or directory

nvidia_peermem was loaded before ib_core was up — registration silently failed. Reload in correct order:

sudo rmmod nvidia_peermem
sudo modprobe nvidia_peermem
ls /sys/kernel/mm/memory_peers/

If nv_mem shows up after, you're good. Make sure nvidia_peermem is listed after ib_core in your modules-load.d files (or in a separate file alphabetically later, e.g. nvidia-peermem.conf rather than 0-nvidia.conf). See nvidia-peermem.

openibd service is dead

openibd is the OFED-provided init service that manages module load + NIC bring-up. On systemd it shows as openibd.service. If it's enabled but inactive (dead), modules won't load reliably:

$ systemctl status openibd
○ openibd.service - openibd
     Loaded: loaded (/lib/systemd/system/openibd.service; enabled)
     Active: inactive (dead)

Start it:

sudo systemctl start openibd
sudo journalctl -u openibd -n 50

Common reason it stays dead: /etc/infiniband/openib.conf references modules that don't exist in the current kernel build. Fix the conf, or rely entirely on /etc/modules-load.d/rdma.conf (which is the more portable approach across reboots and OS upgrades).

See also

• NVIDIA driver stack — pairs with OFED to make GPUDirect work
• nvidia-peermem (GDR) — registers into the IB stack OFED provides
• RDMA fundamentals — verbs, QPs, memory regions explained
• InfiniBand — IB-specific concerns once OFED is healthy
• RoCE v2 + DCB/PFC/ECN — RoCE-specific concerns once OFED is healthy
• NCCL multi-node — the workload that proves OFED is doing its job