RDMA debugging: from libibverbs to the wire

RDMA is the protocol that turns a 400 Gb/s NIC into a useful 400 Gb/s NIC. When it works, multi-node NCCL hits 360 GB/s busbw and the cluster prints money. When it doesn't, you get a vague "vendor err 81" or, worse, "no errors but the bandwidth is half what it should be" — and you have to chase the problem through six layers of stack.

This page is the bottom-up tour. Walk it in order; the further down the stack you find a fault, the less time you waste on the layers above.

The layers

                            ┌─────────────────────────┐
                            │     Application         │   NCCL, MPI, libfabric
                            ├─────────────────────────┤
                            │  Userspace verbs        │   libibverbs, librdmacm
                            ├─────────────────────────┤
                            │  /dev/infiniband nodes  │   ib_uverbs, rdma_ucm, ib_umad
                            ├─────────────────────────┤
                            │  Kernel modules         │   mlx5_ib, ib_core, mlx5_core
                            ├─────────────────────────┤
                            │  Subnet management      │   OpenSM / switch-managed SM
                            ├─────────────────────────┤
                            │  Switch fabric          │   leaf, spine, transceivers, cables
                            ├─────────────────────────┤
                            │  HCA hardware           │   ConnectX silicon, PCIe link
                            └─────────────────────────┘

Each layer has a primary diagnostic tool. Start at the bottom.

Layer 1 — HCA hardware

lspci

$ lspci -nn | grep -i mellanox
b1:00.0 Infiniband controller [0207]: Mellanox Technologies MT2910 Family [ConnectX-7] [15b3:1021]
b3:00.0 Infiniband controller [0207]: Mellanox Technologies MT2910 Family [ConnectX-7] [15b3:1021]
...

You should see one PCIe device per HCA port that the chassis exposes. On an 8-NIC H100 box, you should see 8 entries (or 4 dual-port cards). Missing entries = card not detected by BIOS, broken card, or missing PCIe lane.

# verify PCIe link width and speed
$ lspci -vvv -s b1:00.0 | grep -E "LnkCap|LnkSta"
        LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, ...
        LnkSta: Speed 32GT/s, Width x16, ...

LnkSta should match LnkCap. CX-7 = Gen5 x16 = 32 GT/s × 16. Anything lower means the slot or card is in a degraded state — investigate before assuming the HCA itself is bad.

dmesg

$ dmesg | grep -iE 'mlx5|infiniband' | tail -30
[   5.123] mlx5_core 0000:b1:00.0: firmware version: 28.41.1000
[   5.456] mlx5_core 0000:b1:00.0: Port module event: module 0, Cable plugged
[   5.789] mlx5_core 0000:b1:00.0: mlx5_load_one:1234:(pid 234): Driver loaded
[   8.012] mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0

What you want to see:

• Driver loaded (the success line).
• Cable plugged events with no Cable unplugged afterward.
• No port_module: PME on repeating — that's the link flapping.

Bad signs:

• mlx5_core: ... function reset — firmware crashed and was recovered. Repeated = bad.
• mlx5_core: ... ICM allocation failed — out-of-memory in the firmware, often when too many QPs / MRs are registered.
• port module: bad cable — physical layer.

mlxlink — physical-layer counters

The most useful single tool for "is this cable / transceiver healthy".

$ mlxlink -d /dev/mst/mt4129_pciconf0 -m -e -c
Operational Info
----------------
State                              : Active
Physical state                     : LinkUp
Speed                              : NDR
Width                              : 4x
FEC                                : Standard_RS-FEC
Loopback Mode                      : No Loopback
Auto Negotiation                   : ON

Supported Info
--------------
Enabled Link Speed (Ext.)          : 0x000007e0 (NDR,HDR,EDR,FDR,QDR,DDR,SDR)
Supported Cable Speed (Ext.)       : 0x00000400 (NDR)

Troubleshooting Info
--------------------
Status Opcode                      : 0
Group Opcode                       : N/A
Recommendation                     : No issue was observed.

Tool Information
----------------
Firmware Version                   : 28.41.1000
amBER Version                      : 2.21
MFT Version                        : mft 4.30.0-139

Physical Counters and BER Info
------------------------------
Time Since Last Clear [Min]        : 12345.67
Effective Physical Errors          : 0
Effective Physical BER             : 15E-255
Raw Physical BER                   : 5E-7
Raw Physical Errors Per Lane       : 0,0,0,0
Effective Errors Per Lane          : 0,0,0,0

Eye Opening Info
----------------
Phys Bias Type                     : ...
Height Eye Opening [mV]            : 25,26,25,26
Phase Eye Opening [psec]           : 8,8,8,8

The numbers that matter:

A failing transceiver typically degrades gradually: BER creeps up, eye opening on one lane shrinks. By the time errors are visible to the IB stack, the physical layer has been deteriorating for days.

If Effective Physical Errors is high or growing, action is: swap the cable (cheapest), then the transceiver (next), then the port on the switch, then the HCA card.

Switch port stats

# from the host, query the switch port the HCA is connected to
$ iblinkinfo
CA: hostname-mlx5_0:
       0x0008f1ffffabcdef      4    1[  ]  ==( 4X         200.0 Gbps Active/  LinkUp)==>  17    8[  ] "switch-leaf-1"

$ ibportstate -L 17 8           # switch LID 17, port 8
PortInfo:
# Port info: Lid 17 port 8
LinkState: ............................Active
PhysLinkState: ........................LinkUp
LinkWidthSupported: ...................1X or 4X
LinkWidthEnabled: .....................1X or 4X
LinkWidthActive: ......................4X
LinkSpeedActive: ......................NDR (50 Gbps per lane)

Useful counters from the switch side:

$ perfquery -L 17 8
PortCounters:                                # for switch LID 17, port 8
PortSelect:......................8
CounterSelect:...................0x0000
SymbolErrorCounter:..............0
LinkErrorRecoveryCounter:........0
LinkDownedCounter:...............0           # port flaps
PortRcvErrors:...................0           # bad packets received
PortRcvRemotePhysicalErrors:.....0
PortRcvSwitchRelayErrors:........0           # switch couldn't forward
PortXmitDiscards:................0           # buffer overflow on egress
PortXmitConstraintErrors:........0
PortRcvConstraintErrors:.........0
PortLocalPhysicalErrors:.........0
PortMalformedPktErrors:..........0
PortBufferOverrunErrors:.........0
PortDLIDMappingErrors:...........0
PortVL15Dropped:.................0
PortXmitWait:....................0           # we waited (congestion!) — high = bad

PortXmitDiscards > 0 means the switch dropped packets due to buffer pressure — congestion / lack of PFC. PortXmitWait increasing rapidly = sustained congestion.

For RoCE, switch counters live in your switch vendor's tooling (Cumulus, SONiC, NX-OS) — same concepts, different commands.

Layer 2 — Kernel modules

The canonical 9-module list (see OFED):

$ lsmod | grep -E '^(mlx5_core|mlx5_ib|ib_core|ib_uverbs|ib_cm|rdma_cm|rdma_ucm|ib_umad|ib_ipoib)\s'
mlx5_core            2207744  1 mlx5_ib
mlx5_ib               524288  0
ib_core               573440 11 rdma_cm,ib_ipoib,iscsi_iser,...,ib_uverbs,...,nvidia_peermem
ib_uverbs             192512  2 rdma_ucm,mlx5_ib
ib_cm                 131072  2 rdma_cm,ib_ipoib
rdma_cm                86016  2 ib_iser,rdma_ucm
rdma_ucm               36864  0
ib_umad                32768  4
ib_ipoib              163840  0

Missing one of these = something will fail later. The most common omissions:

• rdma_ucm missing → rdma_cm-based applications (NCCL is one) can't reach /dev/infiniband/rdma_cm.
• ib_umad missing → ibstat returns nothing because management queries can't be issued.
• ib_uverbs missing → no /dev/infiniband/uverbs0..N devices, no userspace verbs at all.

/dev/infiniband listing on a healthy node:

$ ls -la /dev/infiniband
crw-rw-rw- 1 root root 231,  0 issm0
crw-rw-rw- 1 root root 231, 64 rdma_cm
crw-rw-rw- 1 root root 231,  4 ucm0
crw-rw-rw- 1 root root 231,  3 umad0
crw-rw-rw- 1 root root 231,  1 uverbs0
crw-rw-rw- 1 root root 231,  2 uverbs1
...

One uverbs<N> and one umad<N> per HCA port, plus singleton rdma_cm and issm0. If any are missing, the corresponding kernel module isn't loaded.

Layer 3 — Subnet manager (InfiniBand only)

In an IB fabric, something must run a Subnet Manager (SM) to assign LIDs and program the routes. Switch-managed SM (Mellanox switches) is the easiest; OpenSM as a process on a head node also works.

$ sminfo
sminfo: sm lid 1 sm guid 0x506b4b03004f1a5a, activity count 12345 priority 1 state 3 SMINFO_MASTER

State 3 = ACTIVE master. Other states:

If sminfo returns "no SM seen" → no SM in fabric → no LIDs → IB ports stuck in Initializing state, never reach Active. Start an OpenSM on a stable host:

# minimal opensm
$ opensm -B                                  # background, default config
# or with logfile
$ opensm -B -F /etc/opensm/opensm.conf -f /var/log/opensm.log

ibhosts and ibswitches enumerate the fabric:

$ ibhosts
Ca      : 0x506b4b0300abcd01 ports 1 "hostname-1 mlx5_0"
Ca      : 0x506b4b0300abcd02 ports 1 "hostname-2 mlx5_0"
...

$ ibswitches
Switch  : 0x506b4b0300fff001 ports 64 "Mellanox QM9700 leaf-1" base port 0 lid 1 lmc 0

ibdiagnet is the deeper survey:

$ ibdiagnet -ls 25 -lw 4x                    # check all links are 25Gbps × 4 (HDR)
# generates /var/cache/iba/ibdiagnet2/* with link errors, miswiring, etc.

For RoCE, there's no SM — the Ethernet switch does the routing. Skip this layer.

Layer 4 — Userspace verbs

$ ibv_devinfo
hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.41.1000
        node_guid:                      0080:e1ff:feab:cd01
        sys_image_guid:                 0080:e1ff:feab:cd00
        vendor_id:                      0x02c9
        vendor_part_id:                 4129                # CX-7 product ID
        hw_ver:                         0x0
        board_id:                       MT_0000000540
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               17
                        port_lmc:               0x00
                        link_layer:             InfiniBand

What to verify:

• state: PORT_ACTIVE (4) — port is up.
• active_mtu: 4096 (5) — IB path MTU. RoCE uses Ethernet MTU instead. active_mtu < max_mtu means the switch told us a smaller path MTU; check switch config.
• sm_lid non-zero (IB) — port has been assigned by the SM.

For RoCE, list the GIDs:

$ show_gids
DEV     PORT    INDEX   GID                                          IPv4            VER     DEV
mlx5_0  1       0       fe80:0000:0000:0000:0080:e1ff:feab:cd01                      v1      eth0
mlx5_0  1       1       fe80:0000:0000:0000:0080:e1ff:feab:cd01                      v2      eth0
mlx5_0  1       3       0000:0000:0000:0000:0000:ffff:0a00:000a      10.0.0.10       v2      eth0
mlx5_0  1       5       0000:0000:0000:0000:0000:ffff:0a00:000a      10.0.0.10       v2      eth0

For NCCL on RoCE v2 IPv4, you want a v2 entry with an IPv4 address. The index (3 or 5 typically) is what you set in NCCL_IB_GID_INDEX. v1 (legacy) GIDs don't route in any modern switch.

ibv_rc_pingpong — the simplest verbs test

# server
nodeA$ ibv_rc_pingpong -d mlx5_0 -g 3 -i 1
# client
nodeB$ ibv_rc_pingpong -d mlx5_0 -g 3 -i 1 <nodeA-ip>

Output is messages-per-second; just want it to complete without errors. If it does, kernel-to-userspace verbs work, fabric routes between hosts, and basic RC connections terminate. If it fails, the layer above hasn't been built yet.

Layer 5 — Performance: perftest

The bandwidth tests. Build:

$ git clone https://github.com/linux-rdma/perftest
$ cd perftest && ./autogen.sh && ./configure --enable-cuda && make

--enable-cuda is required for --use_cuda flag, which exercises GDR.

Plain ib_send_bw

nodeA$ ib_send_bw -d mlx5_0 -F -R --report_gbits

nodeB$ ib_send_bw -d mlx5_0 -F -R --report_gbits <nodeA-ip>

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      1000           388.45             388.42                0.741
 131072     1000           390.10             390.08                0.372
---------------------------------------------------------------------------------------

400 Gb/s NDR link → expect ~390. 200 Gb/s HDR → ~195. 100 Gb/s EDR → ~96.

If bandwidth is < 85 % of line rate, see "slow but no errors" below.

CUDA-aware ib_send_bw (GDR test)

nodeA$ ib_send_bw -d mlx5_0 --use_cuda=0 -F -R --report_gbits
nodeB$ ib_send_bw -d mlx5_0 --use_cuda=0 -F -R --report_gbits <nodeA-ip>

--use_cuda=0 means "use CUDA device 0 as the buffer source/sink". The data path is GPU → NIC (via peermem) → fabric → NIC → GPU. If this works, GDR is alive end-to-end.

If ib_send_bw works but --use_cuda fails or is much slower:

• peermem isn't loaded (or isn't registered against ib_core),
• ACS is enabled and blocking P2P,
• IOMMU is in iommu=force mode without proper passthrough.

See peermem and ACS.

ib_read_bw, ib_write_bw

ib_send_bw exercises two-sided send/recv (which NCCL uses). ib_read_bw and ib_write_bw exercise one-sided RDMA, which catches a different set of issues (memory protection, address translation, atomic ops). Run all three on a fresh node.

Layer 6 — Application

If everything below works but NCCL fails, see NCCL multi-node failures.

Quick differential: enable NCCL_DEBUG=INFO and look at the bring-up. If NCCL says "Using [no IB devices]" while ibv_devinfo and ib_send_bw from the same shell work, the issue is NCCL's environment (NCCL_IB_HCA filter, container access to /dev/infiniband, etc.).

"Slow but no errors" diagnosis

The hardest scenario. Bandwidth is 70 % of line rate; nothing is logging errors; everything claims healthy. This is almost always PFC pause counters or ECN silently throttling.

PFC pauses

# how often is the receiver telling the sender to stop?
$ ethtool -S enp1s0f0 | grep -E "pause|paused"
     rx_prio0_pause_duration_us: 0
     rx_prio1_pause_duration_us: 0
     rx_prio2_pause_duration_us: 0
     rx_prio3_pause_duration_us: 1234567        # we got told to pause for 1.2 seconds total
     rx_prio4_pause_duration_us: 0
     ...
     tx_prio3_pause_duration_us: 0              # we never told them to pause

     rx_prio3_pause_storm_warning_events: 5     # the receiver paused us so much it counts as a storm

Some pause is expected on RoCE (that's the point of PFC). But:

• High rx_*pause_duration = we are being throttled. Either we're sending too fast (legitimate; investigate workload) or the receiver's port is congested (shared with other traffic, or buffer too small).
• pause_storm_warning_events — the receiver had to pause for very long stretches. This points to a buffer-sized problem on the switch side.

# also check whether PFC is even enabled
$ dcb pfc show dev enp1s0f0
pfc-cap 8
delay 65535
pfc-enabled prio:3
auto-mode off

pfc-enabled prio:3 is what you want for RoCE — priority 3 is the standard PFC class. If empty, PFC isn't on, and you'll see ungraceful drops instead of pauses (faster but loss-prone).

ECN marks

$ ethtool -S enp1s0f0 | grep -iE "ecn|cnp"
     rx_ecn_marked_packets: 12345               # we received marked packets
     tx_cnp_packets: 6789                       # we sent congestion notifications
     rx_cnp_packets: 234                        # we received congestion notifications

CNPs being sent/received are the DCQCN feedback loop in action. A few thousand under load is normal. Hundreds of thousands per second = sustained congestion that DCQCN is fighting; throughput will sag.

QP retransmit counters

$ cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/duplicate_request
0
$ cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/out_of_sequence
0
$ cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_cnp_sent
12345
$ cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/np_ecn_marked_roce_packets
54321
$ cat /sys/class/infiniband/mlx5_0/ports/1/hw_counters/rp_cnp_handled
6789

Non-zero out_of_sequence or duplicate_request = packet drops in flight. PFC should be preventing this; if you see them growing, PFC is misconfigured.

For deeper RoCE / DCB tuning see RoCE and perf-tuning.

Common scenarios decoded

"ib_send_bw works but NCCL doesn't"

Below NCCL is fine; above is broken. Check, in order:

1. lsmod | grep nvidia_peermem — loaded? Refcount > 0 once an RDMA workload runs?
2. dmesg | grep peermem — registration succeeded?
3. NCCL log: GPU Direct RDMA Enabled or Disabled?
4. ACS — lspci -vvv | grep ACSCtl. See ACS.
5. NCCL_IB_HCA — does it match what ib_send_bw is using?
6. Container access — does the pod actually have /dev/infiniband?

"perftest crashes"

Couldn't allocate MR: Cannot allocate memory

or

Failed to register memory: ENOMEM

Almost always one of:

1. MLOCK limit too low — ulimit -l returns less than unlimited. RDMA needs to pin big buffers.

   $ cat /etc/security/limits.d/99-rdma.conf
   *  hard  memlock  unlimited
   *  soft  memlock  unlimited
   

   See ulimits.
2. No hugepages — perftest with large message sizes uses hugepages. cat /proc/meminfo | grep Huge. See hugepages.
3. Out of host memory — try a smaller -s size.

"RNR retry exhausted across one specific node pair"

vendor err 81 between nodes A and B specifically; A↔C and B↔C both work fine.

This is a fabric path issue. Probably:

1. The path between A and B traverses a degraded link (cable, transceiver, switch port).
2. The leaf switch they share has buffer pressure on a specific egress port.
3. ECMP hashing routes A↔B onto a saturated path.

Diagnostic:

1. Identify the path with ibtracert (IB) or traceroute on the management overlay (RoCE).
2. Check mlxlink / switch counters on every hop in the path.
3. Swap a cable / transceiver on the suspect segment.
4. Re-run; if it now works, you found it.

In practice the failing pair often shares a leaf-switch port. Re-cabling A and B to different ports on the same leaf is a fast diagnostic.

See also

• RDMA fundamentals
• RoCE
• InfiniBand
• GPUDirect / GDR
• Subnet manager
• OFED
• peermem
• Triage decision tree
• NCCL multi-node failures

External:

• libibverbs man pages: man.archlinux.org/man/extra/libibverbs/
• perftest: github.com/linux-rdma/perftest
• mlxlink troubleshooting: docs.nvidia.com/networking/display/MFTV4