perftest (ib_send_bw, ib_read_bw, ib_write_bw): RDMA verification before NCCL

If nccl-tests fails on a fresh cluster, you've got two things to debug at once: the fabric and the NCCL stack on top of it. That's miserable. The fix is to validate the fabric first, in isolation, with perftest. When perftest is healthy point-to-point and pair-wise, you know NCCL failures are NCCL's problem (topology, env vars, ring construction). When perftest fails, NCCL was never going to work — fix the fabric first.

This page is the per-tool guide: the right invocation for each scenario, expected numbers, and the cookbook of failure messages mapped to root causes. It's the layer below NCCL tests in the validation stack and a key step in the triage decision tree.

Why perftest first

perftest tests one thing at a time. ib_send_bw is "do these two HCAs talk RDMA SEND/RECV at line rate." --use_cuda adds GPU memory + GDR. --qp=4 adds multi-QP scheduling. Each flag is one variable. When something fails, you flip flags one at a time until you find the variable that broke.

NCCL tests, by contrast, exercise everything simultaneously — useful for acceptance, useless for debugging. The discipline:

1. Verify physical / link layer with ibstat, mlxlink. (See RDMA debugging.)
2. Verify verbs layer with perftest without --use_cuda. Pure host-memory RDMA.
3. Verify GPUDirect with perftest with --use_cuda. Same RDMA path, GPU memory at both ends.
4. Verify cross-NUMA with perftest sourced/sinked across NUMA boundaries.
5. Verify multi-QP scaling with --qp=4.
6. Run pair-wise across every node pair. Build a matrix.
7. Only then run NCCL tests.

Skipping steps to "just run NCCL" feels faster but isn't. A failed NCCL run with no perftest baseline gives you nothing to compare against — the dreaded "is it the fabric or the library?" question with no answer.

perftest is from linux-rdma/perftest

Two ways to get it:

• MLNX_OFED bundle — already installed on most Mellanox/NVIDIA-deployed nodes, in /usr/bin/ib_send_bw and friends. Same set of tools.
• From source — when MLNX_OFED isn't installed (e.g., upstream RHEL/Ubuntu with the inbox driver), build from github.com/linux-rdma/perftest:

git clone https://github.com/linux-rdma/perftest
cd perftest
./autogen.sh
./configure --prefix=/usr/local
make -j
sudo make install

Build prerequisites: libibverbs-dev, librdmacm-dev, libpci-dev. For GPU support, build with --enable-cuda (autotools picks it up if nvcc is on PATH and CUDA_HOME is set):

export CUDA_HOME=/usr/local/cuda
./configure --prefix=/usr/local --enable-cuda
make -j

Verify:

ib_send_bw --version
# perftest version: <version>

If --use_cuda flag is missing from ib_send_bw --help, the binary was built without CUDA support — rebuild.

The tools

For NCCL-relevant validation, focus on ib_write_bw (matches the NCCL data plane), ib_send_bw (sanity), and the latency variants for diagnosing tail latency.

Two-sided vs one-sided in plain English

• Two-sided (SEND/RECV): sender posts a SEND, receiver must have already posted a RECV work request to put the data into. Both sides do work. CPU-light on both ends but the receiver has to provide buffers in advance.
• One-sided (RDMA WRITE/READ): sender directly writes into (or reads from) the receiver's memory at a pre-registered virtual address. Receiver does no work — its CPU isn't involved at all. This is what makes "remote DMA" remote.

NCCL uses RDMA WRITE for almost all bulk data transfers. So ib_write_bw is the single most important perftest tool for NCCL pre-validation.

Test recipes (server / client pattern)

Every perftest tool runs as server first, client second. The flags between server and client must match. Server takes no IP; client takes the server's IP as a positional arg.

Standard ib_write_bw

The baseline: host memory both ends, single QP, RC connection, single port.

# On server (the one waiting for connections):
ib_write_bw -d mlx5_0 --report_gbits

# On client:
ib_write_bw -d mlx5_0 --report_gbits 10.0.0.1

Expected output (server side, after client connects):

---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x0a QPN 0x00b1 PSN 0x77a3a4 RKey 0x1801fa VAddr 0x007f5bcf800000
 remote address: LID 0x0c QPN 0x00b1 PSN 0xea22f1 RKey 0x1801fa VAddr 0x007fa8b1800000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000           395.71             395.43               0.754216
---------------------------------------------------------------------------------------

Read:

• Header sanity: Device : mlx5_0, Transport type : IB (or Ethernet for RoCE), Mtu : 4096[B], Link type : IB. Mismatched MTU here = mismatched MTU end-to-end; fix.
• Connection type : RC: Reliable Connection. The default. Other modes (UC, UD) used in specific tests; we cover them below.
• Local/remote addresses: confirms both ends connected.
• Result line: BW average[Gb/sec] is the headline. With --report_gbits, units are gigabits/sec (Gb/s); without, gigabytes/sec (GB/s). Always use --report_gbits so the number maps directly to the link rate.

For a 400G NDR link, ~395 Gb/s average is healthy line rate (~98% of nominal).

With GPU memory (GDR)

GDR = GPUDirect RDMA: data transferred directly between GPU memory and HCA, bypassing host memory. This is what NCCL actually uses. To test it, both server and client allocate buffers on a GPU instead of host memory.

# On server (using GPU 0):
ib_write_bw -d mlx5_0 --report_gbits --use_cuda=0

# On client (using GPU 0):
ib_write_bw -d mlx5_0 --report_gbits --use_cuda=0 10.0.0.1

--use_cuda=N selects the CUDA device index. You typically pick the GPU on the same PCIe root as the HCA (nvidia-smi topo -m shows the mapping).

Expected: ~370-380 Gb/s on a 400G NDR link with healthy GDR. The ~5-10% drop vs host-memory is normal — the DMA path crosses an extra PCIe hop on the GPU side and BAR1 access has its own overhead.

If --use_cuda produces:

• ~50% of host BW: ACS not disabled, or PCIe routing forcing CPU-mediated DMA.
• Failed to register memory: peermem (nvidia_peermem module) not loaded.
• BAR1 mapping failed: BAR1 size too small for the buffer; check nvidia-smi -q -d MEMORY | grep BAR1 (should be 64 GiB on H100/H200/B200).

See GPUDirect for the full peermem/BAR1/IOMMU dependency chain.

Multi-QP

A single QP doesn't always saturate higher-rate links because of per-QP scheduling overhead in the HCA. Multi-QP creates parallel hardware paths.

# On both server and client:
ib_write_bw -d mlx5_0 --report_gbits --use_cuda=0 --qp=4 [server-ip]

--qp=4 opens 4 QPs in parallel. Effects:

On 800G XDR, a single QP can't saturate the link — multi-QP is essential. On 100G EDR, one QP already saturates. NCCL itself runs multi-QP via NCCL_IB_QPS_PER_CONNECTION (default 1, raise to 4 or 8 for AR-capable fabrics).

For small-message rate (Mpps), multi-QP also helps: more parallel ops in flight = more messages per second. Doesn't change bulk bandwidth at large sizes once line rate is saturated.

Connection types

# RC (Reliable Connection) — default. What NCCL uses.
ib_write_bw -d mlx5_0 --connection=RC ...

# UC (Unreliable Connection) — no retransmission.
ib_write_bw -d mlx5_0 --connection=UC ...

# UD (Unreliable Datagram) — connectionless. Smaller MTU.
ib_send_bw -d mlx5_0 --connection=UD ...

For NCCL pre-validation, always use RC (the default). UC/UD are for specific protocol testing — they don't reflect NCCL's behavior.

Bidirectional

-b runs traffic in both directions simultaneously. This stresses HCA full-duplex behavior; should hit ~1.9× the unidirectional number on a healthy HCA.

ib_write_bw -d mlx5_0 --report_gbits -b 10.0.0.1

A 400G NDR link should hit ~750 Gb/s aggregate with -b. If it's only ~400 Gb/s, the HCA or PCIe slot can't sustain full-duplex; investigate PCIe Gen and BIOS config.

Picking the right device and port

Multi-port HCAs (or multi-HCA hosts) make device selection load-bearing:

# List all RDMA devices.
ibv_devices

# Show ports per device, including link state and rate.
ibstat

# Ports on a specific device.
ibstat mlx5_0

# Verify before testing.
ibstat mlx5_0 | grep -E 'State|Rate|Link layer'
# State:  Active
# Rate:   400
# Link layer: InfiniBand

If a port is Initializing or Down, perftest will hang or fail with ibv_modify_qp failed. Fix the link first.

For RoCE, Link layer: Ethernet is correct. For native IB, InfiniBand. The Mellanox card supports both; the link layer is determined by the switch you're plugged into and the port config.

Cross-NUMA test

The point of this test is to verify that PCIe peer-to-peer DMA works across NUMA boundaries — important when the workload places a tensor on GPU 0 (NUMA 0) and its NIC is on NUMA 1, e.g., during a non-ideal NCCL ring construction.

# On server: HCA on NUMA 0, GPU on NUMA 0 — control case.
numactl --cpunodebind=0 --membind=0 \
    ib_write_bw -d mlx5_0 --report_gbits --use_cuda=0

# Same client.
numactl --cpunodebind=0 --membind=0 \
    ib_write_bw -d mlx5_0 --report_gbits --use_cuda=0 10.0.0.1

# Now: HCA on NUMA 0, GPU on NUMA 1 — cross-NUMA.
numactl --cpunodebind=1 --membind=1 \
    ib_write_bw -d mlx5_0 --report_gbits --use_cuda=4 10.0.0.1

Expected: cross-NUMA loses ~10-20% (~310-330 Gb/s on a 400G NDR link). If it drops to <50%, ACS is not disabled or PCIe peer-to-peer is being routed through the CPU.

Latency tests

Bandwidth tests run thousands of iterations and report sustained throughput. Latency tests run a one-byte ping-pong and report half-RTT.

# Server.
ib_write_lat -d mlx5_0

# Client.
ib_write_lat -d mlx5_0 10.0.0.1

Expected (NDR400, RC, single switch hop):

 #bytes     #iterations    t_min[usec]    t_max[usec]    t_typical[usec]    t_avg[usec]
 2          1000           1.13           4.20           1.18               1.19

Healthy half-RTT latency:

If latency is much higher than expected, the path is going through extra hops (look at traceroute for IPoIB, or check SM routing for native IB).

Reading the output, in detail

Every perftest run produces:

1. Header block — device, transport, MTU, connection type, QP count. Sanity-check this matches what you intended.
2. Address exchange — local and remote LID/QPN/PSN/RKey/VAddr. If this section is missing or shows zeros, the connection setup failed.
3. Result line — for _bw tests: BW peak, BW average, MsgRate. For _lat tests: t_min, t_max, t_avg, t_typical, percentiles.

Important fields:

Expected numbers

These are healthy numbers from real fabrics. ±5% is normal variance.

Single-QP RDMA WRITE bandwidth

A single QP doesn't fully saturate XDR 800G — that's a known per-QP HCA scheduling limit. NCCL accommodates this with NCCL_IB_QPS_PER_CONNECTION=4 or higher.

Latency

Multi-QP scaling

ib_write_bw 400G NDR:
   --qp=1   ~370 Gb/s
   --qp=2   ~390 Gb/s
   --qp=4   ~395 Gb/s
   --qp=8   ~395 Gb/s   (saturated)

If --qp=8 is significantly less than --qp=4, the HCA's QP scheduler is misconfigured or contended — rare but seen with very old firmware.

Failure cookbook

Real error messages. Cause. Fix.

Couldn't connect to <ip>

Couldn't connect to 10.0.0.1:18515
Unable to create rdma_cm id

The server isn't listening or the network can't reach it. Causes:

• Server perftest not started.
• Firewall on either end blocking TCP/18515 (default control port).
• Wrong IP — perftest connects via TCP for RDMA setup, IP must be reachable from client.
• For RoCE: ip route get 10.0.0.1 should resolve to the right interface.

ibv_modify_qp failed

ibv_modify_qp failed
ethernet_read_keys: Couldn't read remote address

QP state transition rejected by the HCA. Causes:

• Partition mismatch (IB only): the two ends are in different IB partitions (PKEYs). Both must be in the same partition.
• No SM / unassigned LID (IB only): the subnet manager hasn't assigned a LID to one or both endpoints. Run sminfo to check; restart opensm if you control it.
• Wrong GID (RoCE): GID index doesn't exist or doesn't match the IP family. show_gids to list, pick the right one with --gid_index=N.
• MTU mismatch: one end requesting 4096, intermediate switch only 2048. Use --mtu=2048 to test.

Connection refused

Couldn't connect to 10.0.0.1:18515
Connection refused

Server not started, or started on a different port. Default port 18515; override with -p 18516.

0 Gbps with no error

The test runs to completion but reports 0 Gb/s. Causes:

• Wrong device specified. -d mlx5_2 when mlx5_2 has no IB peer; HCA accepted the QP but data never moved. ibstat mlx5_2 first to verify.
• Link Down on one end. Both ends came up at QP level but the physical port was Down. ibstat | grep State.
• Wrong port. Multi-port HCA, default port 1, but cabled on port 2. Use -i 2.

--use_cuda fails

Couldn't allocate MR

After --use_cuda=0 is added. Causes:

• nvidia_peermem not loaded. lsmod | grep peermem — should show nvidia_peermem. Load with modprobe nvidia_peermem. Add to /etc/modules-load.d/.
• BAR1 too small. nvidia-smi -q -d MEMORY | grep -A2 BAR1 — should be 64 GiB on Hopper/Blackwell. If 256 MiB, BIOS has resizable BAR disabled; fix in BIOS.
• IOMMU blocking peer-to-peer. dmesg | grep -i iommu for blocks. Either disable IOMMU at boot (intel_iommu=off / amd_iommu=off) or set iommu=pt (passthrough) which is the recommended config for GPU clusters.
• ACS enabled. lspci -vvv | grep -i acsctl — anything not all - is bad. Disable in BIOS or with setpci workaround at boot. See GPUDirect.

High variance / inconsistent results

Each run produces wildly different numbers. Causes:

• Switch congestion. Other tenants on shared fabric. Run during a quiet window to confirm.
• ECN backoff. Switch is marking ECN; HCA is backing off. ethtool -S <iface> | grep ecn for counters; mlxlink -d mlx5_0 -m -e -c for symbol errors.
• Thermal throttling on the HCA. Rare but possible. mlxlink reports temperature.
• PCIe link instability. dmesg | grep -i 'pcie\|aer' for AER (Advanced Error Reporting) events. Marginal cable, marginal slot.

Got bad opcode / Got completion with error

Got bad opcode
Got completion with error: opcode 0, vendor_err 81

vendor_err 81 is the famous "RC retry exhausted" error — same as in NCCL. Causes:

• PFC misconfigured on RoCE — packets dropped, retries exhausted. See RoCE.
• Wrong GID for RoCE v2 — switch only routes v2, your GID points at v1.
• IB SM reassigning LIDs mid-flight.
• Physical link flap during the test — mlxlink for symbol errors.

All-pair pre-validation script

For a multi-node cluster, you want to verify every node-pair × every-rail combination. Eight nodes with eight HCAs each = 28 node pairs × 8 rails = 224 tests. Script it.

#!/usr/bin/env bash
# perftest_matrix.sh — run ib_write_bw across all node pairs and rails.
# Outputs a CSV with one row per (node_a, node_b, hca) tuple.
#
# Assumes:
# - SSH from this host to all nodes works (passwordless).
# - All nodes have the same HCA naming (mlx5_0..mlx5_7).
# - perftest is in PATH on every node.

set -uo pipefail

NODES=( nodeA nodeB nodeC nodeD nodeE nodeF nodeG nodeH )
HCAS=( mlx5_0 mlx5_1 mlx5_2 mlx5_3 mlx5_4 mlx5_5 mlx5_6 mlx5_7 )
PORT_BASE=18515

OUT=/tmp/perftest_matrix.csv
echo "node_a,node_b,hca,bw_gbps,status" > "$OUT"

for i in "${!NODES[@]}"; do
    for j in "${!NODES[@]}"; do
        # Each pair tested once (i < j); skip self.
        [[ $i -ge $j ]] && continue
        SRV="${NODES[$i]}"
        CLI="${NODES[$j]}"
        for k in "${!HCAS[@]}"; do
            HCA="${HCAS[$k]}"
            PORT=$((PORT_BASE + k))
            echo ">>> $SRV($HCA) <-> $CLI($HCA) port $PORT"

            # Start server in background.
            ssh "$SRV" "ib_write_bw -d $HCA -p $PORT --report_gbits -F" \
                > "/tmp/srv-$SRV-$HCA.log" 2>&1 &
            SRV_PID=$!
            sleep 2

            # Run client.
            SRV_IP=$(ssh "$SRV" "ip -4 addr show \$(ibdev2netdev | grep $HCA | awk '{print \$5}') | grep inet | awk '{print \$2}' | cut -d/ -f1")
            CLI_OUT=$(ssh "$CLI" "ib_write_bw -d $HCA -p $PORT --report_gbits -F $SRV_IP" 2>&1)
            CLI_RC=$?

            # Parse BW from client output.
            BW=$(echo "$CLI_OUT" | awk '/^[ ]*[0-9]+/{bw=$4} END{print bw+0}')
            STATUS="OK"
            [[ $CLI_RC -ne 0 ]] && STATUS="FAIL"
            (( $(echo "$BW < 350" | bc -l) )) && STATUS="LOW"

            echo "$SRV,$CLI,$HCA,$BW,$STATUS" >> "$OUT"

            # Clean up server.
            kill -TERM $SRV_PID 2>/dev/null
            wait $SRV_PID 2>/dev/null
        done
    done
done

echo "Done. Results in $OUT"
echo
echo "Failures:"
awk -F, '$5 != "OK" {print}' "$OUT"

Run on a control host that has SSH to all cluster nodes. Output is a CSV; pivot it into a node × rail matrix in your spreadsheet of choice. Any cell that's not green means that specific path is broken — bisect from there.

For an 8-node cluster on 400G NDR:

A↔D mlx5_0 at 372 vs the rest at ~394 — that's a 5% drop on one specific path. Investigate: cable on rail 0 between nodes A and D, or a switch port mid-path. Without the matrix you'd never see it.

Tying it back to NCCL

A perftest matrix where every cell is healthy is necessary but not sufficient for healthy NCCL. Things perftest doesn't test:

• Multi-rail simultaneous traffic. perftest tests one rail at a time. NCCL drives all 8 simultaneously.
• Switch buffer behavior under all-rail load. Per-rail tests don't expose buffer exhaustion or PFC pause storms.
• CCL ring construction overhead. NCCL's startup time isn't measured.
• Topology autodetection. Does NCCL build the right rings? Only NCCL tests answer that.

So the flow is:

1. perftest matrix passes everywhere. Every pair, every rail, healthy bandwidth and latency.
2. NCCL sendrecv_perf passes. Single pair through NCCL's stack.
3. NCCL all_reduce_perf 2-node passes. All rails simultaneously, simple ring.
4. NCCL alltoall_perf 2-node passes. Stress the fabric.
5. NCCL full-cluster passes. All hosts, all rails, congestion-aware.

Each step builds on the previous. If step N fails, step N-1 is your baseline; figure out what step N adds that step N-1 didn't have.

See also

• NCCL tests — the next step in validation
• Multi-node validation flow — end-to-end runbook
• RDMA fundamentals
• GPUDirect / GDR
• InfiniBand
• RoCE v2 + DCB/PFC/ECN
• Subnet Manager
• IB switches L2
• RDMA debugging
• Triage decision tree

Common failure modes