NCCL multi-node failure cookbook

NCCL is good at hiding problems and bad at explaining them. When the fabric is healthy, peermem is loaded, ACS is disabled, the GIDs are correct, and rail binding is right, NCCL just works and you forget it exists. When any one of those is wrong, NCCL produces an error message that almost — but not quite — points at the actual problem.

This page is the cookbook for "NCCL is broken, what now?". It assumes you've already walked the triage decision tree and confirmed the node hardware is healthy. The focus here is the NCCL-specific layer: what to set, what the errors mean, and how to validate end-to-end.

Always-on debug flags

The first action when NCCL misbehaves: re-run with full debugging on. Without these, the logs are useless.

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
# or scope it down once you know what you're looking for:
# export NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING

NCCL_DEBUG_SUBSYS=ALL prints everything — bootstrap, transport selection, ring construction, channel-by-channel routes, GDR negotiation, tuning decisions. The output is a few hundred lines per rank, but every line is decision-quality information.

Useful sister flags:

Vendor errors decoded

NCCL inherits its low-level error codes from libibverbs. The most common error pattern is:

NCCL WARN NET/IB : Got completion with error 12, opcode 0, vendor err 81

Two numbers matter: the error (libibverbs WC status) and vendor err (Mellanox-specific subcode). Decoded:

vendor err 81 deep dive

By far the most common production NCCL failure. The codes say "the receiver said RNR enough times that I gave up", but why the receiver wasn't ready varies:

1. PFC not configured on RoCE — switch isn't pausing the sender when its buffers fill, packets get dropped, RNR cascades. See RoCE. Verify with ethtool -S <iface> | grep -E 'prio.*(pause|paused)' — the receiving side should see rx_prio3_pause increment under load. If it doesn't, PFC is dead.
2. Path MTU mismatch — sender uses 4096 B path MTU, switch in the middle has 2048. NCCL's first packet > 2048 gets dropped. ibv_devinfo -v shows the configured active_mtu. On Ethernet, ip link show MTU should be 9000 (jumbo) plus headroom, and switch must agree.
3. Wrong GID — NCCL_IB_GID_INDEX picks an entry the switch doesn't route. RoCE v2 IPv4 is usually GID index 3 or 5; RoCE v1 (legacy) is 0. show_gids lists them.
4. peermem missing — the receiver tries to write to GPU memory but nvidia_peermem isn't loaded. The MR registration fails and the receiver QP enters a wonky state.
5. Partition / P_Key mismatch on IB — sender on 0x7fff, receiver on a non-default P_Key. Subnet manager log will show partition violations.
6. Subnet manager flap on IB — LIDs being reassigned mid-flight.
7. Link flap — physical layer. dmesg | grep -i link and mlxlink -d <dev> -m -c -e for SerDes counters.

Workaround that buys time: NCCL_IB_TIMEOUT=23 NCCL_IB_RETRY_CNT=7. Don't leave it as a permanent fix — the fabric is broken and you're masking it.

vendor err 12 / 113 deep dive

NCCL WARN NET/IB : Got completion with error 12, opcode 0, vendor err 12

Hardware-level retransmit exhausted. Usually:

• Congestion — too much traffic on too few rails. Check rail binding (nvidia-smi topo -m), check that NCCL_IB_HCA isn't pinning everything to one HCA.
• Bad cable / transceiver — mlxlink -d mlx5_0 -m -e -c for SerDes signal quality. Eye opening should be wide; a degrading transceiver narrows it.
• Switch buffer overflow — same as PFC issue; PFC is supposed to slow the sender before this happens.

Swap the cable / transceiver as a first step if you suspect hardware. Real-world: 10 % of err 12 reports trace back to a transceiver that fails under sustained load but passes link-up tests.

vendor err 88 / 90 — protection errors

These almost always mean peermem isn't doing its job. The verbs layer can't translate a GPU pointer to a NIC-DMA-able address.

NCCL WARN NET/IB : Got completion with error 4 (LOC_PROT_ERR), vendor err 88

Check:

$ lsmod | grep nvidia_peermem
nvidia_peermem         16384  3       # refcount > 0 = OK if you're running training
                                       # refcount = 0 = nothing has registered through it

$ cat /proc/driver/nvidia_peermem  # if exposed (older versions)

Also check ACS:

$ lspci -vvv | grep -E "ACSCtl|^[0-9a-f]{4}:"

ACSCtl: SrcValid+ TransBlk+ ReqRedir+ etc. with + = ACS enabled = P2P blocked. You want all -. See ACS.

"Connection refused" / bootstrap failures

NCCL WARN [Service thread] Error encountered progressing operation=Connect, res=3, closing connection
NCCL WARN socketWait: Connection closed by remote peer

NCCL's bootstrap is plain TCP — rank 0 listens, others connect. If this fails, the data plane never gets a chance.

Determine which side is down:

Real example: kubelet on one node was unreachable (dial tcp 10.5.55.8:10250: i/o timeout). All 8 ranks on that node failed simultaneously with bootstrap errors during a multi-node job. The fabric was fine; one node had a network blip.

What to set:

export NCCL_SOCKET_IFNAME=eth0     # or whichever IS the routable interface
export GLOO_SOCKET_IFNAME=eth0     # PyTorch DDP/Gloo bootstrap uses the same name pattern

Avoid IPoIB interfaces (ibX, ib0) for NCCL_SOCKET_IFNAME — bootstrap-over-IPoIB is reliable but slow and more failure-prone than plain Ethernet.

Firewall:

• udp/4791 — RoCE v2 traffic.
• bootstrap port — NCCL picks a random ephemeral. Make sure ephemeral range is open between nodes.

# allow ephemeral source ports out of NCCL's range
sysctl net.ipv4.ip_local_port_range
# default 32768-60999; make sure firewalls allow this range between training nodes

"no devices found" / "no usable HCAs"

NCCL WARN NET/IB : No usable HCAs found
NCCL INFO NET/IB : Using [no IB devices]

NCCL looked for IB devices and found none. In order of likelihood:

1. OFED kernel modules not loaded — lsmod | grep mlx5_ib. Empty? Modules-load.d misconfigured. See OFED.
2. /dev/infiniband doesn't exist — ib_uverbs isn't loaded. modprobe ib_uverbs.
3. NCCL_IB_HCA filter excludes all devices — common when copying env vars between cluster generations. Set explicitly: NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 (8-NIC H100 layout) or use mlx5 to take all matching.
4. Container missing access to /dev/infiniband — pod spec needs hostpath mount or rdma/<dev> resource via the IB device plugin. See K8s pod failures.
5. Inside container, OFED userspace mismatched against host kernel module — rare, but happens with old containers on new hosts. Rebuild the image with matching OFED.

# from inside a pod
$ ls /dev/infiniband
issm0  rdma_cm  ucm0  umad0  uverbs0  ...
$ ibv_devinfo
hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.41.1000
        ...

If ibv_devinfo fails inside the container but works on the host, the issue is the container's access — not the host config.

Hangs (no error, just stuck)

The hardest case. NCCL hangs typically mean:

1. Rank N failed silently and the others are blocked on it — the most common case.
2. Deadlock in the collective — usually a python-side bug (one rank skipped a step, e.g. due to a conditional).
3. NCCL_TIMEOUT not hit — by default, NCCL never times out. The job hangs forever.

Diagnostic recipe:

# Force NCCL to time out and surface stuck ranks:
export NCCL_TIMEOUT=600          # seconds; PyTorch sets this internally too
export NCCL_BLOCKING_WAIT=1      # makes collective ops blocking on host so backtraces are useful
export TORCH_NCCL_BLOCKING_WAIT=1
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1

# If a hang happens, get backtraces from every rank:
$ pip install py-spy
$ for pid in $(pgrep -f 'python.*train'); do
    py-spy dump --pid $pid > /tmp/dump.$(hostname).$pid.txt 2>&1
  done

Read the backtraces. The rank stuck somewhere different from the others is the one that broke. The others are blocked in c10d::ProcessGroupNCCL::* waiting for it.

Common causes when the stuck rank is in a NCCL collective:

• One GPU's NVLink is bad — see triage tree Step 6.
• Memory pressure — NCCL falls back to slow paths under memory pressure; check nvidia-smi on the stuck rank for OOM warnings.
• Stuck QP — fabric error retried until firmware gave up but didn't propagate. dmesg | grep -i mlx5 on the node hosting the stuck rank.

When the stuck rank is in user code (e.g., a checkpoint write):

• One rank is writing to slow storage while others wait at the next collective. Add dist.barrier() around the slow operation, or move IO out of the critical path.

Topology mistakes

NCCL builds its rings from nvidia-smi topo -m plus an optional NCCL_TOPO_FILE. Three classic mistakes:

Rail mis-binding

GPU N should send through NIC N. If your nvidia-smi topo -m shows GPU 0 — NIC 4 = SYS (cross-socket) and GPU 0 — NIC 0 = PIX, but NCCL still picks NIC 4 for GPU 0, your topology hint is wrong.

Verify with NCCL_DEBUG=INFO:

NCCL INFO Channel 00/02 : 0[1a000] -> 1[1b000] [send] via NET/IB/0/GDRDMA
                                                              ^^
                                                              this should match the GPU index

If GPU 0 (busid 1a000) is sending via NET/IB/4/, that's wrong. Force with NCCL_TOPO_FILE or set NCCL_IB_HCA per-rank with CUDA_VISIBLE_DEVICES=N peeking.

NCCL_TOPO_FILE not loaded

$ NCCL_DEBUG=INFO ./all_reduce_perf 2>&1 | grep -i topo
NCCL INFO Topology detection : (none)

(none) means NCCL fell back to autodetection. On non-DGX boxes (Supermicro, Dell HGX, custom), autodetection sometimes guesses wrong. Generate a topo file:

NCCL_TOPO_DUMP_FILE=/tmp/topo.xml NCCL_DEBUG=INFO ./all_reduce_perf -b 1M -e 1G -f 2 -g 8

Inspect, edit if needed, then deploy:

export NCCL_TOPO_FILE=/etc/nccl/topo.xml

Bake it into the container image or DaemonSet.

Wildcard NCCL_IB_HCA

export NCCL_IB_HCA=mlx5      # too permissive — picks up NVSwitch mgmt HCAs too

On H100/H200 boxes, mlx5_4-7 are NVSwitch management devices, not data-plane NICs. They have no GIDs, no port routing, and NCCL trying to use them fails with ibv_create_ah failed. Always whitelist explicitly:

export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11

Validation recipe

Bring up a fresh node, walk this in order. Don't skip steps; each one rules out a layer.

# ────────────────────────────────────────────────────────────────────────
# Step 1: hardware basics
# ────────────────────────────────────────────────────────────────────────
nvidia-smi                                # all GPUs visible, ECC clean
ibstat                                    # all HCAs Active at expected rate
lsmod | grep -E '^(nvidia|mlx5|ib_|rdma_)'  # all expected modules loaded

# ────────────────────────────────────────────────────────────────────────
# Step 2: verbs work locally
# ────────────────────────────────────────────────────────────────────────
ibv_devinfo                               # IB devices reachable from userspace
ibv_rc_pingpong -d mlx5_0                 # loopback, sanity

# ────────────────────────────────────────────────────────────────────────
# Step 3: verbs work between two nodes
# ────────────────────────────────────────────────────────────────────────
# nodeA:
ib_send_bw -d mlx5_0 -F -R --report_gbits
# nodeB:
ib_send_bw -d mlx5_0 -F -R --report_gbits <nodeA-ip>
# expect ~95% of link rate

# ────────────────────────────────────────────────────────────────────────
# Step 4: GDR works (peermem path is alive)
# ────────────────────────────────────────────────────────────────────────
# nodeA (CUDA-aware perftest):
ib_send_bw -d mlx5_0 --use_cuda=0 -F -R --report_gbits
# nodeB:
ib_send_bw -d mlx5_0 --use_cuda=0 -F -R --report_gbits <nodeA-ip>
# expect similar bandwidth to step 3 (within 5%)

# ────────────────────────────────────────────────────────────────────────
# Step 5: NCCL single-node
# ────────────────────────────────────────────────────────────────────────
./build/all_reduce_perf -b 8 -e 1G -f 2 -g 8
# expect busbw at NVLink line rate (~450 GB/s on H100)

# ────────────────────────────────────────────────────────────────────────
# Step 6: NCCL 2-node
# ────────────────────────────────────────────────────────────────────────
mpirun -np 16 -hostfile hosts -bind-to none -map-by slot \
    -x NCCL_DEBUG=INFO \
    -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
    -x NCCL_IB_GID_INDEX=3 \
    -x NCCL_SOCKET_IFNAME=eth0 \
    ./build/all_reduce_perf -b 1M -e 2G -f 2 -g 1
# expect ~360 GB/s busbw on H100/H200 with 8x400G

# ────────────────────────────────────────────────────────────────────────
# Step 7: NCCL full-size
# ────────────────────────────────────────────────────────────────────────
# scale -np up to your job size; expect busbw to stay flat

If step N fails, the problem is in layer N. Don't move forward.

Bring-up validation script (pseudocode)

#!/usr/bin/env bash
# validate_node.sh - run on every fresh GPU node before adding to production pool.
# Exits non-zero on the first failure; meant to be plugged into CI / Ansible.

set -euo pipefail
NODE=$(hostname)
LOG=/tmp/validate.$NODE.log
exec > >(tee -a $LOG) 2>&1

echo "=== $(date) — validating $NODE ==="

# ─── hardware ────────────────────────────────────────────────────────────
nvidia-smi -L | tee /tmp/gpus.list
GPU_COUNT=$(wc -l < /tmp/gpus.list)
[[ $GPU_COUNT -eq 8 ]] || { echo "FAIL: expected 8 GPUs, got $GPU_COUNT"; exit 1; }

# ECC clean?
nvidia-smi --query-gpu=ecc.errors.uncorrected.aggregate.total --format=csv,noheader,nounits \
    | awk '$1 > 0 { print "FAIL: GPU has uncorrectable ECC: " $0; exit 1 }'

# ─── kernel modules ──────────────────────────────────────────────────────
for mod in nvidia nvidia_uvm nvidia_peermem mlx5_core mlx5_ib ib_core ib_uverbs rdma_cm rdma_ucm ib_umad ib_ipoib; do
    lsmod | grep -q "^$mod " || { echo "FAIL: module $mod not loaded"; exit 1; }
done

# ─── HCA links ───────────────────────────────────────────────────────────
ibstat | awk '
    /^CA / { ca=$2 }
    /State:/ { if ($2 != "Active") { print "FAIL: " ca " not Active: " $0; exit 1 } }
    /Rate:/ { if ($2 < 200) { print "FAIL: " ca " rate too low: " $0; exit 1 } }
'

# ─── DCGM quick diag ─────────────────────────────────────────────────────
dcgmi diag -r 1 | tee /tmp/dcgmi.r1
grep -q "Fail" /tmp/dcgmi.r1 && { echo "FAIL: dcgmi diag -r 1 reported failure"; exit 1; }

# ─── single-node NCCL ────────────────────────────────────────────────────
NCCL_DEBUG=WARN ./build/all_reduce_perf -b 1G -e 1G -f 2 -g 8 \
    | tee /tmp/nccl.single
busbw=$(awk '/^[ ]+1073741824/ { print $11; exit }' /tmp/nccl.single)
awk -v bw=$busbw 'BEGIN { if (bw < 400) { print "FAIL: single-node busbw too low: " bw; exit 1 } }'

echo "=== PASS: $NODE clean ==="

For multi-node, hand off to a SLURM/MPI orchestrator that runs nccl-tests across the candidate set and captures busbw for each pair.

Production env block (reference)

Tested-good for H200 over 8×400G RoCE in containers:

export NCCL_DEBUG=WARN
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11
export NCCL_IB_GID_INDEX=5
export NCCL_IB_TIMEOUT=23
export NCCL_IB_RETRY_CNT=7
export NCCL_NET_GDR_LEVEL=5
export NCCL_SOCKET_IFNAME=eth0
export NCCL_NVLS_ENABLE=0
export NCCL_TOPO_FILE=/etc/nccl/topo.xml

For IB clusters, the same block but NCCL_IB_GID_INDEX is irrelevant (IB uses LIDs, not GIDs).

See also

• NCCL multi-node — protocol depth, env-var reference
• Triage decision tree — start here on any failure
• RDMA debugging — when ib_send_bw also fails
• peermem — vendor err 88/90 root cause
• RoCE — vendor err 81 root cause
• ACS — silent GDR killer
• GPUDirect / GDR — the protocol NCCL relies on

External:

• NCCL docs: docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html
• NCCL troubleshooting: docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html
• nccl-tests repo: github.com/NVIDIA/nccl-tests