NCCL multi-node: tuning, topology, and the failure modes

NCCL is the library that turns "I have 32 GPUs across 4 nodes" into "an allreduce that runs at 363 GB/s." It hides a lot of complexity — RDMA, GDR, ring construction, NVLink, NVSwitch — behind a few collective primitives. When it works, you don't think about it. When it doesn't, the error message is "unhandled cuda error" and you go looking through three layers of abstraction to figure out which one broke.

This page is the operator's view of NCCL. What it does, the env vars worth knowing, how to read the logs, and the failure modes mapped to root causes.

What NCCL does

The collective ops:

For each op, NCCL picks a topology (ring, tree, CollNet) and breaks the buffer into chunks that flow along that topology.

Topology: ring vs tree vs CollNet

NCCL picks the topology automatically based on message size and available transports. Threshold is around 1 MB — below that, tree wins; above, ring wins. You can force with NCCL_PROTO= (LL, LL128, Simple) and NCCL_ALGO= (Ring, Tree, CollNet, NVLS) but rarely should.

For an 8-node × 8-GPU job with 8 NICs per node:

• Intra-node communication uses NVLink + NVSwitch (900 GB/s on H100, 1.8 TB/s on B200).
• Inter-node communication uses the 8 RoCE/IB NICs, with one ring per NIC (so 8 parallel rings at 400 Gb/s each = 3.2 Tb/s aggregate).
• The actual ring path: GPU0 → NIC0 (node A) → NIC0 (node B) → GPU0 (node B) → ... → GPU0 (node A again).

This is why rail binding is everything.

Rail binding

A "rail" is a stripe of identical NICs across nodes. Rail 0 = NIC 0 on every node. Rail 1 = NIC 1 on every node. Rails are typically wired through separate leaf switches so that traffic on rail 0 stays on rail 0.

The rule: GPU N talks through NIC N. GPU 0 and NIC 0 must be on the same PCIe root complex / same NUMA domain. If GPU 0 sends through NIC 4 instead, every byte goes across the inter-socket UPI/Infinity Fabric — adds 30+ µs of latency, halves bandwidth.

Verify with nvidia-smi topo -m:

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7
GPU0     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS
GPU1    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS
...

PIX = same PCIe switch (best). NODE = same NUMA node (OK). SYS = cross-socket (bad). The ideal mapping is the diagonal: GPU N → NIC N at PIX or NODE.

NCCL builds its rail map automatically from this matrix (when running on a recognized topology like DGX). On non-DGX boxes, you may need to hint with NCCL_TOPO_FILE (see below).

The env vars worth knowing

Generating an NCCL_TOPO_FILE

For DGX H100/H200/B200 boxes, NCCL ships canned topology files. For OEM HGX boxes (Supermicro, Dell, etc.) the autodetection is usually right but worth verifying.

A topology XML excerpt:

<system version="1">
  <cpu numaid="0" affinity="00ff" arch="x86_64" vendor="GenuineIntel">
    <pci busid="0000:18:00.0" class="0x060400" link_speed="32 GT/s" link_width="16">
      <pci busid="0000:1a:00.0" class="0x030200" link_speed="32 GT/s" link_width="16">
        <gpu dev="0" sm="90" rank="0" gdr="1"/>
      </pci>
      <pci busid="0000:1b:00.0" class="0x020000" link_speed="32 GT/s" link_width="16">
        <nic>
          <net name="mlx5_0" dev="0" speed="400000" port="1" guid="0xa288c2fffeabcdef"/>
        </nic>
      </pci>
    </pci>
  </cpu>
  ...
</system>

Generate one from your box (NCCL prints the inferred topology when NCCL_TOPO_DUMP_FILE=/tmp/topo.xml is set):

$ NCCL_TOPO_DUMP_FILE=/tmp/topo.xml NCCL_DEBUG=INFO ./all_reduce_perf -b 1M -e 1G -f 2 -g 8

Inspect, edit if needed, then deploy with NCCL_TOPO_FILE=/etc/nccl/topo.xml.

What NCCL_DEBUG=INFO actually shows you

A healthy NCCL bringup, annotated:

NCCL INFO Bootstrap : Using eth0:10.0.0.10<0>
                                                 # bootstrap interface for rank rendezvous
NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE [4]mlx5_8:1/RoCE [5]mlx5_9:1/RoCE [6]mlx5_10:1/RoCE [7]mlx5_11:1/RoCE
                                                 # 8 RoCE NICs, mlx5_4-7 excluded (NVSwitch mgmt)
NCCL INFO NET/IB : GPU Direct RDMA Enabled for HCA 0 'mlx5_0' (read 1, write 1)
                                                 # GDR working — line you want
NCCL INFO Channel 00/02 : 0[1a000] -> 1[1b000] [send] via NET/IB/0/GDRDMA
                                                 # ring 0 channel 0: rank 0 sends to rank 1 via NIC 0 with GDR
NCCL INFO Connected all rings
                                                 # ring topology fully connected
NCCL INFO comm 0x55c... rank 0 nranks 16 cudaDev 0 busId 1a000 - Init COMPLETE
                                                 # comm initialized — collective ops can run

When something is wrong, the line that fails tells you the layer:

nccl-tests: the standard benchmark

Build:

$ git clone https://github.com/NVIDIA/nccl-tests
$ cd nccl-tests
$ make MPI=1 MPI_HOME=/usr/local/openmpi CUDA_HOME=/usr/local/cuda NCCL_HOME=/usr/local/nccl

Run multi-node allreduce, 32 GPUs across 4 nodes, message sizes 1MB to 2GB:

$ mpirun -np 32 -hostfile hosts -bind-to none -map-by slot \
    -x NCCL_DEBUG=INFO \
    -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
    -x NCCL_IB_GID_INDEX=3 \
    -x LD_LIBRARY_PATH \
    ./build/all_reduce_perf -b 1M -e 2G -f 2 -g 1

Output (the part you read):

#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)
     1048576        262144     float     sum      -1     45.7   22.95   22.95      0     45.6   22.99   22.99      0
     8388608       2097152     float     sum      -1     91.2   92.05   92.05      0     91.3   91.91   91.91      0
    67108864      16777216     float     sum      -1    316.0  212.37  212.37      0    315.7  212.51  212.51      0
   268435456      67108864     float     sum      -1    892.4  300.78  300.78      0    891.8  300.97  300.97      0
  1073741824     268435456     float     sum      -1   2991.1  359.00  359.00      0   2990.4  359.08  359.08      0
  2147483648     536870912     float     sum      -1   5912.5  363.13  363.13      0   5910.1  363.28  363.28      0
# Avg bus bandwidth                                                193.01

Read busbw, not algbw. busbw is normalized to the per-GPU bandwidth of the underlying fabric — easier to reason about. For a 32-GPU H200 ring on 8x400G NICs per node, ~363 GB/s busbw at 2GB message is healthy.

What busbw you should expect (rough):

If you're more than ~10% below those, something's wrong. Check GDR, rail binding, PFC, MTU.

Common errors mapped to causes

"vendor err 81" / IBV_WC_RETRY_EXC_ERR

NCCL WARN NET/IB : Got completion with error 12, opcode 0, vendor err 81

RC retry exhausted. Causes:

1. PFC misconfigured on RoCE — most common. See RoCE doc.
2. Path MTU mismatch — switch in the middle has lower MTU.
3. Wrong GID — NCCL_IB_GID_INDEX picks a v1 entry, switch only routes v2.
4. Subnet manager change on IB — LIDs being reassigned mid-flight.
5. Link flap — physical layer.

Bump NCCL_IB_TIMEOUT=23 to give the fabric more time, but only as a workaround while debugging — the underlying fabric is broken.

"Connection refused" / "remote process exited"

NCCL WARN NET/IB : Connect to peer rank X failed
NCCL WARN NET/IB : remote process exited or there was a network error

Could be:

• The peer rank crashed earlier (look at that rank's log first).
• Bootstrap socket failed — check NCCL_SOCKET_IFNAME points to a routable interface.
• Firewall blocking RoCE (udp/4791).
• One node never came up (check workload manager logs).

"unhandled cuda error" at COMM_INIT

NCCL nccl_ring.cu:410: unhandled cuda error

This is "something deeper went wrong inside CUDA during ring construction." Real-world causes seen in production:

• NVLink degraded: one link down on one GPU. NCCL builds the intra-node ring assuming all 18 NVLinks, hits the dead link, fails. Check dcgmi nvlink -s. See NVLink doc.
• NVSwitch fabric error: rare, but dmesg | grep nvswitch will show it.
• NVLS enabled on driver/firmware that doesn't fully support it: seen on H200 in some containerized environments — set NCCL_NVLS_ENABLE=0.
• Mismatched CUDA versions across ranks: rebuild against the deployed CUDA.

"Cuda failure 401 'the operation cannot be performed in the present state'"

transport/nvls.cc:157 NCCL WARN Cuda failure 401

NVLink SHARP (NVLS) requires the IMEX daemon and a specific driver. In containerized environments where IMEX isn't propagated, NVLS fails. Workaround: NCCL_NVLS_ENABLE=0. Doesn't lose any RoCE/RDMA performance — only loses the NVSwitch hardware allreduce.

"Ring 0 timeout"

NCCL WARN [Service thread] Error encountered progressing operation=Connect, res=3, closing connection

Bootstrap connection (rank-to-rank rendezvous over TCP) timed out. Usually:

• NCCL_SOCKET_IFNAME points to a non-routable interface.
• Firewall blocking the bootstrap port.
• DNS resolution failing on rank hostnames.
• One of the nodes is in a network-partitioned state — check kubectl get node / ping / kubectl exec reachability across all ranks.

A real example: kubelet on one node was unreachable (dial tcp 10.5.55.8:10250: i/o timeout). All 8 GPUs on that node failed simultaneously with this exact error during a multi-node training job.

"Cuda failure 'invalid argument'"

Generally means the GPU itself is unhealthy. Run dcgmi diag -r 3 on each node — bad ECC counters, a degraded NVLink, or a thermal-throttled GPU all surface as this at NCCL init.

Tuning knobs that actually matter

NCCL_IB_QPS_PER_CONNECTION

Default 1. Bump to 4 or 8 for higher bandwidth on AR-capable fabrics. More QPs = more parallel hardware queues used = higher PCIe / NIC pipeline utilization. Diminishing returns past 8.

NCCL_NSOCKS_PERTHREAD and NCCL_SOCKET_NTHREADS

Only matter for socket transport (when IB/RoCE is disabled). For multi-stream socket throughput, set both to ~4-8. Useless when running over RDMA.

NCCL_BUFFSIZE

Default 4 MB. Raise to 8 MB or 16 MB on high-bandwidth fabrics for larger pipeline depth. Costs GPU memory.

NCCL_PROTO

Default lets NCCL pick. Choices: LL (low-latency, small messages, double bandwidth required), LL128 (newer, lower bandwidth penalty), Simple (RDMA writes; default for large messages). Forcing Simple for >1 MB sometimes helps on iffy fabrics.

NCCL_ALGO

Default lets NCCL pick. Force Tree if you have many small allreduces, Ring for a few big ones, NVLS to use NVSwitch SHARP. Test before forcing in production.

NCCL_IB_TIMEOUT and NCCL_IB_RETRY_CNT

Hardware retry parameters. Default 18 / 7 = ~16ms × 7 retries. Raise to 22 / 7 if you see transient err 81 on a fabric you've validated. Don't use as a permanent fix — investigate the underlying drop instead.

Reference: a production NCCL env block

Tested-good for 4×H200 over 8×400G RoCE:

export NCCL_DEBUG=WARN
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11
export NCCL_IB_GID_INDEX=5
export NCCL_IB_TIMEOUT=23
export NCCL_IB_RETRY_CNT=7
export NCCL_NET_GDR_LEVEL=5
export NCCL_SOCKET_IFNAME=eth0
export NCCL_NVLS_ENABLE=0
export UCX_NET_DEVICES=eth0
export UCX_TLS=tcp

Reasons:

• NCCL_IB_HCA excludes mlx5_4-7 (NVSwitch mgmt devices, no GIDs).
• GID_INDEX=5 points at RoCE v2 IPv4 entry inside macvlan-bound pods.
• NVLS_ENABLE=0 because IMEX isn't always available in containers.
• UCX_* keeps SLURM srun's MPI on TCP and lets NCCL handle RDMA — UCX trying to use mlx5_4-7 fails with "ibv_create_ah failed".

See also

• RDMA fundamentals
• RoCE v2 + DCB/PFC/ECN
• GPUDirect / GDR
• NVLink / NVSwitch
• Network performance tuning