NCCL tests: what each benchmark proves and how to read the numbers

NCCL tests are how you prove a multi-GPU cluster actually works. Not "the GPUs boot." Not "ibstat says Active." Not even "perftest hits line rate point-to-point." Those are necessary but not sufficient. NCCL tests are sufficient: when a 64-GPU all_reduce_perf lands at the expected bus bandwidth, every layer in the stack — silicon, NVLink, NVSwitch, PCIe, peermem, GDR, IB/RoCE, the leaf and spine switches, the SM, the routing, the congestion handling, the buffer sizing — is working together. When it lands 30% low, one of those layers is broken, and the rest of this page is about figuring out which.

This is the deeper companion to NCCL multi-node. That page covers the library itself: env vars, ring construction, error messages. This one is about the benchmarks: what each one tests, how to read the output, what numbers to expect, and how to bisect a regression.

Why NCCL tests are the gold standard for cluster acceptance

Single-component tests lie by omission. nvidia-smi says the GPU is alive but tells you nothing about NVLink. dcgmi diag exercises the GPU itself but never sends a packet across the fabric. ib_write_bw proves one HCA can drive line rate to one peer but doesn't touch GPU memory unless you remember --use_cuda, and even then it doesn't exercise the all-rails-at-once congestion behavior. iperf doesn't even use RDMA.

NCCL tests stress every layer at the same time:

• GPU compute and memory bandwidth — the reduction operation itself runs on the GPU; the input and output buffers are HBM.
• NVLink and NVSwitch — intra-node ring legs traverse NVLink at full speed.
• PCIe — inter-node legs cross PCIe from GPU to NIC, exercising peer-to-peer DMA.
• IB or RoCE fabric — the actual data plane between nodes.
• Switch buffer sizing and congestion handling — alltoall_perf in particular hammers the fabric in ways no point-to-point test does.
• NCCL's own ring/tree topology decisions — exposes misconfigured NCCL_TOPO_FILE or wrong NCCL_IB_HCA.

Each of the seven benchmarks reveals a different bottleneck. Run them all, in order, before you accept a cluster.

Building nccl-tests

The build is simple when the prerequisites are right; the prerequisites are where people get stuck.

# Prerequisites that must be on PATH / in well-known locations:
# - nvcc (CUDA toolkit)
# - mpicxx (OpenMPI or MPICH)
# - libnccl.so + nccl.h (NCCL library, matching the version your workload uses)

git clone https://github.com/NVIDIA/nccl-tests
cd nccl-tests
make MPI=1 \
     CUDA_HOME=/usr/local/cuda \
     NCCL_HOME=/usr/local/nccl \
     MPI_HOME=/usr/local/openmpi \
     -j

What you get in build/:

all_reduce_perf
all_gather_perf
reduce_scatter_perf
broadcast_perf
reduce_perf
alltoall_perf
sendrecv_perf
hypercube_perf
scatter_perf
gather_perf

For a clean cluster acceptance you only need the first seven. The rest are useful for specific debugging.

Common build failures

If your cluster runs containers, build inside the same image you'll deploy with. NCCL ABI is stable across patch versions but always test before promoting.

The seven benchmarks

Each benchmark exercises a different communication pattern. Read them as: pattern → what fabric property it stresses → what failure mode it catches.

1. all_reduce_perf — the most important benchmark

all_reduce is what every gradient sync in distributed training does. Every rank contributes a tensor; every rank ends up with the elementwise sum of every rank's tensor. That requires both a reduction phase (data is summed as it flows around the ring) and a broadcast phase (the summed result is distributed back). In a ring algorithm, this means every byte traverses the slowest link in the ring 2(n-1)/n times, where n is the number of ranks.

What it stresses: essentially everything. NVLink/NVSwitch intra-node, PCIe across the GPU↔NIC boundary, IB/RoCE inter-node, fabric routing, congestion handling. If all_reduce_perf is healthy across all your sizes, the rest of the suite usually is too.

Bandwidth law: busbw = algbw * 2(n-1)/n. As n grows, 2(n-1)/n approaches 2. So a healthy busbw at 64 ranks is essentially 2 * algbw.

Expected scaling vs message size:

• Tiny (8 B): latency-bound. busbw will be low (a few GB/s); this measures fabric latency, not bandwidth.
• Small (1 MB): per-op overhead still visible. Tree algorithm typically chosen by NCCL.
• Medium (16-64 MB): saturating the ring. busbw climbs.
• Large (1-4 GiB): bandwidth-saturated. Ring algorithm. This is the number you quote when accepting hardware.

Healthy 4 GiB busbw, by topology (see the full table further down).

Common failure patterns:

2. all_gather_perf — fabric bisection bandwidth

all_gather concatenates each rank's tensor at every rank. No reduction; each rank just publishes its chunk and collects everyone else's. There's no compute — it's pure data movement.

What it stresses: the bisection bandwidth of the fabric. Half the bytes from each side need to cross to the other side. If your spine is undersubscribed or your routing is unbalanced, all_gather exposes it before all_reduce does (because there's no reduction to mask the asymmetry).

Bandwidth law: busbw = algbw * (n-1)/n. Approaches algbw as n grows; this is the "1x cap" — you can't beat the fabric's per-link bandwidth.

Use case: tensor-parallel forward pass in transformers (Megatron-style). If you're sizing a cluster for TP, this is the number that matters at the TP-group scale.

Common failure patterns:

• all_reduce_perf healthy, all_gather_perf 30% low → asymmetric fabric routing. One direction is faster than the other; the reduction phase of all_reduce hides it but the pure gather doesn't. Investigate switch routing tables and rail balancing.
• all_gather_perf and reduce_scatter_perf both 50% low → bisection bandwidth literally halved. Likely one leaf-to-spine link down, or one spine member missing.

3. reduce_scatter_perf — symmetric to all_gather

reduce_scatter is the inverse of all_gather: every rank ends up with one chunk of the reduced sum (size total/n). It's the gradient-sync primitive used by ZeRO/FSDP.

What it stresses: identical to all_gather from the fabric's perspective — (n-1)/n of the data crosses bisection. The compute is small (just summation).

Bandwidth law: busbw = algbw * (n-1)/n. 1x bus bandwidth cap.

Tip: all_gather_perf and reduce_scatter_perf should land at roughly the same busbw. If they diverge by more than 5%, suspect compute asymmetry (a slow GPU adding latency to the reduction phase) or a NUMA pinning issue.

4. broadcast_perf — single-source scaling

broadcast sends one rank's tensor to every other rank. Used for distributing initial parameters or any "fan-out from rank 0" pattern.

What it stresses: egress bandwidth of the root rank's NIC, plus the tree/ring topology NCCL builds to fan it out.

Bandwidth law: busbw = algbw * 1. The bottleneck is the root rank — it can only push at its own per-NIC bandwidth, regardless of how many ranks need the data.

What "healthy" looks like: busbw very close to your single-NIC line rate (e.g., ~46 GB/s on a 400 Gb/s link, ~92 GB/s on 800 Gb/s). It does not scale with more ranks; that's expected, not a bug.

Failure pattern to watch for: broadcast_perf significantly below single-NIC line rate. If ib_write_bw --use_cuda hits line rate but broadcast_perf doesn't, suspect NCCL's tree construction (try NCCL_ALGO=Ring to compare) or a slow GPU on the root.

5. reduce_perf — single-destination

The mirror of broadcast: every rank sends its tensor to a single root, which sums them. Used for logging, eval reductions, distributed metric aggregation.

What it stresses: ingress bandwidth at the root, plus the tree NCCL builds to fan-in.

Bandwidth law: busbw = algbw * 1. Same 1x cap as broadcast.

Healthy: busbw close to root NIC line rate, again not scaling with rank count.

6. alltoall_perf — the most demanding benchmark for the fabric

In alltoall, every rank sends a different chunk to every other rank. With n ranks, there are n*(n-1) simultaneous flows. Used by mixture-of-experts (MoE) for expert routing — and notorious for finding fabric problems other benchmarks miss.

What it stresses: switch buffering, congestion handling, ECN/PFC tuning, fairness. Every NIC tries to send to every other NIC at the same time; the fabric has to schedule the chaos. This is where:

• Undersized switch buffers cause head-of-line blocking and bandwidth collapse.
• Unbalanced multipath routing concentrates flows on a subset of links.
• PFC pause storms cripple a previously fast cluster.
• ECN backoff is too aggressive (or not aggressive enough).

Bandwidth law: busbw = algbw * (n-1)/n. 1x cap, like all_gather.

Why it's the canary: I've seen clusters where all_reduce_perf runs at 95% of expected and alltoall_perf runs at 40%. The all_reduce ring traffic is uniform and predictable; the switch's QoS handles it fine. The alltoall pattern bursts onto the fabric and exposes whatever buffer-sizing or PFC-threshold mistake the network team hasn't found yet.

If alltoall is bad and the rest are fine: focus on the network. Talk to the network team. Show them mlxlink counters during the run, especially pfc_paused_seconds and ECN-mark counters. See RoCE and IB switches.

7. sendrecv_perf — point-to-point baseline

A pair of ranks: one sends, one receives. No collective. This is the simplest possible NCCL benchmark and the closest analog to ib_send_bw from inside NCCL.

What it stresses: a single rank pair through NCCL's transport layer. If sendrecv_perf is healthy but all_reduce_perf is not, the problem is in the collective machinery (ring construction, multi-rail coordination), not in the per-pair fabric.

Bandwidth law: busbw = algbw * 1. Just point-to-point.

Use it to: isolate whether a regression is per-pair or collective-specific. Especially useful when bisecting which node pair is the problem in a flat 2-node test.

Algorithm bandwidth vs bus bandwidth

This is the single most misunderstood thing about NCCL output. Skip this section and you'll either celebrate hardware that's actually broken or tear up hardware that's actually fine.

What algbw measures

algbw (algorithm bandwidth) is the application-visible throughput: bytes_per_second computed from the size of the user-provided buffer divided by the wall-clock time of the operation.

algbw = buffer_size / time

That's a useful number for an application developer ("how fast does my all_reduce go?") but it's misleading for a fabric operator. Why? Because different collectives push different amounts of data over the wire for the same buffer size. A broadcast moves (n-1) * buffer_size total bytes; an all_reduce in a ring moves 2 * (n-1) * buffer_size. Same algbw headline number — but the all_reduce is doing twice the fabric work.

What busbw measures

busbw (bus bandwidth) is algbw corrected for the per-collective overhead, so that the resulting number represents the bandwidth at the underlying interconnect, regardless of which collective you ran or how many ranks you used.

The point: a healthy 8-GPU NVSwitch box should report ~470 GB/s busbw on all_reduce_perf and on all_gather_perf and on broadcast_perf (within a few percent). The collective formula corrects for the different patterns. If busbw is healthy on one and unhealthy on another, that's diagnostic.

Formulas

Worked example

8 GPUs, 4 GiB all_reduce, ring algorithm, NCCL reports algbw = 90 GB/s.

busbw = 90 * 2*(8-1)/8
      = 90 * 14/8
      = 90 * 1.75
      = 157.5 GB/s

That's the real per-link bus bandwidth — comparable to other 8-rank tests, comparable to single-NIC RDMA results, comparable across hardware generations.

For 32 GPUs at the same algbw of 90 GB/s:

busbw = 90 * 2*(32-1)/32
      = 90 * 62/32
      = 90 * 1.9375
      = 174.4 GB/s

Same algbw, different busbw — and the busbw is the honest fabric number. That's why operators read the busbw column.

Why this matters in practice

A vendor delivers a 4-node H100 cluster. The acceptance script runs all_reduce_perf -b 4G -e 4G -g 8 -np 32. The output says algbw = 240 GB/s. Is that good?

busbw = 240 * 2*(32-1)/32 = 240 * 1.9375 = 465 GB/s

That's right in the expected band for 4-node H100 with 8x400G NDR per node. Acceptance passes.

Same hardware, different benchmark configuration, vendor B delivers algbw = 240 GB/s on a broadcast_perf instead.

busbw = 240 * 1 = 240 GB/s

That's only ~50% of single-NIC line rate (~46 GB/s × 8 NICs = ~370 GB/s for a unidirectional broadcast). But broadcast is a 1x cap operation — the headline number doesn't scale with rank count. So algbw = 240 GB/s is suspicious and worth investigating, while algbw = 240 GB/s on all_reduce was healthy. Same number, different verdict. Read the busbw, know the law, don't get fooled.

Reading nccl-tests output

A real run, annotated:

# nThread 1 nGpus 1 minBytes 8 maxBytes 4294967296 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
# Using devices
#  Rank  0 Group  0 Pid  12345 on   nodeA device  0 [0x1b] NVIDIA H100 80GB HBM3
#  Rank  1 Group  0 Pid  12346 on   nodeA device  1 [0x43] NVIDIA H100 80GB HBM3
# ... (header continues for all ranks)
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             2     float     sum      -1    21.45    0.00    0.00      0    21.30    0.00    0.00      0
          16             4     float     sum      -1    21.51    0.00    0.00      0    21.40    0.00    0.00      0
          32             8     float     sum      -1    21.58    0.00    0.00      0    21.49    0.00    0.00      0
        ...
     1048576        262144     float     sum      -1    45.71   22.94   42.89      0    45.68   22.96   42.93      0
     8388608       2097152     float     sum      -1    91.20   91.97  171.94      0    91.15   92.03  172.06      0
    67108864      16777216     float     sum      -1   316.04  212.32  396.86      0   315.71  212.54  397.27      0
   268435456      67108864     float     sum      -1   892.41  300.78  562.21      0   891.85  300.97  562.57      0
  1073741824     268435456     float     sum      -1  2991.10  359.00  670.96      0  2990.42  359.08  671.11      0
  4294967296    1073741824     float     sum      -1  9211.17  466.32  871.62      0  9209.84  466.39  871.75      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 405.66

What to read, in order:

1. Header / config dump. Confirms the rank count, GPU model, device IDs, message size range, validation enabled. Mismatches here (e.g., wrong GPU model on one node) abort acceptance.
2. Per-size table. Each row is one message size. Columns:
   • size / count / type — the test parameters.
   • redop / root — reduction op (sum) and root rank (-1 = N/A for symmetric ops).
   • time (μs) — wall time per iteration after warmup.
   • algbw (GB/s) — buffer_size / time.
   • busbw (GB/s) — the corrected per-link number. Read this column.
   • #wrong — number of mismatched elements vs reference. Must be 0. Anything > 0 is data corruption; abort, suspect hardware.
3. Out-of-place vs in-place. Out-of-place uses separate input/output buffers (more typical of training); in-place reuses the same buffer (lower memory but extra synchronization). Should be within ~5% of each other.
4. Final summary. # Avg bus bandwidth is the geometric mean across all sizes. Useful for tracking regressions over time, but don't quote it as the headline — quote the busbw at your largest size (e.g., 4 GiB).

The 0 OK line confirms validation found no incorrect bytes. Any other text there means something was wrong, even if #wrong looked like 0 on every row.

Expected numbers by hardware tier

These are the busbw numbers a healthy cluster should produce on all_reduce_perf at a 4 GiB message size, ring algorithm, with rail binding and GDR all working. They are reference points, not contracts — your network design choices (rail count, switch oversubscription, IB vs RoCE) shift them by ±5-10%.

The big lessons from this table:

• Intra-node H100/H200 caps at ~470 GB/s. That's the NV18 NVSwitch generation. Cross-node never exceeds it on H100 either, because the inter-node legs become the bottleneck.
• Multi-node bandwidth flattens. A healthy 4-node cluster and a healthy 8-node cluster should both hit ~430-465 GB/s. If your 8-node number is below your 4-node number, the spine is undersubscribed or routing is broken.
• Single-rail kills you. One 400G rail per node is ~46 GB/s of fabric; that floors all_reduce at roughly that number. Always multi-rail.
• B200 doubles everything. NVSwitch v4 + XDR800 ~doubles H100 numbers across the board.

If your run lands more than ~10% below the table number, see What does it mean if I see X below.

Designing scaling tests

Don't just run one shape and call it done. Design a test matrix that exposes different bottlenecks.

Intra-node scaling

# 2 GPUs (NVLink only, no NVSwitch traversal)
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 2

# 4 GPUs (engages NVSwitch)
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 4

# 8 GPUs (full NVSwitch fabric)
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 8

Expected: each step should roughly double busbw on H100 (NVSwitch is full bisection). If 8-GPU busbw is less than 4-GPU busbw, you have an NVSwitch fault — see NVLink/NVSwitch.

Inter-node scaling

Run the same operation at 2, 4, and 8 nodes. Healthy busbw plateaus (doesn't keep growing past 2 nodes); the question is whether it stays flat or drops.

# 2 nodes, 16 ranks
mpirun -np 16 -hostfile hosts2.txt ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1

# 4 nodes, 32 ranks
mpirun -np 32 -hostfile hosts4.txt ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1

# 8 nodes, 64 ranks
mpirun -np 64 -hostfile hosts8.txt ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1

Expected: 2-node, 4-node, 8-node all within ~5% of each other. A drop from 2-node to 4-node or 4-node to 8-node points at fabric oversubscription (insufficient spine bandwidth) or asymmetric routing.

Message size sweep

# Latency-bound through bandwidth-bound, factor of 2 between sizes
./build/all_reduce_perf -b 8 -e 4G -f 2 -g 8

Three regions to read:

A healthy curve climbs monotonically until it asymptotes around 1-4 GiB. Anything that drops at large sizes (instead of plateauing) suggests buffer exhaustion, congestion collapse, or a degraded link kicking in under sustained load.

Run recipes

Single-node, 8 GPUs, all_reduce

# Sanity check on one node — no MPI needed.
./build/all_reduce_perf -b 1M -e 4G -f 2 -g 8

# Same, but with topology dump for review.
NCCL_DEBUG=INFO \
NCCL_TOPO_DUMP_FILE=/tmp/topo.xml \
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 8 2>&1 | tee /tmp/nccl-debug.log

Expected on a healthy 8× H100: ~470 GB/s busbw at 4 GiB.

2-node, 16 GPUs, mpirun

# hosts.txt:
#   nodeA slots=8
#   nodeB slots=8

mpirun -np 16 \
    --hostfile hosts.txt \
    --bind-to none --map-by slot \
    -x LD_LIBRARY_PATH \
    -x NCCL_DEBUG=INFO \
    -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
    -x NCCL_IB_GID_INDEX=3 \
    -x NCCL_SOCKET_IFNAME=eth0 \
    ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1

Notes:

• -g 1 here means "1 GPU per MPI rank"; -np 16 because 2 nodes × 8 GPUs = 16 ranks.
• --bind-to none --map-by slot lets NCCL handle GPU↔NIC pinning instead of MPI.
• NCCL_IB_HCA excludes mlx5_4-7 (typical NVSwitch management HCAs on H100/H200 OEM boxes — they have no GIDs and trying to use them errors out).
• NCCL_IB_GID_INDEX must point at the v2 entry on RoCE; check show_gids to find it.

2-node, 16 GPUs, Slurm srun

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --time=00:10:00

export NCCL_DEBUG=INFO
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11
export NCCL_IB_GID_INDEX=3
export NCCL_SOCKET_IFNAME=eth0
export UCX_TLS=tcp

srun --mpi=pmix \
     ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1

Notes:

• --mpi=pmix is the modern launcher; older clusters use --mpi=pmi2. Check srun --mpi=list.
• UCX_TLS=tcp keeps PMIx bootstrap on TCP and lets NCCL own the RDMA transport. UCX trying to use the NVSwitch mgmt HCAs is a classic source of ibv_create_ah failed at startup.

Capturing topology decisions

Always run with NCCL_DEBUG=INFO for the first acceptance run. The log answers:

• Which HCAs did NCCL pick? (NET/IB : Using [0]mlx5_0:1/IB ...)
• Did GDR enable? (GPU Direct RDMA Enabled for HCA 0)
• What rings did it build? (Channel 00/02 : 0[1a000] -> 1[1b000] [send] via NET/IB/0/GDRDMA)
• What protocol? (Connected all rings using LL128)

Pipe to a file, search for Disabled, Failed, WARN, error. See NCCL multi-node for full debug interpretation.

Verifying topology detection

NCCL_TOPO_DUMP_FILE=/tmp/topo.xml \
NCCL_DEBUG=INFO \
./build/all_reduce_perf -b 1G -e 1G -g 8

# Inspect the XML.
cat /tmp/topo.xml | head -100

Look for: every GPU has its expected NIC peer, every NIC reports the right speed (speed="400000" for 400G), the PCIe topology (<pci ...> nesting) matches nvidia-smi topo -m. If the topology is wrong, NCCL is using suboptimal rings — set NCCL_TOPO_FILE=/etc/nccl/topo.xml to a hand-curated version.

Testing custom tuners

NCCL_TUNER_PLUGIN=/opt/nccl/libnccl-tuner-custom.so \
./build/all_reduce_perf -b 1M -e 4G -f 2 -g 8

Used when the default NCCL tuner doesn't pick the right algorithm/protocol for your fabric (rare; needed for some hyperscale topologies). Compare busbw with and without the plugin to verify it helps.

What does it mean if I see X

This is the cookbook. Symptom on the left, root cause on the right, action below.

busbw at ~50% of expected

Most common causes, in order of frequency:

1. One rail down. One HCA Down or LinkDown means NCCL drops to N-1 rails. With 8 rails, going from 8 to 7 is ~12% loss; going from 8 to 4 is ~50% loss. Check ibstat on every node, all ports.
2. Missing peermem. Without nvidia_peermem, NCCL falls back to host-staged copies. ~2x bandwidth loss, silent. Check lsmod | grep peermem and NCCL_DEBUG=INFO for GPU Direct RDMA Disabled.
3. ACS not disabled. Access Control Services on PCIe blocks peer-to-peer DMA between GPU and NIC, forcing IOMMU translation. ~50% bandwidth loss. Check lspci -vvv | grep -i acsctl — should show all flags clear.
4. Rail binding broken. NCCL is sending GPU 0's traffic through NIC 4, traversing the inter-socket link. Verify with nvidia-smi topo -m.
5. Wrong NCCL_IB_HCA. NCCL is using the NVSwitch management HCAs (mlx5_4-7) instead of the data-plane HCAs.

busbw at ~75% of expected

Subtler issues:

1. Minor fabric congestion. Other tenants on shared fabric, or one rail running hotter than the rest.
2. NIC pinned to wrong NUMA. Cross-NUMA DMA adds latency. nvidia-smi topo -m shows NODE (cross-NUMA) instead of PIX (same root complex).
3. Suboptimal protocol. NCCL picked LL when LL128 or Simple would be better. Try NCCL_PROTO=Simple for >1 MB.
4. PCIe Gen3 instead of Gen4/Gen5. Lspci shows link width 16 link speed 8 GT/s (Gen3) instead of 16 GT/s (Gen4) or 32 GT/s (Gen5). Bad cable, marginal slot, or wrong BIOS setting.
5. ECC overhead on a marginal GPU. Soft single-bit errors corrected by HBM but adding latency. Check nvidia-smi -q -d ECC.

#wrong > 0

STOP. Data corruption in a collective is never a software bug; NCCL has been hammered for a decade and the validation is correct. Causes:

1. GPU memory hardware failure — failing HBM cell. Check ECC counters. Reseat or RMA the GPU.
2. NVLink data corruption — bad NVLink cable or NVSwitch port. nvidia-smi nvlink -e for error counters; reseat GPU or NVSwitch tray.
3. IB cable — marginal fiber, dirty connector. mlxlink -d mlx5_0 -m -e -c for symbol error counters.
4. Bad memory in a switch buffer — extremely rare but possible. Network team's job.

Drain the affected node from the cluster immediately. Don't run any production workload on it until the source is identified.

Variance > 10% between runs

Nothing physically broken, but inconsistent:

1. Switch congestion from other tenants. Shared fabric with noisy neighbors.
2. Thermal throttling jitter. GPU clock drops during the run; check nvidia-smi --query-gpu=clocks.current.sm,clocks.throttle_reasons.active --format=csv -l 1.
3. NUMA imbalance. Workload pinned across sockets, with cross-socket memory traffic competing with NCCL's PCIe traffic.
4. Background process noise. Other things running on the node (DCGM, monitoring agents) jittering CPU schedules. Pin NCCL to specific cores.

8-GPU intra-node OK, 16-GPU 2-node bad

The intra-node ring is fine; the inter-node legs are the problem. Layer-by-layer:

1. Per-pair RDMA test. Run ib_write_bw --use_cuda between each pair of nodes. Should hit ~46 GB/s per 400G NIC. Failures here = fabric, not GPU.
2. All rails active. ibstat on every NIC — all 8 ports per node, all Active.
3. MTU mismatch. Path MTU drops to 4096 when one switch in the path is misconfigured. Check ibv_devinfo -v | grep mtu.
4. Wrong GID for RoCE. NCCL_IB_GID_INDEX pointing at v1 entry — switch only routes v2.
5. PFC misconfigured. RoCE without correct PFC is best-effort; congestion collapses bandwidth. See RoCE.

Per-rank stragglers

NCCL output shows one rank consistently slower than the others (visible in NCCL_DEBUG=INFO logs at the per-channel level, or by running per-pair sendrecv tests).

1. Bad GPU. Run dcgmi diag -r 3 on the suspected node — looks for thermal, power, NVLink issues.
2. CPU pinning. That rank's CPU is on a different NUMA from its GPU/NIC.
3. HCA degradation. That rank's NIC is running at a lower speed (e.g., 200G instead of 400G — happens with marginal cables negotiating down). Check ibstat | grep Rate.
4. Storage contention. That rank's host is doing other I/O — iotop to see what.

Inconsistent topology dump

NCCL_TOPO_DUMP_FILE shows different topology on different ranks (different nesting, different NIC counts). Causes:

1. Different OS images across nodes. Driver versions don't match, or modules-load.d is different.
2. One node has a different BIOS. PCIe enumeration order shifted.
3. One node has a different physical layout. Rare — but if a node was rebuilt with a different chassis or the GPUs were reseated in different slots, topology shifts.

A heterogeneous cluster is fixable but every change must be deliberate. See driver/firmware mismatch.

NCCL tuner and topology files

For non-DGX hardware, NCCL's autodetection is usually right but worth verifying. The two knobs:

• NCCL_TOPO_FILE=/etc/nccl/topo.xml — overrides autodetected topology. Useful when the autodetection misses NVSwitch presence, or gets the GPU↔NIC mapping wrong on an OEM HGX board.
• NCCL_TUNER_PLUGIN=/path/to/libnccl-tuner.so — overrides the default decision logic for which algorithm/protocol to use at which message size. Vendors ship custom tuners for their interconnect (e.g., AWS EFA, custom DPU offloads).

For most OEM HGX H100/H200/B200 boxes the autodetection is correct — verify with the topology dump on first bring-up, then leave it alone.

Cluster acceptance template

A minimal acceptance run for a fresh multi-node GPU cluster:

# Phase A: per-node intra-node sanity (run on every node)
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 8
# Pass: ~470 GB/s busbw at 4 GiB on H100/H200, ~880 GB/s on B200.

# Phase B: pair-wise 2-node tests (run for every pair)
mpirun -np 16 -H nodeA:8,nodeB:8 ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1
# Pass: ~330-465 GB/s depending on rail count.

# Phase C: full-cluster all_reduce
mpirun -np $((8*N)) --hostfile all-hosts.txt ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1
# Pass: within 10% of phase B.

# Phase D: alltoall stress
mpirun -np $((8*N)) --hostfile all-hosts.txt ./build/alltoall_perf -b 1G -e 4G -f 2 -g 1
# Pass: within 15% of all_reduce busbw.

# Phase E: 1-hour soak
for i in $(seq 1 60); do
    mpirun -np $((8*N)) --hostfile all-hosts.txt ./build/all_reduce_perf -b 4G -e 4G -g 1 -n 100
done
# Pass: every iteration within ±5% of the median; no #wrong > 0; no warnings in dmesg.

See multi-node validation for the full end-to-end flow.

See also

• NCCL multi-node: tuning, topology, failure modes — the library and env vars
• perftest validation — RDMA verification before NCCL
• Multi-node cluster validation — the full delivery-to-production flow
• RDMA fundamentals
• GPUDirect / GDR
• NVLink / NVSwitch
• PCIe topology
• Network performance tuning
• Health check runbook
• Triage decision tree
• NCCL failure modes

Common failure modes