Multi-node GPU cluster validation: from delivery to production

A new GPU cluster shows up at the cage door looking great. Vendor confirms every node booted; nvidia-smi says all GPUs present; nccl-tests README says "just run all_reduce_perf." If you go from "powered on" straight to "production training job," you'll find out about the bad cable, the missing peermem, the mis-cabled rail, or the firmware drift the same way your customers will: by an expensive job dying at hour eight.

The discipline is to validate in phases, each phase narrowing where a fault could be. By the end of phase 7, every node has been individually sanity-checked, every rail has been pair-tested, every collective pattern has been benchmarked, every fabric path has been stressed under load, and a representative training run has actually completed. Then the cluster goes into the production pool.

This page is the runbook for that flow. Seven phases, pass criteria for each, and a vendor acceptance template at the end.

The phases at a glance

Phases 1-3 can overlap (different nodes); phases 4-7 are sequential. Total wall-clock is ~3 days for a 64-node cluster with parallelization, ~5-7 days end-to-end.

Phase 1 — Per-node sanity

Every node must be individually healthy before any cross-node test runs. This is the same content as health check runbook, but mandatory and exhaustive — not the daily quick-check version.

What's in scope

1. nvidia-smi clean, all expected GPUs present, ECC zeroed, persistence enabled.
2. dcgmi diag -r 3 clean (~10 minutes; runs DCGM's "long" tier including memory, SM, NVLink, integration tests).
3. lsmod shows all required modules: nvidia, nvidia_uvm, nvidia_peermem, mlx5_core, mlx5_ib, ib_core, ib_uverbs, ib_umad, rdma_cm, rdma_ucm.
4. ibstat — all 8 (or however many) HCAs Active at expected rate, expected link layer (IB or Ethernet/RoCE), expected MTU.
5. nvidia-smi topo -m — diagonal mapping clean (GPU N to NIC N at PIX or NODE).
6. BAR1 64 GiB (nvidia-smi -q -d MEMORY | grep -A2 BAR1).
7. ACS disabled (lspci -vvv | grep -i acsctl — all flags cleared).
8. PCIe link width/speed correct on every GPU and HCA (lspci -vv | grep -E 'LnkSta|LnkCap' — Gen5 x16 on Hopper/Blackwell).
9. Firmware versions match across all nodes (HCA, GPU, BMC, BIOS, NVSwitch).
10. CPU governor performance, swap off, transparent hugepages on, nofile ulimit raised.

Pass criteria

Every check passes on every node. A heterogeneous cluster (different driver versions, different firmware, different BIOS settings) is not acceptable even if every individual node looks healthy.

# One-shot validation script per node:
/usr/local/sbin/gpu_node_acceptance.sh

# Compare across all nodes:
for n in $NODES; do
    ssh $n "/usr/local/sbin/gpu_node_acceptance.sh --json" \
        > /tmp/acceptance-$n.json
done

# Diff the JSON outputs — anything different is suspicious.
jq -s 'reduce .[] as $item ({}; . + {($item.hostname): $item})' \
    /tmp/acceptance-*.json > /tmp/all_nodes.json

Common phase-1 discoveries

A cluster where 10% of nodes have at least one phase-1 finding is normal. A cluster where 30% have findings is a vendor problem — escalate.

What to do on failure

Drain the node from the cluster manifest, file a finding with the vendor or the install team, fix it, re-run phase 1 on that node, then proceed with the rest. Don't continue to phase 2 with a known-bad node in the topology — it pollutes results.

Phase 2 — Pair-wise RDMA validation

Now that every node is individually healthy, validate every node × every NIC × every other node. This is the perftest matrix from perftest validation.

What's in scope

1. ib_write_bw between every node pair, every HCA-on-HCA combination, host memory.
2. ib_write_bw --use_cuda for the same matrix, GPU memory + GDR.
3. ib_write_lat for the same matrix, latency.
4. ib_send_bw spot-check (a subset; mostly we trust write_bw).

Pass criteria

For 800G XDR, scale up: ≥ 770 Gb/s on --qp=4, ≥ 750 Gb/s on --use_cuda --qp=4.

Bisection moves on failure

A failing cell in the matrix narrows the fault. Reading the matrix:

Bisecting one bad cell:

1. Verify both ends ibstat — both Active at full rate.
2. Run ib_write_bw again with --report_gbits -F -i 1 (force port 1).
3. Pull mlxlink -d <hca> -m -e -c on both ends — symbol errors on the bad cable.
4. Swap the cable; re-test.

Common phase-2 discoveries

• One cable showing intermittent symbol errors. mlxlink reports it; perftest sometimes hits line rate, sometimes 70%. Replace.
• One port mis-cabled to wrong leaf. Topology says rail 3 should go to leaf 3; one node's rail 3 is plugged into leaf 4. Fabric still works but routing is asymmetric — caught by inter-rail variance.
• MTU misnegotiated on one path. End-to-end MTU drops to 2048 on one rail. ibv_devinfo -v | grep mtu shows 2048 instead of 4096.
• PFC config drift. RoCE only — one switch in the path doesn't have PFC enabled on the right priority. Bandwidth fine in clean tests, collapses under congestion (caught later in phase 4 or 5, but the symptom is hinted at in phase 2 by occasional retransmits in ethtool -S counters).

Phase 3 — Small NCCL (intra-node)

Now we exercise NCCL itself, but inside one node so the fabric isn't a variable. This validates NVLink/NVSwitch, GPU memory, the NCCL library version, and the basic ring construction.

What's in scope

# 2-GPU all_reduce — minimal NVLink test.
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 2

# 4-GPU all_reduce — engages NVSwitch on H100/H200.
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 4

# 8-GPU all_reduce — full NVSwitch fabric.
./build/all_reduce_perf -b 1G -e 4G -f 2 -g 8

# 8-GPU all_gather — also intra-node, different pattern.
./build/all_gather_perf -b 1G -e 4G -f 2 -g 8

# 8-GPU alltoall — NVSwitch full bisection.
./build/alltoall_perf -b 1G -e 4G -f 2 -g 8

Run on every node. Compare results across nodes — they should be within 2% of each other.

Pass criteria (single-node, 4 GiB busbw)

Cross-node consistency: every node's number within 2% of the median.

Common phase-3 discoveries

• One node 50% low on 8-GPU all_reduce. NVSwitch fault or NVLink link down. nvidia-smi nvlink -e for error counters; dmesg | grep -i nvswitch. See NVLink/NVSwitch.
• All nodes 75% of expected. NVLS disabled or NCCL using suboptimal protocol. Verify NCCL_DEBUG=INFO shows NVLS enabled (look for NVLS in init log); if not, check IMEX daemon.
• One node intermittent failures. Specific GPU running hot or marginal NVLink. dcgmi diag -r 3 and nvidia-smi -q -d ECC on that node.
• 2-GPU and 4-GPU OK, 8-GPU bad. A specific NVLink between two GPUs is degraded. NCCL only uses it at 8-GPU. Look at the nvidia-smi nvlink -i 0 -e per-link counters.

What to do on failure

A phase-3 failure on one node = drain that node, RMA the suspected component, re-run on that node only when fixed. Phase 3 failure on every node = NCCL config issue (env vars, LD_LIBRARY_PATH, container image), not hardware.

Phase 4 — 2-node NCCL

This is where the fabric meets NCCL. Every flag, every env var, every routing decision becomes load-bearing.

What's in scope

# hosts.txt:
#   nodeA slots=8
#   nodeB slots=8

# All_reduce — the headline.
mpirun -np 16 --hostfile hosts.txt --bind-to none --map-by slot \
    -x NCCL_DEBUG=INFO \
    -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
    -x NCCL_IB_GID_INDEX=3 \
    -x NCCL_SOCKET_IFNAME=eth0 \
    -x LD_LIBRARY_PATH \
    ./build/all_reduce_perf -b 8 -e 4G -f 2 -g 1 \
    | tee /tmp/phase4-allreduce.log

# All_gather — bisection.
mpirun -np 16 --hostfile hosts.txt --bind-to none --map-by slot \
    -x NCCL_IB_HCA=... -x NCCL_IB_GID_INDEX=3 -x NCCL_SOCKET_IFNAME=eth0 \
    ./build/all_gather_perf -b 8 -e 4G -f 2 -g 1 \
    | tee /tmp/phase4-allgather.log

# Reduce_scatter — symmetric to all_gather.
... ./build/reduce_scatter_perf -b 8 -e 4G -f 2 -g 1

# Alltoall — fabric stress.
... ./build/alltoall_perf -b 8 -e 4G -f 2 -g 1

# Sendrecv — point-to-point baseline.
... ./build/sendrecv_perf -b 1M -e 4G -f 2 -g 1

Run for every pair of nodes (or, if too many nodes, a representative sample: nearest-neighbor pairs + furthest pairs).

Pass criteria (2-node, 16 GPU, 4 GiB busbw)

Variance across node pairs (in the same hardware tier): within 5%.

#wrong must be 0 on every line of every test. This is non-negotiable.

Common phase-4 discoveries

A phase-4 failure where multiple node pairs are bad with the same pattern (e.g., all alltoall 50%) means fabric-wide config issue — likely PFC, ECN, or switch buffer config. Network team's problem. Don't proceed to phase 5 until fixed; the bigger you scale the more this hurts.

Phase 5 — Full-scale NCCL

All nodes in the cluster, simultaneously. This is where spine bandwidth, SM scaling, and the entire control plane get exercised.

What's in scope

# Full-cluster all_reduce — the cluster-acceptance number.
mpirun -np $TOTAL_GPUS --hostfile all-hosts.txt --bind-to none --map-by slot \
    -x NCCL_DEBUG=INFO \
    -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
    -x NCCL_IB_GID_INDEX=3 \
    -x NCCL_SOCKET_IFNAME=eth0 \
    ./build/all_reduce_perf -b 1G -e 4G -f 2 -g 1 \
    | tee /tmp/phase5-allreduce.log

# Full-cluster alltoall — the canary.
mpirun -np $TOTAL_GPUS --hostfile all-hosts.txt --bind-to none --map-by slot \
    -x NCCL_IB_HCA=... -x NCCL_IB_GID_INDEX=3 -x NCCL_SOCKET_IFNAME=eth0 \
    ./build/alltoall_perf -b 1G -e 4G -f 2 -g 1 \
    | tee /tmp/phase5-alltoall.log

# Tree algorithm specifically (helps surface SM/control plane issues).
mpirun ... -x NCCL_ALGO=Tree ./build/all_reduce_perf -b 1M -e 1G -f 2 -g 1

Pass criteria (full-scale, 4 GiB busbw)

For an N-node H100 cluster on 8× NDR400 dual-rail, healthy busbw at full scale should be within 5% of the 2-node number from phase 4. Full-scale should not be slower than 2-node.

If your N-node number is meaningfully worse than your 2-node number, you have:

• Spine oversubscription (insufficient spine bandwidth for full bisection at scale).
• Asymmetric fat-tree routing (some flows take longer paths than others).
• Adaptive routing not enabled, or enabled badly.
• SM-side rate-limiting (rare on modern fabrics).

Variance across runs (run the same test 5 times in a row): ≤ 3% standard deviation. Higher variance at scale = fabric not converged on a stable routing decision; talk to network team.

Common phase-5 discoveries

• Per-node stragglers within rank reports. NCCL_DEBUG logs show one rank consistently lagging. Bisect to that node — phase 1 missed something, or it's marginal hardware that only shows up under full-cluster pressure.
• alltoall collapses at scale. Phase 4 alltoall was 90% but phase 5 alltoall is 50%. Almost always switch buffering — buffers don't fit the all-to-all pattern at scale. Network team.
• Boot-up takes minutes. NCCL bootstrap has to set up rank-to-rank TCP connections; if your Slurm or MPI launcher is slow, this dominates short jobs but not real training. Optimize PMIx config.
• One iteration in 100 spikes. Investigate that timestamp — often correlates with switch firmware doing background tasks (BFD, SNMP polling). Talk to network team about scheduling those out of acceptance windows.

What to do on failure

A phase-5 failure where phase 4 was healthy almost always points to fabric (spine) issues. The remediation is on the network team's plate; your job is to provide the data:

1. The specific test command and full output.
2. NCCL_DEBUG logs showing the rings constructed.
3. mlxlink counters from every NIC immediately after the test (PFC, ECN, symbol errors).
4. Switch port counters from the network team's tooling (correlate to your run timestamp).

Phase 6 — Real workload smoke

NCCL tests prove the fabric. Real workload tests prove the system: the dataloader, the checkpointing, the Slurm/k8s plumbing, the storage backend, the monitoring, the failure handling.

What's in scope

A few iterations (not a full epoch) of a representative workload. Options:

1. A small known model. GPT-NeoX 1.3B for ~100 steps. Or a 7B Llama-style model with FSDP for ~50 steps. Anything that uses the collective primitives and checkpoints once.
2. Megatron-LM --exit-interval 100. Pretrains for 100 steps then exits cleanly. Tests TP, PP, DP collectives all together.
3. DeepSpeed example. ZeRO-3 with offload, ~50 steps.
4. A custom workload. If the cluster is for a specific customer, run their actual training script — best-case scenario for catching workload-specific config bugs.

The training run should:

• Read training data from the actual production storage (not local disk).
• Write checkpoints to the actual production storage.
• Use the actual production scheduler (Slurm or Kubernetes batch).
• Use the actual production container image (or, if bare-metal, the actual production CUDA/NCCL/Python versions).

Pass criteria

Common phase-6 discoveries

• Dataloader bottleneck. GPU utilization 60-70% instead of 95%. Storage backend (Weka, Lustre, NFS) saturated, dataloader workers undersized, or network from compute to storage saturated.
• Checkpoint write hangs the cluster. Synchronous checkpointing pegs storage; collective sync after checkpoint times out. Tune checkpoint frequency or use async checkpointing.
• Mid-run NCCL timeout on one rank. That rank crashed silently; usually OOM, sometimes hardware. dmesg on that node.
• OOM at specific batch size. Memory math wrong for this configuration; reduce micro-batch or increase TP.
• Slow startup. Container pull is slow, or Slurm prolog is slow, or k8s pod scheduling is slow. Optimize before promoting to production.

What to do on failure

Phase 6 failures are almost always configuration, not hardware (since phases 1-5 covered the hardware). Triage by layer:

1. Workload layer: is the model definition correct? batch size? TP/PP/DP layout?
2. Framework layer: PyTorch / DeepSpeed / Megatron version compatibility.
3. NCCL layer: env vars correct in the production container?
4. System layer: ulimits, hugepages, scheduler queue limits, storage mount options.
5. Hardware layer: drop back to phase 5; re-run NCCL all_reduce. If still healthy, hardware is fine.

Phase 7 — 24-hour soak

Twenty-four hours of continuous load with monitoring. The point is to catch:

• Thermal issues that take an hour of full power to surface.
• Link flap events that the daily MTBF predicts will hit ~1× in 24 hours on a marginal fabric.
• Memory leaks in NCCL, the framework, or the workload.
• Storage backend degradation under sustained load.
• BMC firmware bugs that surface at specific uptime intervals (rare but real).

Recipe

# Run a continuous all_reduce loop on the entire cluster.
# Capture metrics every minute.

cat <<'EOF' > /tmp/soak.sh
#!/usr/bin/env bash
set -uo pipefail
ITER=0
while true; do
    ITER=$((ITER + 1))
    echo "=== iteration $ITER, $(date -Is) ==="
    mpirun -np $TOTAL_GPUS --hostfile all-hosts.txt --bind-to none --map-by slot \
        -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
        -x NCCL_IB_GID_INDEX=3 \
        -x NCCL_SOCKET_IFNAME=eth0 \
        ./build/all_reduce_perf -b 4G -e 4G -g 1 -n 100
done
EOF
chmod +x /tmp/soak.sh
nohup /tmp/soak.sh > /tmp/soak.log 2>&1 &

# In parallel: continuous DCGM monitoring.
for n in $NODES; do
    ssh $n "dcgmi dmon -e 1010,150,1009,1011,1012 -d 60000" \
        > /tmp/dcgm-$n.log 2>&1 &
done

# In parallel: continuous mlxlink counters.
for n in $NODES; do
    ssh $n 'while true; do for h in mlx5_{0..7}; do
        mlxlink -d $h -m -e -c >> /tmp/mlxlink-$h.log 2>&1
    done; sleep 300; done' &
done

Let it run for 24 hours. Then analyze.

Pass criteria

Common phase-7 discoveries

• One link flaps once during the 24 hours. Marginal cable. Replace it preemptively even if the cluster keeps running.
• One GPU shows correctable ECC creeping. HBM marginal. Schedule for replacement; should be fine for short jobs but won't make a 30-day training run.
• busbw drops 5% over the run. Thermal throttling. One node is running warmer than the others; check airflow, dust, fan curves.
• Storage backend degrades. Sustained load exposes a storage-side bottleneck not visible in short tests. Customer's problem to fix or yours, depending on contract.
• One node reboots. BMC or kernel panic. Triage: dmesg on next boot, BMC SEL log. Often points to specific firmware.

What to do on failure

Failure during soak is the time to find it. Drain the bad component, re-run a smaller-scale targeted test (just the suspect node, just the suspect rail) to confirm, then RMA / fix / replace.

A successful soak is the gate to production handover. Don't skip it.

Acceptance criteria template

Hand the vendor (or the install team) a one-page document at the start of the engagement with the explicit pass conditions. This makes the acceptance unambiguous: either the cluster meets these numbers, or it doesn't.

GPU CLUSTER ACCEPTANCE CRITERIA — <CLUSTER NAME>
================================================

Configuration:
  Nodes: N
  GPU per node: 8 (H100 SXM5 / H200 SXM5 / B200 SXM)
  HCAs per node: 8 (NDR400 / XDR800)
  Fabric: InfiniBand / RoCE v2

Phase 1 — Per-node sanity
  All N nodes pass /usr/local/sbin/gpu_node_acceptance.sh
  Pass criteria:
    - nvidia-smi: all 8 GPUs present, ECC zeroed, persistence on
    - dcgmi diag -r 3: clean
    - All required modules loaded (nvidia, nvidia_peermem, mlx5_*, ib_*, rdma_*)
    - 8 HCAs Active at expected rate, expected MTU, expected link layer
    - nvidia-smi topo -m: diagonal GPU<->NIC mapping at PIX or NODE
    - BAR1 = 64 GiB on every GPU
    - ACS disabled on every PCIe link
    - Firmware and BIOS versions identical across all nodes

Phase 2 — Pair-wise RDMA
  ib_write_bw between every node pair, every HCA-on-HCA combination.
  Pass criteria (NDR400):
    - host memory:  >= 390 Gb/s
    - --use_cuda:   >= 370 Gb/s
    - --use_cuda --qp=4 (XDR only): >= 770 Gb/s
    - latency, 1 hop: <= 1.5 us typical
    - variance across pairs on same rail: <= 5%

Phase 3 — Intra-node NCCL
  all_reduce_perf, all_gather_perf, alltoall_perf at -g 8.
  Pass criteria, 4 GiB busbw:
    - 8x H100/H200: >= 450 GB/s
    - 8x B200:      >= 850 GB/s
    - cross-node consistency: within 2% of median

Phase 4 — 2-node NCCL
  16-rank tests across every node pair. all_reduce, all_gather,
  reduce_scatter, alltoall, sendrecv at full rail count.
  Pass criteria, 4 GiB busbw:
    - all_reduce on H100 8-rail dual:    >= 460 GB/s
    - all_reduce on H100 8-rail single:  >= 330 GB/s
    - all_reduce on B200 8-rail XDR:     >= 700 GB/s
    - alltoall: within 15% of all_reduce
    - #wrong = 0 on every test

Phase 5 — Full-cluster NCCL
  All N nodes, all collectives, message size sweep 8 B to 4 GiB.
  Pass criteria:
    - busbw at full scale within 5% of 2-node phase 4 number
    - run-to-run variance <= 3% stddev
    - #wrong = 0 on every test

Phase 6 — Real workload smoke
  Provided reference workload (e.g., Megatron-LM 1.3B, 100 steps).
  Pass criteria:
    - Training completes 100 steps without error
    - Checkpoint write+reload succeeds
    - Iteration time variance <= 10%
    - GPU utilization >= 90%
    - No NCCL errors / timeouts

Phase 7 — 24-hour soak
  Continuous all_reduce loop + DCGM + mlxlink monitoring.
  Pass criteria:
    - 0 failed iterations
    - 0 #wrong events
    - busbw stddev <= 3%
    - 0 new uncorrectable ECC errors
    - 0 link flap events
    - 0 unexpected reboots
    - mlxlink BER within manufacturer threshold

Sign-off:
  When all phases pass, customer/operator signs off and cluster
  enters production pool.

  Failures: documented and assigned to vendor or operator;
  cluster does not enter production until all P1 findings closed.

This template, filled in with the specific cluster's hardware tier, becomes the contract. Both sides know the target. Disputes about "is the cluster ready" reduce to "did the numbers in this document hit."

Tooling for the whole flow

A reasonable validation toolkit:

# /usr/local/bin/cluster-validate — wraps everything.

#!/usr/bin/env bash
set -uo pipefail

PHASE=${1:-help}

case "$PHASE" in
    help)
        cat <<EOF
Usage: cluster-validate <phase>

Phases:
  1   per-node sanity (parallel ssh)
  2   perftest matrix
  3   intra-node NCCL on every node
  4   2-node NCCL pairwise
  5   full-cluster NCCL
  6   workload smoke
  7   24-hour soak
  all run all phases sequentially
EOF
        ;;
    1) /opt/validate/phase1_per_node.sh ;;
    2) /opt/validate/phase2_perftest_matrix.sh ;;
    3) /opt/validate/phase3_intranode_nccl.sh ;;
    4) /opt/validate/phase4_pairwise_nccl.sh ;;
    5) /opt/validate/phase5_full_nccl.sh ;;
    6) /opt/validate/phase6_workload.sh ;;
    7) /opt/validate/phase7_soak.sh ;;
    all)
        for p in 1 2 3 4 5 6 7; do
            /usr/local/bin/cluster-validate $p || {
                echo "PHASE $p FAILED — stopping"
                exit 1
            }
        done
        ;;
    *) echo "unknown phase: $PHASE"; exit 2 ;;
esac

Each phase script writes structured output to /var/log/cluster-validate/<phase>/<timestamp>/ with:

• summary.json — pass/fail per check
• raw/ — full output of every command
• metrics.csv — busbw, latency, etc., for trending

This makes re-runs and diffs trivial. After fixing a finding in phase 2, re-run phase 2 only; the structured output lets you confirm the regression area cleared without re-running the whole flow.

See also

• NCCL tests — phases 3, 4, 5 in detail
• perftest validation — phase 2 in detail
• Health check runbook — phase 1 in detail
• NCCL multi-node — library tuning
• RDMA fundamentals
• GPUDirect / GDR
• InfiniBand
• RoCE v2 + DCB/PFC/ECN
• Subnet Manager
• PCIe topology
• Triage decision tree
• Driver / firmware mismatch
• NCCL failure modes
• RDMA debugging

Common failure modes