Network validation runbook per speed tier

A new GPU node arrives. The provider says it's ready. Before it sees a tenant's training run, it has to pass a layered validation pass. The pass is gated by network speed tier — a 100G node finishes in 10 minutes; a 400G node takes 45-60 minutes; an 800G node, longer still.

This page is the runbook. It assumes the kernel/driver stack is already installed (NVIDIA driver, MLNX_OFED, kernel cmdline, sysctls — see drivers/nvidia, drivers/ofed, kernel-tuning/grub). What we're validating here is does this node actually deliver the network performance we paid for.

The validation is also the input to the vendor letter at the bottom of this page when results don't match contract.

Pre-flight: ensure the node is ready

Before running any throughput tests:

# All GPUs visible and healthy
nvidia-smi
# 8x H100 / H200 / B200 listed, no XID errors, ECC clean
nvidia-smi -q | grep -E 'ECC|Pending|Volatile' | head -20

# All NICs link up
ibstat | grep -E 'Port|State|Rate'
# State: Active for every port; Rate matches expected (200/400/800)

# All cards visible with correct PCIe link state
sudo lspci -vvv | grep -E 'ConnectX|LnkSta' | head -40

# Module set loaded
lsmod | grep -E 'mlx5|ib_|peermem|nvidia'
# nvidia, nvidia_peermem, mlx5_core, mlx5_ib, ib_uverbs, ib_core, ib_umad, rdma_cm, rdma_ucm

# Kernel cmdline matches expectation
cat /proc/cmdline
# Expected tokens: iommu=pt, default_hugepagesz=1G, hugepages=N, etc.

# Hugepage pool reserved
grep ^Huge /proc/meminfo
# HugePages_Total: 64 (or whatever the spec says)

# Firmware version on NICs
sudo flint -d /dev/mst/mt4129_pciconf0 q
# FW Version: 28.39.1002 (or similar)

If any of these fails, stop and resolve before proceeding. Throughput tests on a node with broken modules give meaningless numbers.

Tier 1: 100 Gbps validation (~10 minutes)

Setup: this node + a known-good peer at 100G. Replace <peer> with the peer's IP.

# 1. Single-port iperf3 baseline
iperf3 -s -p 5201 &
ssh <peer> "iperf3 -c $(hostname) -p 5201 -P 8 -t 30 -i 5"
# Expected: aggregate 90+ Gbps
# Failure threshold: <80 Gbps = node fails

# 2. RDMA point-to-point on each NIC
for hca in mlx5_0 mlx5_1; do
  ib_send_bw -d $hca -F &
  PID=$!
  sleep 1
  ssh <peer> "ib_send_bw -d $hca -F $(hostname)"
  wait $PID
done
# Expected per HCA: BW: 12 GB/s
# Failure threshold: <11 GB/s = node fails

# 3. RDMA with GPU memory (GDR check)
for hca in mlx5_0 mlx5_1; do
  ib_send_bw -d $hca -F --use_cuda=0 &
  sleep 1
  ssh <peer> "ib_send_bw -d $hca -F --use_cuda=0 $(hostname)"
done
# Expected: 11+ GB/s; verifies peermem is loaded and GDR works

# 4. Ping with jumbo
ping -M do -s 8972 -c 5 <peer>
# Expected: 5/5 success, sub-1ms latency

If all four pass: node accepted at tier 1. If any fails, see "When to declare a node good vs go back to provider" below.

Tier 2: 200 Gbps validation (~20 minutes)

Setup: this node + a known-good peer at 200G NDR. 8 NICs typical (4 NUMA × 2 ports).

# 1. NIC sanity — all 8 ports active at 200G
ibstat | grep -E 'Rate|State'
# Rate: 200 / State: Active for every port (8 of them)

# 2. PCIe link state for each NIC
for bdf in $(lspci | grep ConnectX | awk '{print $1}'); do
  echo -n "$bdf: "
  sudo lspci -vvv -s $bdf | grep LnkSta | head -1
done
# Expected: Speed 32GT/s, Width x16 for every NIC (gen5 x16)
# Anything less = degraded; investigate slot / cable / firmware

# 3. iperf3 multi-stream on one NIC
iperf3 -s -p 5201 -B 10.0.10.1 &       # bind to specific NIC IP
ssh <peer> "iperf3 -c 10.0.10.1 -p 5201 -P 16 -t 30 -i 5"
# Expected: 180+ Gbps aggregate
# Failure threshold: <160 Gbps

# 4. RDMA per-NIC throughput
for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F -s 65536 &
  PID=$!
  sleep 1
  ssh <peer> "ib_send_bw -d mlx5_$i -F -s 65536 $(hostname)"
  wait $PID
done
# Expected per HCA: 22-24 GB/s
# Failure threshold per HCA: <20 GB/s

# 5. RDMA with GDR per NIC (GPU - NIC pairing)
# Pair each GPU to its NUMA-local NIC per nvidia-smi topo
nvidia-smi topo -m
# Look for the lowest-cost path between each NIC and each GPU

for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F --use_cuda=$i -s 65536 &
  sleep 1
  ssh <peer> "ib_send_bw -d mlx5_$i -F --use_cuda=$i -s 65536 $(hostname)"
done
# Expected: 21-23 GB/s per pair

# 6. Aggregate test — all 8 NICs simultaneously
# Run 8 ib_send_bw on different ports; sum aggregate
for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F -p $((10001+i)) -s 65536 -D 30 &
done
wait
# Expected aggregate: 175-185 GB/s = 1400-1480 Gbps (8 NICs × 200G)

# 7. nccl-tests inter-node
cat > hosts <<EOF
$(hostname) slots=8
<peer> slots=8
EOF
mpirun -np 16 --hostfile hosts \
  -x NCCL_DEBUG=WARN \
  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  ./build/all_reduce_perf -b 1G -e 8G -f 2 -n 5
# Expected busbw: 22-23 GB/s at large message sizes

Tier 2 takes 20-30 minutes including the nccl-tests run. If any per-NIC test fails, isolate that NIC; if all are below threshold, the issue is platform-wide (BIOS, kernel, OFED).

Tier 3: 400 Gbps validation (~45 minutes)

Setup: this node + a known-good peer at 400G NDR. 8 NICs.

# 1. NIC sanity
ibstat | grep -E 'Rate|State'
# Rate: 400 / State: Active for every port

# 2. PCIe link state — must be gen5 x16
for bdf in $(lspci | grep ConnectX | awk '{print $1}'); do
  speed=$(sudo lspci -vvv -s $bdf | grep LnkSta | awk -F: '{print $2}' | head -1)
  echo "$bdf: $speed"
done
# Every NIC: Speed 32GT/s, Width x16

# 3. CQE compression enabled (mlxconfig)
for nic in $(ls /dev/mst/mt*pciconf*); do
  sudo mlxconfig -d $nic q | grep -i cqe_comp
done
# CQE_COMPRESSION = 1 (or BALANCED)

# 4. NIC ring buffer sized for 400G
for n in $(ls /sys/class/net/ | grep -E 'ens|enp'); do
  ring=$(ethtool -g $n 2>/dev/null | grep -A4 Current | grep RX | awk '{print $2}')
  [ -n "$ring" ] && echo "$n: rx ring $ring"
done
# Expected: 16384 minimum

# 5. NIC channels (queues)
for n in $(ls /sys/class/net/ | grep -E 'ens|enp'); do
  ch=$(ethtool -l $n 2>/dev/null | grep -A4 Current | grep Combined | awk '{print $2}')
  [ -n "$ch" ] && echo "$n: $ch queues"
done
# Expected: 64+ per NIC

# 6. Hugepages
grep ^Huge /proc/meminfo | head -3
# HugePages_Total: 128 (or matches profile spec)

# 7. iperf3 multi-stream — kernel TCP cap
iperf3 -s -p 5201 -B 10.0.10.1 &
ssh <peer> "iperf3 -c 10.0.10.1 -p 5201 -P 32 -t 60 -i 10"
# Expected: 350+ Gbps aggregate
# Failure threshold: <280 Gbps

# 8. RDMA per-NIC at 400G
for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F -s 65536 -t 128 &
  PID=$!
  sleep 1
  ssh <peer> "ib_send_bw -d mlx5_$i -F -s 65536 -t 128 $(hostname)"
  wait $PID
done
# Expected per HCA: 46-48 GB/s
# Failure threshold per HCA: <42 GB/s

# 9. GDR per pair (NIC + NUMA-local GPU)
nvidia-smi topo -m
for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F -s 65536 --use_cuda=$i -t 128 &
  sleep 1
  ssh <peer> "ib_send_bw -d mlx5_$i -F -s 65536 --use_cuda=$i -t 128 $(hostname)"
done
# Expected: 45-47 GB/s per pair

# 10. PFC counters baseline
ethtool -S ens3np0 | grep -i 'pause\|pfc\|drop' > /tmp/pfc-before.txt

# 11. Aggregate — all 8 NICs simultaneously, sustained 60s
for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F -s 65536 -p $((10001+i)) -D 60 -t 128 &
done
wait
# Expected aggregate: 360-380 GB/s (8 × 46-48 GB/s)

ethtool -S ens3np0 | grep -i 'pause\|pfc\|drop' > /tmp/pfc-after.txt
diff /tmp/pfc-before.txt /tmp/pfc-after.txt
# Some pause is fine (PFC working); growth >> 1000/s is suspicious

# 12. nccl-tests inter-node
cat > hosts <<EOF
$(hostname) slots=8
<peer> slots=8
EOF

mpirun -np 16 --hostfile hosts \
  -x NCCL_DEBUG=WARN \
  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  -x NCCL_IB_GID_INDEX=3 \
  -x NCCL_IB_TC=106 \
  ./build/all_reduce_perf -b 1G -e 16G -f 2 -n 5
# Expected busbw at 16G: 45-47 GB/s

# 13. dcgmi diag (GPU-level)
dcgmi diag -r 3
# All checks PASS at level 3 (most thorough)

Tier 3 takes 45-60 minutes including nccl-tests. The dcgmi run can take 10-20 min on its own.

Tier 4: 800 Gbps validation (~60-90 minutes)

Setup: this node + a known-good peer at 800G XDR. 8 NICs typical at this tier (or 4 with two ports each).

Largely the same shape as tier 3 but with bigger expected numbers and stricter PCIe requirements (gen6 x16 or gen5 x32).

# 1. NIC sanity at 800G
ibstat | grep -E 'Rate|State'
# Rate: 800 / State: Active

# 2. PCIe link state — gen6 x16 expected
for bdf in $(lspci | grep ConnectX | awk '{print $1}'); do
  echo -n "$bdf: "
  sudo lspci -vvv -s $bdf | grep LnkSta | head -1
done
# Every NIC: Speed 64GT/s, Width x16    (gen6 x16)

# 3. Hugepages
grep ^Huge /proc/meminfo | head -3
# HugePages_Total: 256

# 4. RDMA per-NIC at 800G
for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F -s 65536 -q 16 -t 256 &
  PID=$!
  sleep 1
  ssh <peer> "ib_send_bw -d mlx5_$i -F -s 65536 -q 16 -t 256 $(hostname)"
  wait $PID
done
# Expected per HCA: 90-95 GB/s
# Failure threshold per HCA: <80 GB/s

# 5. Aggregate sustain — 60s on all NICs
for i in 0 1 2 3 4 5 6 7; do
  ib_send_bw -d mlx5_$i -F -s 65536 -q 16 -p $((10001+i)) -D 60 -t 256 &
done
wait
# Expected aggregate: 720-760 GB/s

# 6. nccl-tests
mpirun -np 16 --hostfile hosts \
  -x NCCL_DEBUG=WARN \
  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  ./build/all_reduce_perf -b 1G -e 32G -f 2 -n 5
# busbw at 32G: 88-92 GB/s expected

Per-tier expected vs measured (table)

The acceptance criteria, summarized:

Marginal results require investigation but don't auto-reject. Reject thresholds are where you should be confident the node has a problem and you can document it for the provider.

When to declare a node "good" vs "go back to provider"

Three buckets after running the validation:

Bucket A: All tests pass at "healthy" thresholds — accept

Document the result, store baselines for ongoing health monitoring, and uncordon the node into the cluster pool.

# Store baseline for future drift detection
mkdir -p /var/lib/cluster-health/$(hostname)
ibstat > /var/lib/cluster-health/$(hostname)/ibstat.baseline
ethtool -S ens3np0 > /var/lib/cluster-health/$(hostname)/counters.baseline
nvidia-smi -q > /var/lib/cluster-health/$(hostname)/nvidia.baseline
sudo lspci -vvv > /var/lib/cluster-health/$(hostname)/lspci.baseline
echo "Validated $(date -u +%FT%TZ)" >> /var/lib/cluster-health/$(hostname)/history.log

Bucket B: One test marginal but rest pass — investigate, then decide

Common patterns:

• One NIC slightly slow: maybe a flaky cable. Replace cable; re-test that NIC. If now passes, accept. If still slow, send back the cable + re-validate. If still slow with new cable, escalate to provider.
• iperf3 fine, RDMA marginal: probably PFC misconfiguration on the switch (provider's responsibility), not the host.
• GDR marginal: peermem rebuild may have failed. Reboot, validate lsmod | grep peermem, re-test.

The principle: marginal can be a host issue or a fabric issue. Don't blame the provider until you've ruled out host-side.

Bucket C: Any test in "reject" — go back to provider

Common reasons for sending a node back:

• PCIe link width or speed wrong (gen3 instead of gen5; x8 instead of x16) — physical/firmware issue
• One or more NICs not at expected rate (100G shows 50G) — hardware or firmware
• Aggregate throughput consistently 30%+ below contract — platform-level issue
• ECC errors on GPUs already at delivery — hardware
• Link instability (port goes Active → Down → Active during validation) — hardware

Document the symptom and the diagnostic data (counters, lspci, dmesg, ethtool output) before the node leaves your hands. Without that data the provider will say "looks fine to us".

Letter template for vendor when measured throughput is below contract

This goes to the provider in plain English prose, the way operations/incident-response shows for vendor comms. No markdown headers, no AI attribution, no marketing language. Paste counters and outputs verbatim.

Hi,

We've validated node gpu-01 (rack R12, asset tag XYZ123) against the 400 Gbps tier acceptance criteria as part of new-node delivery. Several measurements are below contract.

The contract specifies 8 ConnectX-7 NICs at 400 Gbps NDR, PCIe gen5 x16, with line-rate RDMA throughput per port. We expected ib_send_bw per HCA at 46-48 GB/s and aggregate cross-host RDMA throughput at 360+ GB/s on 8 NICs running concurrently against a known-good peer.

Measured results:

ib_send_bw -d mlx5_0 -F -s 65536: 38 GB/s
ib_send_bw -d mlx5_3 -F -s 65536: 41 GB/s
ib_send_bw -d mlx5_5 -F -s 65536: 47 GB/s
ib_send_bw -d mlx5_7 -F -s 65536: 46 GB/s
Aggregate (8 NICs concurrent, 60s sustained): 312 GB/s

mlx5_0 and mlx5_3 are noticeably below the other six NICs. The PCIe link state on mlx5_0 reports as gen4 x16 instead of gen5 x16:

$ sudo lspci -vvv -s 1b:00.0 | grep LnkSta
LnkSta: Speed 16GT/s, Width x16
LnkCap: Speed 32GT/s, Width x16

The other six NICs all show Speed 32GT/s, Width x16. We've reseated the card on mlx5_0 and the symptom persists. The cable was replaced once; no change. We've also verified BIOS PCIe slot configuration is set to gen5 across all populated slots, and the firmware on the NIC is current (FW 28.39.1002).

mlx5_3 shows correct gen5 x16 link state but consistent 41 GB/s — it tracks below mlx5_5 and mlx5_7 by 5-7 GB/s on every test. We don't have a hardware-level explanation; PCIe and firmware look identical.

Could you confirm whether (a) the platform is delivering full gen5 to all 8 slots, and (b) NICs in slots 1b:00 and 86:00 (mlx5_0 and mlx5_3) need replacement or reseating from your side. We've left the node in this configuration so you can run any diagnostics you need from your end. Cluster identifier and support contract attached.

Once these two NICs come up to spec we'll re-validate end-to-end and accept the node into the production pool. Until then it stays cordoned.

Thanks, [name], [role]

The shape: factual symptom → measurement output → what you've already ruled out → specific ask → what you're doing in the meantime. The provider's engineer reads it once and knows what to do.

Checklist — what to capture before the node leaves your hands

Whether you accept the node or send it back, capture this set of artifacts. They become the baseline for ongoing drift detection and the evidence package if there's a contract dispute later.

NODE=$(hostname)
OUT=/var/lib/cluster-health/$NODE/$(date +%FT%T)
mkdir -p $OUT

# Hardware inventory
sudo dmidecode > $OUT/dmidecode.txt
sudo lspci -vvv > $OUT/lspci.txt
sudo lshw -class network -class display > $OUT/lshw.txt

# Kernel + modules
uname -a > $OUT/uname.txt
cat /proc/cmdline > $OUT/cmdline.txt
lsmod > $OUT/lsmod.txt
dpkg -l 2>/dev/null | grep -E 'linux-image|nvidia|mlnx' > $OUT/packages.txt
ofed_info -s > $OUT/ofed-version.txt 2>/dev/null

# NIC + IB state
ibstat > $OUT/ibstat.txt
for n in $(ls /sys/class/net/ | grep -E 'ens|enp'); do
  ethtool $n > $OUT/ethtool.$n.txt 2>/dev/null
  ethtool -g $n >> $OUT/ethtool.$n.txt 2>/dev/null
  ethtool -l $n >> $OUT/ethtool.$n.txt 2>/dev/null
  ethtool -k $n >> $OUT/ethtool.$n.txt 2>/dev/null
  ethtool -S $n > $OUT/counters.$n.txt 2>/dev/null
done

# GPU state
nvidia-smi -q > $OUT/nvidia-smi.txt
nvidia-smi topo -m > $OUT/nvidia-topo.txt

# Firmware
for nic in $(ls /dev/mst/mt*pciconf* 2>/dev/null); do
  sudo flint -d $nic q > $OUT/flint.$(basename $nic).txt 2>/dev/null
done

# Validation test results
echo "=== iperf3 ===" > $OUT/validation.txt
echo "iperf3 results pasted here" >> $OUT/validation.txt
echo "=== ib_send_bw ===" >> $OUT/validation.txt
# (paste each test's output)

# Tar it
tar czf $OUT.tgz -C $(dirname $OUT) $(basename $OUT)

This is your evidence package. Keep it. If three months later the node starts showing degraded performance, you compare current state to the baseline tarball and the diff is the answer.

Common failure modes during validation

One NIC slow; everything else fine

ib_send_bw -d mlx5_0 -F: 38 GB/s   <-- slow
ib_send_bw -d mlx5_1 -F: 47 GB/s
ib_send_bw -d mlx5_2 -F: 47 GB/s
...

Investigation order:

1. lspci -vvv -s <bdf> | grep LnkSta — gen5 x16?
2. Cable rated for the speed?
3. mlxlink output — eye opening, BER?
4. Switch port speed forced wrong?

If 1-4 all check out, swap NIC and re-test.

All NICs ~10% slow

Pattern: every NIC reports ~42 GB/s instead of 47 GB/s. Suggests platform-wide issue:

• Kernel cmdline tunables not applied (check /proc/cmdline matches expectation)
• Hugepage pool not reserved (check HugePages_Total)
• CPU governor on powersave instead of performance
• BIOS power profile set to "balanced" or "energy efficient"
• Wrong NUMA pinning during the test

# Quick checks
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# performance       <-- not powersave

cat /proc/cmdline
# Should include intel_iommu=on iommu=pt default_hugepagesz=1G ...

cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
# Should match GRUB cmdline

cat /sys/devices/system/cpu/intel_pstate/no_turbo
# 0     <-- turbo on

dmesg | grep -i 'thermal\|throttle' | tail
# (should be empty or only boot-time)

iperf3 fine, RDMA slow

iperf3 hits 350 Gbps but ib_send_bw only does 30 GB/s per NIC. Suggests RDMA-specific issue:

• Wrong MR memory locking (limits)
• Wrong GID index
• PFC misconfigured on the switch — paused traffic

ulimit -l                           # unlimited expected
cat /etc/security/limits.d/99-rdma.conf
# *    soft    memlock    unlimited

show_gids                           # verify GID v2 IPv4 is selected for RoCE

ethtool -S ens3np0 | grep -i pause  # PFC pauses growing means switch isn't draining

nvidia-smi missing GPUs

nvidia-smi
# 7 of 8 GPUs listed   <-- one missing!

dmesg | grep -i 'xid\|nvidia'
# NVRM: Xid 79: GPU has fallen off the bus

A GPU "fell off the bus" — PCIe link to GPU lost. Causes:

• Bad PCIe slot or riser
• Bad PSU rail to that GPU
• Thermal trip
• Firmware bug

This is a hardware issue. Send the node back — don't accept with 7/8 GPUs.

Link flaps during validation

journalctl -k --since=-1h | grep -iE 'mlx5|link.*down|link.*up'
# mlx5_core 0000:1b:00.0: Port module event[error]: module 0, Cable error, Bus stuck(I2C or data shorted)
# mlx5_core 0000:1b:00.0: Port module event[unplugged]: module 0
# mlx5_core 0000:1b:00.0: Port module event[plugged_enabled]: module 0

A port flapping during validation is a symptom of bad cable, bad transceiver, or bad cage. Replace the cable; if it persists, send back the node.

dcgmi diag failures

dcgmi diag -r 3
# Test results:
#   Memory:       FAIL on GPU 3

A GPU failing dcgmi memory test = ECC issue or memory hardware fault. Do not accept — send the node back.

Re-validation after maintenance / cable swap / firmware update

The same checklist runs after any of:

• NIC replaced
• Cable replaced
• Switch firmware update
• BIOS update
• Kernel upgrade
• OFED version change
• Major NVIDIA driver upgrade

Drift detection: diff against the stored baseline.

NODE=$(hostname)
OLD=/var/lib/cluster-health/$NODE/$(ls /var/lib/cluster-health/$NODE/ | tail -2 | head -1)
NEW=/var/lib/cluster-health/$NODE/$(ls /var/lib/cluster-health/$NODE/ | tail -1)

diff $OLD/lspci.txt $NEW/lspci.txt | head -30
diff $OLD/ibstat.txt $NEW/ibstat.txt
diff $OLD/cmdline.txt $NEW/cmdline.txt

Any diff in cmdline.txt or lspci.txt LnkSta is a red flag — compare it against expected change.

Validation cadence after acceptance

Once a node is in production, periodic re-validation catches silent regressions:

The full per-tier validation lives in this runbook; the daily/weekly subsets live in operations/health-check-runbook.

See also

• Networking: perf tuning — runtime tuning that pairs with this validation
• Network speed tiers — the per-tier tuning regime that produces these numbers
• GRUB cmdline — boot params
• Kernel versions — which kernels support which features
• Drivers: NVIDIA — DKMS, peermem, secure boot
• Drivers: OFED — kernel-version matrix, persistent module loading
• Operations: health-check runbook — daily/weekly cadence
• Operations: incident response — escalation, vendor comms templates
• Operations: RDMA debugging — when validation fails

External:

• NVIDIA Mellanox Performance Tuning Guide
• perftest documentation: github.com/linux-rdma/perftest
• nccl-tests: github.com/NVIDIA/nccl-tests
• DCGM diagnostics: docs.nvidia.com/datacenter/dcgm/