GPU node health check runbook

A healthy GPU cluster looks healthy because someone proactively keeps it healthy. The choice is between finding a bad node during a planned health check and finding a bad node when a customer's training job dies at step 47000 of 50000. The first costs you a 10-minute cordon; the second costs them a checkpoint restart and you a postmortem.

This page is the canonical schedule of checks. Three tiers — quick, deep, and pre-deploy — by frequency. Each tier has a defined scope, a runtime budget, and an expected outcome. The quick check is meant to be cron'd; the deep check is a planned weekly or monthly job; the pre-deploy is a one-shot before a node enters the production pool.

Schedule overview

The quick check should be running on every node every day. Drift is the most common cause of "the cluster used to work and now doesn't" — a kernel module that didn't auto-load after a reboot, a disk that filled up overnight, an ECC counter that crept up. Catching drift daily prevents incidents.

Quick check (~10 min, read-only)

Cron'd, runs on every node. Output goes to a log file and (ideally) to a metrics system that pages on regression.

What's in it

1. nvidia-smi — driver alive, all GPUs visible, ECC clean, persistence on.
2. dcgmi diag -r 1 — quick health (~30 s; see DCGM).
3. ibstat — all HCA ports Active at expected rate.
4. lsmod | grep nvidia_peermem — peermem loaded.
5. kubectl get node $(hostname) — Ready, no taints we don't expect.
6. Disk space on /, /var/lib/containerd, /var/lib/kubelet — under 80%.
7. dmesg --since '24 hour ago' | grep -iE 'error|xid|nvlink|fault' — no recent kernel errors.
8. systemctl --failed — no failed units.

The script

Save as /usr/local/sbin/gpu_quickcheck.sh. Add a cron entry: 0 */6 * * * /usr/local/sbin/gpu_quickcheck.sh.

#!/usr/bin/env bash
# gpu_quickcheck.sh — daily/6-hourly health check on a GPU node.
# Exits 0 on healthy, 1 on warning, 2 on critical.
# Logs to /var/log/gpu_quickcheck.log; ship to your monitoring backend.

set -uo pipefail
LOG=/var/log/gpu_quickcheck.log
NODE=$(hostname)
EXPECTED_GPUS=${EXPECTED_GPUS:-8}
EXPECTED_HCAS=${EXPECTED_HCAS:-8}
EXPECTED_RATE=${EXPECTED_RATE:-400}     # Gb/s
DISK_THRESHOLD=${DISK_THRESHOLD:-80}    # %

EXIT=0
warn()  { echo "$(date -Is) WARN  $NODE $*" | tee -a "$LOG"; [[ $EXIT -lt 1 ]] && EXIT=1; }
fail()  { echo "$(date -Is) CRIT  $NODE $*" | tee -a "$LOG"; EXIT=2; }
info()  { echo "$(date -Is) INFO  $NODE $*" | tee -a "$LOG"; }

info "=== quickcheck start ==="

# ─── 1. nvidia-smi ────────────────────────────────────────────────────────
if ! command -v nvidia-smi >/dev/null; then
    fail "nvidia-smi not found"
elif ! nvidia-smi >/dev/null 2>&1; then
    fail "nvidia-smi failed (driver/library mismatch?)"
else
    GPU_COUNT=$(nvidia-smi -L | wc -l)
    [[ $GPU_COUNT -eq $EXPECTED_GPUS ]] || fail "expected $EXPECTED_GPUS GPUs, got $GPU_COUNT"

    # ECC: any uncorrectable aggregate errors?
    UNCORR=$(nvidia-smi --query-gpu=ecc.errors.uncorrected.aggregate.total \
        --format=csv,noheader,nounits 2>/dev/null | awk '$1 > 0' | wc -l)
    [[ $UNCORR -eq 0 ]] || fail "uncorrectable ECC on $UNCORR GPU(s)"

    # Persistence mode
    PM=$(nvidia-smi --query-gpu=persistence_mode --format=csv,noheader \
        | sort -u | tr -d '\n')
    [[ "$PM" == "Enabled" ]] || warn "persistence mode not enabled ($PM)"

    # Temperature
    HOT=$(nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits \
        | awk '$1 > 85' | wc -l)
    [[ $HOT -eq 0 ]] || warn "$HOT GPU(s) above 85°C"
fi

# ─── 2. dcgmi diag -r 1 ───────────────────────────────────────────────────
if command -v dcgmi >/dev/null; then
    if dcgmi diag -r 1 2>&1 | tee /tmp/dcgmi.r1.$$ | grep -qE '^\s*Fail' ; then
        fail "dcgmi diag -r 1 reported failures"
        grep -E 'Fail' /tmp/dcgmi.r1.$$ | head -5 >> "$LOG"
    fi
    rm -f /tmp/dcgmi.r1.$$
else
    warn "dcgmi not installed"
fi

# ─── 3. ibstat — all HCA ports Active ─────────────────────────────────────
if command -v ibstat >/dev/null; then
    ACTIVE=$(ibstat | awk '/State:/ && $2 == "Active"' | wc -l)
    [[ $ACTIVE -ge $EXPECTED_HCAS ]] || fail "only $ACTIVE HCA ports Active (expected $EXPECTED_HCAS)"

    # rate check
    SLOW=$(ibstat | awk -v r=$EXPECTED_RATE '/Rate:/ && $2 < r' | wc -l)
    [[ $SLOW -eq 0 ]] || warn "$SLOW HCA(s) below expected rate $EXPECTED_RATE Gb/s"
else
    warn "ibstat not installed"
fi

# ─── 4. peermem loaded ────────────────────────────────────────────────────
lsmod | grep -q '^nvidia_peermem' || fail "nvidia_peermem not loaded"

# ─── 5. node Ready in K8s ─────────────────────────────────────────────────
if command -v kubectl >/dev/null; then
    READY=$(kubectl get node "$NODE" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null)
    [[ "$READY" == "True" ]] || warn "kubectl reports node not Ready: $READY"
fi

# ─── 6. disk space ────────────────────────────────────────────────────────
for mount in / /var/lib/containerd /var/lib/kubelet; do
    [[ -d $mount ]] || continue
    USE=$(df -P "$mount" | awk 'NR==2 {sub("%",""); print $5}')
    if   [[ $USE -ge 95 ]]; then fail "$mount $USE% full"
    elif [[ $USE -ge $DISK_THRESHOLD ]]; then warn "$mount $USE% full"
    fi
done

# ─── 7. recent kernel errors ──────────────────────────────────────────────
RECENT_ERR=$(dmesg --since '24 hour ago' 2>/dev/null \
    | grep -iE 'xid|nvlink|fatal|machine check|fabric manager' | wc -l)
[[ $RECENT_ERR -eq 0 ]] || warn "$RECENT_ERR recent kernel errors in dmesg"

# ─── 8. failed systemd units ──────────────────────────────────────────────
FAILED=$(systemctl --failed --no-legend | wc -l)
[[ $FAILED -eq 0 ]] || warn "$FAILED failed systemd units"

info "=== quickcheck end (exit $EXIT) ==="
exit $EXIT

For Prometheus integration, wrap this in a textfile collector or a node-exporter custom check.

What to alert on

Deep check (~1 hour, drains the node)

Weekly or monthly, on a rolling drain. Costs you GPU-hours, but it's the only way to catch latent issues like a slow-growing transceiver eye-degradation or a flaky NVLink that only fails under sustained load.

What's in it

1. dcgmi diag -r 3 — full memory + bandwidth + NVLink test (~30 min). See DCGM.
2. nccl-tests all_reduce_perf 2-node with this and a known-good neighbor.
3. ib_send_bw --use_cuda across all 8 HCAs paired against a known-good node.
4. fio against local NVMe and against shared FS (Weka, Lustre, etc.).
5. GPU thermal sweep — sustained 100% utilization for 30 min, watch temps.

Runbook

#!/usr/bin/env bash
# gpu_deepcheck.sh — weekly deep check, run on a drained node.
# Requires: kubectl drain has already been run; there's a peer node available.
# Usage: gpu_deepcheck.sh <peer-hostname>
set -euo pipefail
NODE=$(hostname)
PEER=${1:?usage: $0 <peer-hostname>}
WORK=/var/tmp/deepcheck.$NODE.$(date +%Y%m%d-%H%M%S)
mkdir -p $WORK
cd $WORK
echo "=== deepcheck start: $NODE vs $PEER ==="
echo "logs in: $WORK"

# 1. dcgmi diag -r 3
echo "[1/5] dcgmi diag -r 3"
dcgmi diag -r 3 -j > dcgmi.r3.json
grep -q '"status":\s*"Fail"' dcgmi.r3.json && { echo "FAIL: dcgmi -r 3 failed"; exit 1; }

# 2. nccl-tests 2-node
echo "[2/5] nccl-tests all_reduce_perf 2-node"
mpirun -np 16 -H "$NODE:8,$PEER:8" -bind-to none -map-by slot \
    -x NCCL_DEBUG=WARN \
    -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
    -x NCCL_SOCKET_IFNAME=eth0 \
    /opt/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -g 1 \
    > nccl.out 2>&1
busbw=$(awk '/^[ ]+1073741824/ { print $11; exit }' nccl.out)
awk -v bw=$busbw 'BEGIN { if (bw < 320) { print "FAIL: 2-node busbw too low: " bw; exit 1 } }'
echo "  -> 2-node busbw: $busbw GB/s"

# 3. ib_send_bw across all 8 HCAs
echo "[3/5] ib_send_bw across all 8 HCAs"
for hca_idx in 0 1 2 3 8 9 10 11; do
    hca="mlx5_${hca_idx}"
    cuda_dev=$([[ $hca_idx -lt 4 ]] && echo $hca_idx || echo $((hca_idx - 4)))
    # peer-side launch first (assumes ssh + already-installed perftest):
    ssh "$PEER" "ib_send_bw -d $hca --use_cuda=$cuda_dev -F -R --report_gbits" > /dev/null &
    sleep 1
    ib_send_bw -d "$hca" --use_cuda="$cuda_dev" -F -R --report_gbits "$PEER" \
        > "ib_send_bw.$hca.out" 2>&1 || { echo "FAIL: ib_send_bw on $hca"; exit 1; }
    wait
    bw=$(awk '/^[ ]+8388608/ { print $4; exit }' "ib_send_bw.$hca.out")
    echo "  -> $hca: $bw Gb/s"
    awk -v bw=$bw 'BEGIN { if (bw < 350) { print "FAIL: $hca BW too low: " bw; exit 1 } }'
done

# 4. fio
echo "[4/5] fio against local NVMe"
fio --name=randread --filename=/var/lib/fio.test --size=10G \
    --rw=randread --bs=4k --iodepth=32 --runtime=60 --direct=1 \
    --output-format=json > fio.local.json 2>&1
rm -f /var/lib/fio.test

# Optional: against shared FS, e.g. /mnt/weka
[[ -d /mnt/weka ]] && fio --name=randread --filename=/mnt/weka/fio.test.$NODE \
    --size=10G --rw=randread --bs=4k --iodepth=32 --runtime=60 --direct=1 \
    --output-format=json > fio.weka.json 2>&1 && rm -f /mnt/weka/fio.test.$NODE

# 5. thermal sweep — sustained 100% util
echo "[5/5] thermal sweep (30 min)"
nvidia-smi -q -d POWER,TEMPERATURE > thermal.before.txt
# kick off compute load — gpu-burn or any sustained kernel
for i in 0 1 2 3 4 5 6 7; do
    CUDA_VISIBLE_DEVICES=$i /opt/gpu-burn/gpu_burn 1800 > /tmp/burn.$i.log 2>&1 &
done
sleep 1700  # let it run for ~28 min
nvidia-smi -q -d POWER,TEMPERATURE > thermal.during.txt
wait

# Check that no GPU exceeded 90°C during the run
HOT=$(grep "GPU Current Temp" thermal.during.txt | awk '$5 > 90' | wc -l)
[[ $HOT -eq 0 ]] || { echo "WARN: $HOT GPU(s) exceeded 90°C during sweep"; }

echo "=== deepcheck PASS ==="

Expected outputs (sanitized)

[1/5] dcgmi diag -r 3
  -> all subsystems Pass
[2/5] nccl-tests all_reduce_perf 2-node
  -> 2-node busbw: 363.1 GB/s
[3/5] ib_send_bw across all 8 HCAs
  -> mlx5_0: 388.4 Gb/s
  -> mlx5_1: 387.9 Gb/s
  -> mlx5_2: 388.2 Gb/s
  -> mlx5_3: 387.5 Gb/s
  -> mlx5_8: 388.1 Gb/s
  -> mlx5_9: 387.8 Gb/s
  -> mlx5_10: 388.0 Gb/s
  -> mlx5_11: 387.6 Gb/s
[4/5] fio against local NVMe
  -> 1.2M IOPS @ 4k randread
[5/5] thermal sweep (30 min)
  -> max GPU temp: 78°C (within budget)
=== deepcheck PASS ===

If any HCA shows < 350 Gb/s on a 400-link, dig into RDMA debugging. Mlxlink eye-opening is the next read.

Pre-deploy check (~4-8 hours, full burn-in)

Run before adding a node to the production pool. The cost of catching a problem here is one engineer-day; the cost of catching it after the node has run customer workloads is much higher.

Checklist

Anything failing the pre-deploy check → fix or RMA before the node leaves the staging pool. Don't put a node into production "to see if it works" — the production cluster is not your staging environment.

Sample pre-deploy validation script (sketch)

#!/usr/bin/env bash
# gpu_predeploy.sh — full burn-in. Run on a drained, cordoned node not yet in production pool.
# Expected runtime: 4-8 hours.
set -euo pipefail
NODE=$(hostname)
WORK=/var/tmp/predeploy.$NODE.$(date +%Y%m%d-%H%M%S)
mkdir -p $WORK; cd $WORK

# Capture baseline
nvidia-bug-report.sh

# 1. dcgmi diag -r 4 (extended, ~1-2h)
dcgmi diag -r 4 -j > dcgmi.r4.json

# 2. gpu-burn 4 hours
for i in 0 1 2 3 4 5 6 7; do
    CUDA_VISIBLE_DEVICES=$i /opt/gpu-burn/gpu_burn 14400 > burn.$i.log 2>&1 &
done
wait

# 3. fio NVMe burn-in
fio --name=mix --filename=/var/lib/fio.predeploy --size=100G \
    --rw=randrw --rwmixread=70 --bs=4k --iodepth=64 --runtime=3600 --direct=1 \
    --output-format=json > fio.local.json
rm -f /var/lib/fio.predeploy

# 4. memtester 4h on a chunk of host RAM
memtester 1G 4 > memtester.log 2>&1

# 5. ib_send_bw against multiple peers (delegated to deepcheck)
for peer in $PEER_NODES; do
    /usr/local/sbin/gpu_deepcheck.sh "$peer" || exit 1
done

# 6. K8s test pod
cat > test-pod.yaml <<EOF
apiVersion: v1
kind: Pod
metadata: {name: predeploy-test-$NODE, namespace: validation}
spec:
  nodeSelector: {kubernetes.io/hostname: $NODE}
  restartPolicy: Never
  containers:
  - name: t
    image: nvcr.io/nvidia/cuda:12.4.0-devel-ubuntu22.04
    command: ["bash", "-c", "nvidia-smi && ibv_devinfo"]
    resources: {limits: {nvidia.com/gpu: 8}}
EOF
kubectl apply -f test-pod.yaml
kubectl wait pod -n validation predeploy-test-$NODE --for=condition=Ready --timeout=120s
kubectl logs -n validation predeploy-test-$NODE
kubectl delete -f test-pod.yaml

# 7. quickcheck (must pass)
/usr/local/sbin/gpu_quickcheck.sh

echo "=== PASS: $NODE ready for production pool ==="

Specific GPU thermal sweep notes

Sustained 100% util for 30+ min is the standard thermal stress. What you're looking for:

What "bad" really means: thermal throttling kicked in, sustained throughput is silently lower than rated. The job runs but the busbw / FLOPS will be 10-20% below baseline. Detect with:

# during the sweep:
nvidia-smi --query-gpu=temperature.gpu,clocks.current.sm,clocks.max.sm \
    --format=csv,noheader,nounits -lms 1000 \
    | awk -F',' '$2 < $3 - 100 { print "throttled" }'

If clocks.current.sm is sustained below clocks.max.sm while the sweep is at 100% util, the GPU is throttling. Cooling problem (fan curve, dust, ambient temp) or PSU droop.

Pre-cron checklist

Before turning any of this loose:

1. The script is checked into the cluster's automation repo, not just /usr/local/sbin/ on every node.
2. Every node has a recent (< 30 day) successful pre-deploy log archived.
3. Quickcheck output goes to a central log + metrics system.
4. Alerts are wired to your paging system, not just email.
5. The on-call engineer can find the script and the alerts within 60 seconds.

See also

• Triage decision tree — when a check fails
• Incident response — when a failure becomes an incident
• NCCL multi-node failures — deep-check NCCL component
• RDMA debugging — deep-check fabric component
• DCGM — dcgmi diag reference
• K8s GPU pod failures — pre-deploy K8s sanity

External:

• gpu-burn: github.com/wilicc/gpu-burn
• nccl-tests: github.com/NVIDIA/nccl-tests
• perftest: github.com/linux-rdma/perftest
• nvidia-bug-report.sh: ships with the NVIDIA driver