GPU node health

Per-node GPU sanity checks + a fleet-wide three-step audit.

Two things on this page:

Per-node checks — copy-ready commands for the ten things you should look at when a single node is in question.
Fleet audit — paste three kubectl outputs and find nodes with missing or unreachable GPU pods.

Per-node checklist

GPU presence + driver

nvidia-smi -L

Should list 8 H100 / H200 / GH200 (depending on SKU). If it errors with "NVML: Unknown Error" the driver / fabric-manager is broken.

GPU full status

nvidia-smi

Look for ECC errors in the lower table, persistence-mode = On, and no dashes ("--") under temp.

Per-GPU PCIe link speed/width

nvidia-smi --query-gpu=index,pcie.link.gen.current,pcie.link.width.current --format=csv

On H100 expect Gen5 x16. Gen3 / x8 = bus is parked or the slot is misseated.

NVLink topology

nvidia-smi nvlink --status

8-GPU NVSwitch: every GPU should have 18 links Up. Anything Disabled = needs fabric-manager investigation.

NVSwitch peer test (DCGM)

dcgmi diag -r 1

Run after a node has been fully drained. Catches ECC double-bit errors, NVLink down, fabric-manager not running.

Persistence mode

nvidia-smi -pm 1

Critical for keeping the kernel module loaded between container starts. Should be enabled at boot via nvidia-persistenced.

Fabric manager

systemctl status nvidia-fabricmanager

Required on NVSwitch boards (DGX, HGX H100). Must match the running driver version.

IB devices + state

ibstat | grep -E "CA '|State|Rate"

On 8-rail H100 box: 8 CAs, all "State: Active", all "Rate: 400 Gb/sec".

IB RDMA test (peer)

# server
ib_send_bw -d mlx5_0 -F
# client
ib_send_bw -d mlx5_0 -F <peer-ip>

Confirms the rail is actually carrying RDMA, not just link-up.

Container Toolkit hook

nvidia-container-cli info

If this fails the runtime can't inject /dev/nvidia* into containers — every workload will see "no GPU available".

Fleet audit

Step 1 · All cluster nodes

command

kubectl get nodes --selector='!node-role.kubernetes.io/login' -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

Step 2 · Pod → node mapping

command

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\n"}{end}' | grep <pod-prefix>

Step 3 · kubectl exec — reach each pod

command

for node in $(kubectl get nodes --selector='!node-role.kubernetes.io/login' -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  pod=$(kubectl get pods -A --field-selector spec.nodeName=$node \
        --no-headers -o custom-columns=":metadata.namespace,:metadata.name" 2>/dev/null \
        | grep "<pod-prefix>" | head -1)
  if [ -z "$pod" ]; then continue; fi
  ns=$(echo "$pod" | awk '{print $1}'); name=$(echo "$pod" | awk '{print $2}')
  out=$(kubectl exec -n "$ns" "$name" -- echo ok 2>&1)
  if [ $? -eq 0 ]; then echo "OK: $name"; else echo "FAIL: $name => $out"; fi
done