GPU node triage decision tree

When a node misbehaves at 03:00, the worst thing you can do is start at the top of the stack ("the pod is failing, let me look at kubelet logs"). The right move is to start at the bottom — silicon, then drivers, then verbs, then app, then scheduler — and stop the moment you find a layer that's broken. Every layer above a broken layer will produce noise that wastes your time.

This page is the decision tree. It is opinionated: walk it from top to bottom, in order, on every triage. Don't skip levels.

The order, in one table

Stop at the first failing step. Fix that, then re-run from there.

Step 1 — nvidia-smi

The single most informative command on any GPU node. It exercises the userspace driver library, which talks to the kernel module, which talks to the hardware. If nvidia-smi works, the bottom of the stack is fine.

Healthy output:

$ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3      On      | 00000000:1B:00.0 Off   |                    0 |
| N/A   34C    P0             71W /  700W |       0MiB /  81559MiB |      0%      Default |
| ...                                                                                     |

Things to read:

• Driver Version — should match the version you expect; mismatch with fabric-manager / nvidia-peermem will surface here.
• All N GPUs present — on an 8-GPU H100 box, you should see GPU 0 through 7. If a GPU is missing, the kernel never enumerated it. Check lspci | grep -i nvidia next.
• ECC column — non-zero Volatile Uncorr. ECC is bad. Aggregate Uncorr. ECC accumulates over the GPU lifetime — anything > 0 means a row got remapped or an SRAM cell went bad.
• Persistence-M = On — without persistence mode, the driver tears down between CUDA contexts. Always nvidia-smi -pm 1 on a real node.
• Power, Temp — sustained > 80 °C on H100 / > 90 °C on A100 means cooling problem.

What it means when nvidia-smi fails

Fabric-manager check (HGX / DGX boxes only)

On nodes with NVSwitch (HGX H100/H200/B200, DGX), the GPUs are gated by nvidia-fabricmanager:

$ systemctl status nvidia-fabricmanager
● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Active: active (running) since Wed 2026-04-22 12:00:00 UTC; 13 days ago

$ tail -20 /var/log/fabricmanager.log
...
Successfully configured all the available GPUs and NVSwitches to route NVLink traffic.

If fabric-manager is dead or version-skewed against the driver, GPUs will be visible to nvidia-smi but unusable for any multi-GPU workload — cudaGetDeviceCount returns 8, but launching anything across them hangs. See fabric-manager.

Step 2 — dmesg

$ dmesg -T | grep -iE 'nvidia|nvrm|gpu|xid' | tail -50

The XIDs you care about most:

XID 79 + persistent NVLink errors = stop debugging, pull the node, raise an RMA. Don't keep retrying the workload.

If dmesg shows nothing GPU-related but nvidia-smi failed → driver userspace problem (library mismatch), not a hardware problem.

Step 3 — lsmod

$ lsmod | grep -E '^(nvidia|mlx5|ib_|rdma_|nv)'
nvidia_peermem         16384  0
nvidia_uvm           1564672  0
nvidia_drm            106496  0
nvidia_modeset       1306624  3 nvidia_drm
nvidia              56602624 87 nvidia_uvm,nvidia_peermem,nvidia_modeset
ib_ipoib              163840  0
ib_umad                32768  4
rdma_ucm               36864  0
rdma_cm                86016  2 ib_iser,rdma_ucm
ib_uverbs             192512  2 rdma_ucm,mlx5_ib
mlx5_ib               524288  0
ib_core               573440 11 rdma_cm,ib_ipoib,...,nvidia_peermem
mlx5_core            2207744  1 mlx5_ib

What to verify on a fresh-booted node:

• nvidia loaded — driver is up.
• nvidia_uvm loaded — Unified Memory; lazy-loaded on first CUDA call. Missing here is fine if no CUDA has run yet, but force-load it (modprobe nvidia_uvm) before training.
• nvidia_peermem loaded with refcount > 0 (the last column, nvidia_peermem 16384 0) — see peermem. If refcount is 0, no RDMA workload has registered against it yet. If the module isn't even listed, GDR is dead.
• mlx5_core, mlx5_ib, ib_core, ib_uverbs, rdma_cm, rdma_ucm, ib_umad, ib_ipoib all loaded — the canonical 9-module RDMA list. See OFED.

Missing modules → check /etc/modules-load.d/. Loaded but module dependency error → modinfo nvidia_peermem and look for the kernel version it was built for; mismatch means kernel was upgraded but modules weren't rebuilt.

Step 4 — ibstat

$ ibstat
CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.41.1000
        Hardware version: 0
        Node GUID: 0xa088c2030099abcd
        System image GUID: 0xa088c2030099abcd
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xa288c2fffe99abcd
                Link layer: Ethernet

What you want to see, on every HCA:

• State: Active — anything else (Initializing, Down, Polling) means the link isn't usable.
• Physical state: LinkUp — physical layer is good. LinkDown = cable / port problem; Polling = no peer responding.
• Rate — should match your HCA spec. CX-7 NDR = 400 Gb/s. CX-7 with HDR transceiver = 200. CX-6 = 200 Gb/s. Anything lower = wrong cable, switch port misconfig, or signal-integrity issue.
• Link layer: Ethernet for RoCE, InfiniBand for IB. Mix-up here means the port is configured wrong (mlxconfig to flip).

For IB clusters, also check the SM:

$ sminfo
sminfo: sm lid 1 sm guid 0x506b4b03004f1a5a, activity count 12345 priority 1 state 3 SMINFO_MASTER

State 3 = ACTIVE master. State 0 = no SM seen. No SM means no LIDs, no routes, no traffic.

If ibstat fails or returns nothing → lsmod | grep ib_umad. If ib_umad isn't loaded, ibstat can't query the device.

Step 5 — dcgmi diag -r 1

The shortest active health check NVIDIA ships. Run level 1 = ~30 seconds, runs the absolute essentials.

$ dcgmi diag -r 1
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+---------------------------+------------------------------------------------+

Any line that says Fail is enough to stop scheduling work on the node. The most common in production:

• Page Retirement/Row Remap Fail — GPU has exhausted its remap pool. Schedule for RMA.
• Persistence Mode Fail — nvidia-smi -pm 1.
• Inforom Fail — corrupted GPU EEPROM, rare but happens. RMA.

For deeper validation use -r 3 (~30 minutes, exercises memory, NVLink, and CUDA kernels) or -r 4 (extended). See DCGM.

Step 6 — nvidia-smi nvlink -e

$ nvidia-smi nvlink -e
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-1234abcd-...)
         Link 0: Replay Errors: 0
         Link 0: Recovery Errors: 0
         Link 0: CRC Errors: 0
         ...
         Link 17: Replay Errors: 0
         Link 17: Recovery Errors: 0
         Link 17: CRC Errors: 0

H100 = 18 NVLinks per GPU; H200 = 18; B200 = 18 (× more bandwidth per link). Want all zeros across all GPUs.

If you see Replay Errors > 0 on a specific link → the link is degraded. NCCL will still build a ring through it, training will be slow and may fail with unhandled cuda error at NCCL_COMM_INIT. Cordon the node, schedule for RMA.

CRC errors that recover are fine in small numbers; rapidly growing CRC counters mean an NVSwitch port or the GPU's NVLink PHY is failing.

# zero counters after RMA / repair (only meaningful with reset capability)
$ nvidia-smi nvlink -sc 0bz

See NVLink / NVSwitch.

Step 7 — ib_send_bw host-to-host

Once steps 1-6 pass, exercise the fabric end-to-end. On node A:

nodeA$ ib_send_bw -d mlx5_0 -i 1 -F -R --report_gbits

On node B (peer):

nodeB$ ib_send_bw -d mlx5_0 -i 1 -F -R --report_gbits <nodeA-ip>

Healthy NDR (CX-7) result, full message size sweep:

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      1000           388.45             388.42                0.741
 131072     1000           390.10             390.08                0.372
---------------------------------------------------------------------------------------

390 Gb/s on a 400 Gb/s link = 97.5 % efficiency. Anything below ~85 % indicates a fabric problem (PFC missing on RoCE, MTU mismatch, cable, switch port). If ib_send_bw works but NCCL doesn't, the problem is either NCCL config or peermem/GDR — see NCCL multi-node failures.

For GDR-specific verification, build perftest with CUDA support and add --use_cuda=0 (or whichever CUDA device):

nodeA$ ib_send_bw -d mlx5_0 --use_cuda=0 --report_gbits

Bandwidth should be similar (~380-390 Gb/s on NDR). A big drop = peermem isn't working. See peermem.

Step 8 — nccl-tests 2-rank single-node, then 2-node

Single-node, 2-GPU, sanity:

$ ./build/all_reduce_perf -b 8 -e 1G -f 2 -g 2

You should see busbw ramp toward the per-GPU NVLink bandwidth (450+ GB/s on H100). If this fails or busbw is wildly low, the problem is intra-node (NVLink, NVSwitch, peermem, CUDA driver), not the fabric.

Multi-node:

$ mpirun -np 16 -hostfile hosts \
    -x NCCL_DEBUG=INFO \
    -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11 \
    ./build/all_reduce_perf -b 1M -e 2G -f 2 -g 1

See NCCL multi-node for expected busbw per topology.

Step 9 — kubectl get node

If everything below the orchestrator is healthy, look at K8s next:

$ kubectl get node gpu-node-01 -o wide
NAME          STATUS   ROLES   AGE     VERSION         INTERNAL-IP   ...
gpu-node-01   Ready    <none>  127d    v1.30.6+rke2r1  10.0.0.10     ...

Ready is what you want. Anything else:

$ kubectl describe node gpu-node-01 | grep -A 20 Conditions:

You want all conditions False except Ready=True. Look for MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable.

If Ready but something is wrong, check taints — a custom taint can effectively cordon the node from your workload.

Step 10 — kubectl describe pod

$ kubectl describe pod my-training-job-0 -n tenant-foo

Read the Events: section bottom-up. The most recent event tells you the current state:

If the pod started: kubectl logs <pod> -c <container>. If it never started, the events block tells you why.

For deeper debugging see K8s GPU pod failures.

Worked examples

"Training job hangs at step 0"

1. nvidia-smi on every node — all 8 GPUs visible, all healthy. ✓
2. dmesg | grep -iE 'nvidia|xid' — clean. ✓
3. lsmod | grep peermem — nvidia_peermem 16384 0. Loaded but refcount = 0 → no RDMA app has used it yet. Suspicious if training has been running.
4. ibstat on every node — all ports Active at 400. ✓
5. dcgmi diag -r 1 — Pass. ✓
6. NCCL log: GPU Direct RDMA Disabled for HCA 0.

Verdict: GDR is off. Either nvidia_peermem was loaded after mlx5_ib (load order), or ACS isn't disabled on PCIe root. Re-load peermem after IB stack, check ACS. See peermem and ACS.

"One specific node fails NCCL all-reduce while others succeed"

1. nvidia-smi on the bad node — all 8 GPUs, ECC clean. ✓
2. dmesg | grep -i nvlink — XID 74 NVLink error: link 12 down. Found it.
3. nvidia-smi nvlink -e — Replay Errors: 8472 on GPU 3 link 4.

Verdict: NVLink degraded on GPU 3. The intra-node NCCL ring fails because one of the 18 links between GPU 3 and NVSwitch is dead. Cordon, RMA. Don't waste time on NCCL config or the fabric.

"Pod stuck Pending forever"

1. nvidia-smi not relevant — pod hasn't started.
2. kubectl get node — all nodes Ready. ✓
3. kubectl describe pod — 0/12 nodes are available: 8 Insufficient nvidia.com/gpu, 4 node(s) didn't match Pod's node affinity/selector.

Verdict: Scheduler problem. 8 nodes have no free GPUs (workload competition), 4 don't match affinity (probably wrong tenant or wrong nodepool label). Not a hardware issue at all. See K8s GPU pod failures and Reservations.

"Pod ContainerCreating for 10 minutes"

1. kubectl describe pod events:

   Warning  FailedCreatePodSandBox  3m  ...  failed to setup network for sandbox ...
   

2. kubectl logs -n kube-system <cni-pod-on-this-node> — CNI daemon log shows IPAM exhaustion.

Verdict: CNI / IPAM. Not a GPU problem at all. The triage tree saved you from poking at nvidia-smi for an hour — the pod isn't even at the runtime stage yet.

When to skip levels

You can skip the bottom of the tree if:

• Multiple healthy nodes are all failing the same workload simultaneously — almost certainly orchestrator, network, or app config. Skip to step 9.
• The workload was running healthy 5 minutes ago and just stopped — usually app, network, or scheduler event. Check kubectl get events -A --sort-by='.lastTimestamp' | tail -50 first.

You should NOT skip levels if:

• One node is failing, others are not.
• "It worked yesterday" after any infra change (driver, kernel, OFED, K8s).
• After a hardware swap or reboot.

When in doubt, walk it from the bottom. The 10 minutes you spend on steps 1-8 is cheaper than the 4 hours you'll waste assuming it's the orchestrator.

See also

• NCCL multi-node failures — expanded NCCL error decoding
• RDMA debugging — deeper Step 4-7 walkthrough
• Driver / firmware mismatch matrix — when Step 1 reports version skew
• K8s GPU pod failures — expanded Step 9-10
• Health check runbook — daily / weekly / pre-deploy version of this tree
• DCGM — what dcgmi diag actually does
• NCCL multi-node — bandwidth expectations