Operations troubleshooting — the layered triage playbook

When a tenant says "my training job is broken" or a pod is stuck ContainerCreating, you don't pick a fix at random. There's a layering you can walk: hardware, kernel modules, userspace tools, application. The lower layer being broken makes everything above it broken; verifying each layer in order isolates the actual cause within a few minutes.

This page is that playbook plus a table of common error messages mapped to likely causes and next steps.

The four layers

┌────────────────────────────────────┐
│  4. Application                    │   trainer, NCCL, Slurm job
│      (logs, exit codes, stdout)    │
├────────────────────────────────────┤
│  3. Userspace tools                │   nvidia-smi, ibstat, nvcc,
│      (binaries that talk to        │   dmesg, journalctl
│       kernel and devices)          │
├────────────────────────────────────┤
│  2. Kernel modules / device files  │   lsmod, /dev/nvidia*,
│      (the kernel side of drivers)  │   /dev/infiniband/*
├────────────────────────────────────┤
│  1. Hardware                       │   lspci, dmesg, smartctl,
│      (PCIe, link state, BIOS)      │   nvme list
└────────────────────────────────────┘

Always start at the bottom. If the GPU isn't visible to lspci, no amount of kubectl describe pod is going to help.

Layer 1: Hardware

# All NVIDIA GPUs visible to PCIe?
lspci | grep -i nvidia
# 17:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# 31:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# 4b:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# 65:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# 97:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# c1:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# da:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# f4:00.0 3D controller: NVIDIA Corporation Device 2330 (rev a1)
# Expected count: matches the spec sheet (8 for an 8-GPU box).
# Missing: BIOS/POST issue, or hardware fault.

# All Mellanox NICs visible?
lspci | grep -i mellanox
# 19:00.0 Infiniband controller: Mellanox Technologies MT2910 ...
# ...

# PCIe link width and speed for a GPU
sudo lspci -vv -s 17:00.0 | grep -E 'LnkSta:|LnkCap:'
# LnkCap:  Port #0, Speed 32GT/s (ok), Width x16 (ok), ASPM L1, ...
# LnkSta:  Speed 32GT/s (ok), Width x16 (ok)
# LnkSta should be at LnkCap. Down-trained = bad slot, dirty connector, or BIOS.

# Recent hardware errors
dmesg -T | grep -iE 'error|fail|nvme|pcie|mce|edac' | tail -50
# Look for: "PCIe Bus Error", "AER" reports, NVMe controller resets,
# "Machine Check Exception" (mce), DIMM EDAC errors.

# NVMe drives
nvme list
sudo smartctl -a /dev/nvme0n1 | grep -E 'Critical|Available|Percentage|Temperature'

Red flags: GPU count short, link width below x16, AER errors, NVMe Critical Warning != 0x00, Machine Check Exceptions.

Layer 2: Kernel modules and device files

# NVIDIA driver loaded
lsmod | grep nvidia
# nvidia_peermem         16384  0
# nvidia_uvm           1646592  4
# nvidia_drm            122880  6
# nvidia_modeset       1310720  10 nvidia_drm
# nvidia              56631296  463 nvidia_uvm,nvidia_modeset

# Device files exist
ls -l /dev/nvidia*
# /dev/nvidia0  /dev/nvidia1  ... /dev/nvidia7
# /dev/nvidiactl  /dev/nvidia-uvm  /dev/nvidia-uvm-tools

# RDMA / IB
lsmod | grep -E 'mlx5|ib_'
# ib_uverbs             184320  3
# mlx5_ib               401408  0
# mlx5_core            2105344  1 mlx5_ib
# ...

ls -l /dev/infiniband/
# rdma_cm  uverbs0  uverbs1  ...

# wekafs (if applicable)
lsmod | grep weka
cat /sys/module/wekafsio/refcnt 2>/dev/null

Red flags: missing nvidia_peermem (no GPUDirect RDMA), missing mlx5_ib (no RDMA at all), /dev/nvidia* device count mismatch with what lspci showed.

Layer 3: Userspace tools

# GPU runtime
nvidia-smi
# Driver Version, GPU count, memory, persistence mode.
# All eight GPUs at idle should report util=0%, mem~0, temp=30-40°C.

# GPU topology — which GPUs talk to which over what
nvidia-smi topo -m

# IB / RDMA
ibstat
# CA 'mlx5_0'
#         Port 1:
#                 State: Active
#                 Physical state: LinkUp
#                 Base lid: 0x4
#                 LMC: 0
#                 Rate: 400

ibv_devinfo
# (more verbose, includes guid, fw version)

# NIC link state for Ethernet RDMA / IPoIB
ethtool ens14f0np0 | grep -E 'Speed|Link'

# Quick connectivity check between GPUs
nvidia-smi topo -p2p r
# Pairs marked OK or NS. NS on same-baseboard pair = ACS or topology issue.

# DCGM diagnostics — runs the canonical health suite
dcgmi diag -r 2
# Run level 2 (basic + memory + bandwidth tests). 3 = full (longer).

Red flags: nvidia-smi shows GPUs but topo -p2p is NS everywhere (ACS), ibstat shows INIT instead of Active (link down), DCGM diag reports failures on specific GPUs.

Layer 4: Application

# Pod side
kubectl -n <ns> describe pod <pod>
# Look at the Events section, image pull state, mounts, init container status.

kubectl -n <ns> logs <pod> -c <container>            # current
kubectl -n <ns> logs <pod> -c <container> --previous # if it crashed and restarted

# Slurm side
kubectl -n <ns> exec login-0 -- squeue -j <jobid> -l
kubectl -n <ns> exec login-0 -- sacct -j <jobid> --format=JobID,JobName,State,ExitCode

# Inside a still-running container
kubectl -n <ns> exec -it <pod> -- bash
ulimit -a            # confirm ulimits propagated
echo $LD_LIBRARY_PATH; echo $CUDA_HOME
nvidia-smi           # works inside the container?
ibv_devinfo          # RDMA visible inside the container?

Common error patterns table

A standard triage flow

When a tenant reports "training is broken on gpu-01":

# 1. SSH to the node, check hardware first
ssh gpu-01
lspci | grep -i nvidia | wc -l       # expected GPU count
dmesg -T | tail -100 | grep -iE 'error|xid|fault|aer'

# 2. Kernel modules
lsmod | grep -E 'nvidia|mlx5|weka'

# 3. Userspace
nvidia-smi
ibstat | grep -E 'Port|State'

# 4. Now the application — kubectl back at the workstation
kubectl -n tenant-foo describe pod <pod>
kubectl -n tenant-foo logs <pod> --previous | tail -200

Most issues resolve in this flow within 10 minutes. Hardware issues take longer because they involve coordinating with the datacenter or a vendor.

When the issue is "the cluster is slow today"

This is harder. Multi-node NCCL slowdown without a clear node-level failure usually traces to:

1. Network path congestion — switch buffer drops, broken link in the spine. Pull switch counters.
2. Shared FS saturation — Weka backend overwhelmed. weka stats throughput.
3. One bad node dragging down a collective — find it by running nccl-tests on subsets.
4. Cross-tenant noisy neighbor if the tenants share fabric.

The nccl-tests ring-test approach: split the nodes in half, run all_reduce_perf on each half. Whichever half is slow, split again. Halves until you isolate the bad node.

Always-useful escalation data

When opening a ticket (Weka, NVIDIA, vendor), include:

• Output of dmesg -T | tail -500 (or full dmesg)
• lspci -tv and lspci -vvv -s <device>
• nvidia-bug-report.sh (NVIDIA collects everything)
• For RDMA issues: ibdiagnet -P all=1 -r --get_phy_info
• For Weka: weka diagnostics from the affected client and from a backend host
• Time range of the failure (UTC) and what changed in the previous 24 hours

See also

• ACS disable — common P2P fix
• Ulimits — RLIMIT_MEMLOCK + NOFILE
• Sysctl tuning — vm.max_map_count, inotify
• Weka troubleshooting — stale mounts
• ArgoCD — sync gotchas
• Incident response — when triage becomes incident

External:

• nvidia-bug-report.sh: ships with the driver
• DCGM diagnostics: developer.nvidia.com/dcgm
• Mellanox ibdiagnet: docs.nvidia.com/networking/category/ofed