AMD GPU troubleshooting: failure modes mapped to NVIDIA equivalents

The fastest way to debug ROCm if you've spent a decade on CUDA is to find the analogous NVIDIA failure and translate. The kernel module is different, the userspace is different, but the categories of things that go wrong are remarkably similar — kernel mismatch, missing module load order, environment-variable misconfiguration, thermal throttling, ECC silent corruption, fabric link degradation.

This page is the symptom-to-cause map. Each section names a real failure, shows the symptoms, gives the AMD diagnostic, and points at the NVIDIA equivalent so the analogy is explicit.

1. amdgpu module won't load — kernel mismatch

Symptom: After a kernel upgrade and reboot, rocm-smi returns nothing, or:

$ rocm-smi
ERROR: Unable to find any GPUs

$ lsmod | grep amdgpu
(empty)

$ dmesg | tail -20
[   12.345] amdgpu: module verification failed: signature and/or required key missing
[   12.346] amdgpu: disagrees about version of symbol drm_dev_alloc

Cause: DKMS didn't rebuild amdgpu.ko against the new kernel. Either headers were missing at upgrade time, or the build silently failed.

Diagnostic:

$ dkms status
amdgpu/6.10.5.70203-1, 6.8.0-49-generic, x86_64: installed
amdgpu/6.10.5.70203-1, 6.8.0-50-generic, x86_64: WARNING! Diff between built and installed module

# Or worse:
$ dkms status
amdgpu/6.10.5.70203-1, 6.8.0-49-generic, x86_64: installed
# (no entry for the kernel you just booted into)

Fix:

# Make sure headers are present
sudo apt install "linux-headers-$(uname -r)"

# Rebuild
sudo dkms autoinstall

# Reload
sudo modprobe amdgpu

# Verify
lsmod | grep amdgpu
rocm-smi

NVIDIA equivalent: This is exactly the NVIDIA driver kernel mismatch failure — nvidia-smi returns "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver". Same root cause: DKMS didn't rebuild after a kernel bump. Same fix flow: install headers, sudo dkms autoinstall, reload module.

2. rocm-smi shows GPUs but inference fails

Symptom: rocm-smi lists all 8 GPUs, GPUs look healthy, but the application errors:

$ python infer.py
RuntimeError: HIP error: no HIP-capable device is detected

Or:

hipErrorNoDevice

Or PyTorch:

torch.cuda.is_available() == False
torch.cuda.device_count() == 0

(Note: PyTorch on ROCm still uses torch.cuda.* API names — the Python interface is unchanged.)

Cause: One of three things:

1. HIP_VISIBLE_DEVICES is set wrong — restricting to a non-existent index, or to a device the user doesn't have access to.
2. User not in render and video groups — /dev/kfd and /dev/dri/render* permissions deny.
3. Container missing --device=/dev/kfd --device=/dev/dri.

Diagnostic:

# Check device files
$ ls -la /dev/kfd /dev/dri/render*
crw-rw---- 1 root render  236, 0 May  6 09:12 /dev/kfd
crw-rw---- 1 root render  226, 128 May  6 09:12 /dev/dri/render128

# Check group membership
$ id
uid=1000(youruser) gid=1000(youruser) groups=1000(youruser),44(video),109(render)

# Check env
$ env | grep -i 'HIP_VISIBLE\|ROCR_VISIBLE\|GPU_DEVICE'
HIP_VISIBLE_DEVICES=0,1,2,3

Fix:

# If user not in render group:
sudo usermod -a -G render,video $USER
# (must log out and back in — or run as root)

# If running in a container, run with:
docker run --device=/dev/kfd --device=/dev/dri --group-add video --group-add render ...

# For Kubernetes via the AMD GPU Operator, the device-plugin handles this automatically;
# see the k8s-amd-gpu-operator doc.

# If HIP_VISIBLE_DEVICES is wrong, unset and let the runtime see everything:
unset HIP_VISIBLE_DEVICES
unset CUDA_VISIBLE_DEVICES

NVIDIA equivalent: nvidia-smi shows GPUs but torch.cuda.is_available() == False — usually CUDA_VISIBLE_DEVICES set wrong, or container missing --gpus all, or /dev/nvidia* not exposed. Same conceptual failure: the kernel can see the device, the userspace can't access it.

3. HIP_VISIBLE_DEVICES vs ROCR_VISIBLE_DEVICES vs CUDA_VISIBLE_DEVICES

Three env vars, subtly different layers, easy to confuse. This is one of the gotchas that doesn't have a clean NVIDIA analog.

Operational rule: for any Python ML workload on AMD, set both HIP_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES to the same list. Belt and suspenders.

export HIP_VISIBLE_DEVICES=0,1,2,3
export CUDA_VISIBLE_DEVICES=0,1,2,3

If you only set CUDA_VISIBLE_DEVICES, some workloads (Horovod, DeepSpeed in certain configurations, custom kernels) ignore it and grab all 8 GPUs anyway.

4. PyTorch errors with hipErrorNoBinaryForGpu

Symptom:

RuntimeError: HIP error: no kernel image is available for execution on the device

Or in some builds:

hipErrorNoBinaryForGpu

Cause: The PyTorch (or vLLM, or whatever) container was compiled for a different gfx* target than the silicon you're running on.

Diagnostic:

# What gfx is the GPU?
$ rocminfo | grep -E "Name:.*gfx"
  Name:                    gfx942

# What gfx was PyTorch compiled for?
$ python -c "import torch; print(torch.cuda.get_arch_list())"
['gfx90a', 'gfx940']
# ↑ no gfx942 — this PyTorch will not run on MI300X

Fix:

• Use a container compiled for your GPU. rocm/pytorch:latest covers gfx942 today.
• Or rebuild PyTorch with PYTORCH_ROCM_ARCH=gfx942 (or whichever target).
• As a last resort, set HSA_OVERRIDE_GFX_VERSION=11.0.0 (or appropriate ID) — this lies to ROCm about the GPU type and may produce wrong numerical results. Useful only for development sanity checks.

NVIDIA equivalent: CUDA error: no kernel image is available for execution on the device — same conceptual failure. CUDA architecture mismatch (TORCH_CUDA_ARCH_LIST set wrong, or container built for sm_80 running on sm_90).

5. Thermal throttling

Symptom: GPU under heavy load runs slower than expected. Bandwidth halves. Throughput drops 30%+. No errors logged.

Diagnostic:

# Snapshot during the load
$ amd-smi monitor -ptu
GPU  POWER  GPU_TEMP  MEM_TEMP  GFX_UTIL  GFX_CLOCK   THROTTLE
0    520 W  98°C      102°C     100%      1750 MHz    PROCHOT,THERM
1    615 W  86°C       92°C     100%      2100 MHz    -
2    620 W  85°C       91°C     100%      2100 MHz    -
...

GPU 0 is throttled. THROTTLE column shows PROCHOT (ASIC at thermal limit) and/or THERM (HBM at thermal limit). Note the lower clock (1750 vs 2100 MHz) and lower power (520 vs 620 W).

For continuous monitoring during a run:

# Log every 2 seconds
amd-smi monitor -ptu --interval 2 --csv > /tmp/thermal.csv

Cause:

• Cooling problem. Air-cooled box exceeded its thermal envelope (MI300X at 750 W is at the edge of air cooling; many real-world OAM boards exceed safe air-cooled operation under sustained load).
• One GPU's heatsink loose or thermal-paste degraded — that one GPU runs 10°C hotter than its peers.
• Datacenter inlet temperature too high — ambient air entering the rack is above spec.
• Liquid loop issue — coolant flow rate dropped, or CDU pump has degraded.

Fix:

• For sustained 100% utilization on MI300X+, liquid cooling is effectively required. Air-cooled MI300X works for inference with bursty utilization, struggles for sustained training.
• Check that all fans are spinning (ipmitool sdr type fan).
• Check inlet temperature at the rack (datacenter monitoring).
• Reduce power cap as a stop-gap: rocm-smi --setpoweroverdrive 600 (sets cap to 600 W; performance drops but throttling stops).

NVIDIA equivalent: H100/H200 throttling shows up as reduced SM clocks in nvidia-smi -q, with Throttle Reasons: HW Slowdown / SW Thermal Slowdown / HW Power Brake. Same operational picture: GPU is fine, environment isn't. Same set of fixes: better cooling, lower inlet temp, reduce power cap.

6. ECC scrubbing and the slow-burn correctness problem

Symptom: Training run produces NaN losses occasionally. Or inference output occasionally garbled. No hard errors, no kernel panics.

Diagnostic:

# Check ECC counters
$ rocm-smi --showrasinfo all
GPU[0]:
   UE (Uncorrectable Error) count: 2
   CE (Correctable Error)   count: 14872
   block: SDMA0   ue: 0   ce: 12
   block: VRAM    ue: 2   ce: 14860
GPU[1]:
   UE count: 0
   CE count: 23
GPU[2]:
   ...

UE (uncorrectable error) count > 0 is bad. The GPU detected an error in HBM that single-bit ECC couldn't correct — your data is now wrong. Even one is grounds to evict the GPU from the pool.

CE (correctable error) counts in the thousands are normal across a long-running fleet. Sudden growth is suspect.

Fix:

• For UE > 0: drain the node, RMA the GPU. Cosmic-ray-induced UEs do happen; if you see them clustering on one GPU over time, it's a defective HBM stack.
• For CE growing rapidly: investigate whether VRAM clock is being pushed too hard, whether the HBM is overheating (correctable error rates rise with temperature), or whether the GPU is approaching end-of-life.
• For periodic NaN losses: enable RAS (Reliability, Accessibility, Serviceability) page retirement: rocm-smi --setecccountingenable 1. After a UE in a memory page, the kernel will retire that page so the same fault doesn't keep happening.

To clear counters after a known event (e.g. you reseated a card):

sudo rocm-smi --resetrasinfo

NVIDIA equivalent: nvidia-smi -q | grep -A 5 "ECC Errors" — same Volatile/Aggregate, single-bit (correctable) and double-bit (uncorrectable). Same operational rule: any double-bit error → drain and replace. Same page-retirement mechanism on H100+.

7. xGMI link degraded — falls back to PCIe

Symptom: AllReduce on a single 8-GPU MI300X box runs at PCIe-bandwidth (50 GB/s busbw instead of 590 GB/s). All 8 GPUs visible. No obvious errors.

Diagnostic:

# Topology — should be all XGMI off-diagonal
$ rocm-smi --showtopo
       GPU0   GPU1   GPU2   GPU3   GPU4   GPU5   GPU6   GPU7
GPU0   X      XGMI   XGMI   PCIe   XGMI   XGMI   XGMI   XGMI
                            ^^^^
GPU1   XGMI   X      XGMI   XGMI   XGMI   XGMI   XGMI   XGMI
GPU2   XGMI   XGMI   X      XGMI   XGMI   XGMI   XGMI   XGMI
GPU3   PCIe   XGMI   XGMI   X      XGMI   XGMI   XGMI   XGMI
       ^^^^
...

GPU0↔GPU3 has fallen back to PCIe. Any traffic between those two GPUs is now going through the CPU at PCIe Gen5 speeds (~50 GB/s) instead of xGMI (~128 GB/s per link).

xGMI error counters:

$ rocm-smi --showxgmierr
GPU[0]: XGMI Link Error Count: 14
GPU[3]: XGMI Link Error Count: 14

If both ends of a link show non-zero error counts, that link's physical layer is degraded.

Fix:

• First: power-cycle the chassis (full AC drop, not warm reboot). Many xGMI link issues are training-state problems that re-init on cold boot.
• If still bad: reseat the OAM modules.
• If still bad: RMA. xGMI is in the package — you can't service it.

After power-cycle, verify:

# Counters reset on cold boot, but any count > 0 within minutes of boot is bad
sleep 600
rocm-smi --showxgmierr

NVIDIA equivalent: NVLink link degradation. nvidia-smi nvlink -e shows Replay Errors, Recovery Errors. Same operational answer: power-cycle, then escalate to RMA. NVSwitch can sometimes mask single-link failures via re-routing; AMD's mesh cannot.

8. GPUDirect RDMA running at half speed (or not at all)

Symptom: Multi-node rccl-tests runs at ~50% of expected bandwidth. NCCL_DEBUG=INFO shows GDR enabled. No error messages.

Diagnostic:

# Is GDR actually being used? Look for:
$ NCCL_DEBUG=INFO mpirun ... 2>&1 | grep -i 'gdr\|peer-to-peer'
NCCL INFO Channel 00 : 0[18000] -> 4[3a000] [send] via NET/IB/0/GDRDMA
NCCL INFO Channel 00 : 0[18000] -> 4[3a000] [receive] via NET/IB/0/GDRDMA

# Check IOMMU mode
$ dmesg | grep -i iommu
[    1.234] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.
[    1.345] iommu: Default domain type: Translated

# Check if FINE_GRAIN_PCIE is set
$ env | grep HSA_FORCE_FINE_GRAIN_PCIE
(empty)

Cause (most common): HSA_FORCE_FINE_GRAIN_PCIE=1 not set, and IOMMU is in Translated mode rather than passthrough. Traffic is bouncing through host RAM via the IOMMU, halving effective bandwidth.

Fix:

# Quick fix — set the env var
export HSA_FORCE_FINE_GRAIN_PCIE=1
# Run the test again — should see ~2× the bandwidth

# Permanent fix — IOMMU in passthrough mode
# Edit /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
sudo update-grub
sudo reboot

After IOMMU passthrough, HSA_FORCE_FINE_GRAIN_PCIE=1 is no longer required (but harmless if set).

NVIDIA equivalent: GPUDirect RDMA running at half speed when IOMMU is in Translated mode — exact same root cause. NVIDIA recommends iommu=pt in the kernel cmdline for the same reason. See GPUDirect RDMA.

9. amd-smi and rocm-smi disagree

Symptom: rocm-smi reports 8 GPUs, amd-smi list reports 7. Or amd-smi reports power but rocm-smi doesn't.

Cause: ROCm version mismatch between the tools and the kernel module. Usually because someone ran apt install rocm-smi from a different repo than amdgpu-dkms, or upgraded one but not the other.

Diagnostic:

$ rocm-smi --version
ROCm-SMI version: 7.0.0
ROCm-SMI-LIB version: 7.0.0

$ amd-smi version
amd-smi version: 24.7.2

$ modinfo amdgpu | grep version
version:        6.10.5.70203-1

If the major versions don't all line up, weird stuff happens.

Fix:

# Reinstall the full meta-package to get consistent versions
sudo apt install --reinstall rocm

NVIDIA equivalent: nvidia-smi and dcgmi disagreeing — usually because the host driver version doesn't match the userspace library version. Fixed by reinstalling consistently.

10. Container can see GPUs at runtime but not at build time

Symptom: Building a custom container with RUN python -c "import torch; print(torch.cuda.is_available())" returns False during the build, even though the runtime container would see GPUs fine.

Cause: Docker's BuildKit doesn't expose /dev/kfd and /dev/dri to RUN steps by default. Build steps see no GPU.

Fix:

This is a build-time vs runtime distinction. Don't try to access the GPU during docker build. Move any GPU-touching test into the runtime path.

If you absolutely must, use Docker BuildKit with --device:

docker buildx build --allow security.insecure -t myimage --output=type=docker .
# In the Dockerfile:
# RUN --security=insecure ...

But really: don't.

NVIDIA equivalent: Same issue. docker build doesn't have GPU access by default; you can hack around it with nvidia-container-toolkit and BuildKit insecure mode, but the right answer is "don't try."

11. RCCL hangs on init

Symptom: mpirun ./all_reduce_perf starts, prints the bootstrap line, then hangs forever. No error messages.

Diagnostic:

# Re-run with verbose debug
NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH \
  mpirun ... ./all_reduce_perf -b 1G -e 1G -g 8 2>&1 | tee /tmp/rccl.log

# Common patterns to grep for in /tmp/rccl.log:
grep -E 'ERROR|Failed|Could not|misc/socket|Bootstrap' /tmp/rccl.log

Common causes:

• NCCL_SOCKET_IFNAME unset or wrong → bootstrap is trying IPoIB or a private interface that isn't reachable across nodes. Set explicitly to your control-plane interface (eth0, bond0, etc.).
• HCA selection mismatch across nodes → Node A picks mlx5_0 to talk to Node B, Node B isn't routing return traffic on mlx5_0. Pin both with NCCL_IB_HCA=mlx5_0,mlx5_1,....
• Firewall → some node has firewalld/ufw blocking the bootstrap port. NCCL/RCCL bootstraps over a TCP socket on a random high port; if the firewall blocks it, hang.
• GID index wrong on RoCE → check show_gids and pin NCCL_IB_GID_INDEX to the GID matching your RoCE v2 IPv4 subnet.

Fix: Same procedure as for NCCL — see NCCL multi-node. The logs and the env vars are the same.

NVIDIA equivalent: Identical. NCCL hang on init is one of the most common multi-node failures, and the debug procedure ports over directly.

12. ROCm container size and the "build is too big" problem

Symptom: rocm/pytorch is 30+ GB. Pulling 100 of them onto fresh nodes ties up the registry for an hour.

Cause: ROCm ships fat. The full ROCm + PyTorch + math libraries weigh in at 25–35 GB depending on version, vs ~10–15 GB for an equivalent NVIDIA image.

Fix:

• Use distroless / runtime-only images for inference. rocm/dev-ubuntu-22.04 contains the SDK; rocm/rocm-terminal and the various *-runtime images are smaller.
• Multi-stage builds: compile in rocm/dev, copy artifacts into rocm/rocm-terminal for runtime.
• Strip unused gfx targets. If you only run on MI300X, build PyTorch with PYTORCH_ROCM_ARCH=gfx942 only — drops 5–10 GB of unused arch-specific kernels.
• Local registry mirror in the cluster. Always.

NVIDIA equivalent: Same problem — nvcr.io/nvidia/pytorch is 20+ GB, fixes are the same (multi-stage, runtime images, strip arch list, local mirror).

13. The dmesg lines that mean trouble

A few specific kernel log patterns worth grepping for periodically:

# All AMD GPU stuff
sudo dmesg | grep -E 'amdgpu|drm|kfd'

# Specifically the ones that mean "investigate":
sudo dmesg | grep -E 'amdgpu.*ERROR|amdgpu.*FATAL|RAS|GPU reset|SMU|RLC'

What each means:

NVIDIA equivalent: dmesg | grep -E 'NVRM|Xid'. Xid codes (Xid 79 = GPU fell off the bus, Xid 13 = graphics engine error, Xid 31 = MMU fault, etc) play the same role. Same operational responses: drain on Xid 79; investigate on Xid 13; expected on Xid 31 (app bug).

A symptom-to-cause cheat sheet

See also

• ROCm stack — the layers underneath these failures
• RCCL vs NCCL — multi-node-specific failure modes
• AMD GPU stack overview — generation/feature context
• AMD GPU Operator on Kubernetes — many of these failures show up as Pod errors
• NVIDIA driver stack — analogous NVIDIA failure modes
• GPUDirect RDMA — the multi-node fast-path that's identical in concept on both vendors
• PCIe topology — IOMMU and PCIe constraints apply equally