Kernel-level troubleshooting playbook

Boot, memory, network, PCIe, disk, and lockup diagnostics — what to read, what to capture, and how to find what drifted.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

When a GPU node misbehaves and the obvious checks (nvidia-smi, kubectl get nodes, dmesg | tail) come back clean, the cause is usually one layer down. This page is the playbook for pulling kernel-level evidence: which tools, which counters, which paths in /proc and /sys, and how to read them.

The structure is by symptom class. Pick the section that matches what's broken, work through the commands top to bottom, and you'll either find the cause or have enough evidence to escalate.

Universal first steps

Before drilling down, capture the basics. These are cheap, fast, and useful in every investigation.

# Kernel + distro
uname -r                                     # 5.15.0-105-generic
cat /etc/os-release | head                   # PRETTY_NAME, VERSION_ID

# Uptime — when did this start?
uptime
# 14:32:11 up 23 days,  4:12,  3 users,  load average: 8.45, 6.78, 5.12

# Last reboot reason
last reboot | head -3

# Kernel cmdline
cat /proc/cmdline

# Loaded modules of interest
lsmod | grep -iE 'nvidia|mlx5|nvme|wekafs|gpu'

# dmesg in human time
sudo dmesg -T | tail -100

# journalctl since this boot
sudo journalctl -b 0 --no-pager | tail -100

# Critical errors only since boot
sudo journalctl -b 0 -p err --no-pager

Pipe the output to a file and timestamp it:

mkdir -p /tmp/diag-$(date +%Y%m%d-%H%M)
cd /tmp/diag-$(date +%Y%m%d-%H%M)
{ dmesg -T; echo ---; journalctl -b 0 -p err --no-pager; } > kernel.log
sysctl -a 2>/dev/null > sysctl.txt
ip a; ip r; ss -s > network.txt
nvidia-smi -q > gpu.txt 2>&1
lspci -vvv > pci.txt

That snapshot is what you take to the kernel team. Make it before doing anything destructive.

Boot issues

Boot console logs go to journald. If the node is reachable via console (IPMI, BMC), capture:

# From inside a working boot, look at the previous boot's logs
sudo journalctl --list-boots
# 0 fa6a... Mon May  5 14:00 — current
# -1 abc1... Mon May  5 13:30 — Mon May  5 13:55     <-- problem boot
# -2 ...

sudo journalctl -b -1 -p err --no-pager
sudo journalctl -b -1 -u systemd-udev-trigger.service
sudo journalctl -b -1 | grep -i 'failed\|error' | head -50

If you don't have a working boot, capture from BMC serial console. If neither, single-user mode:

# At GRUB menu, press 'e' on the kernel line, append:
systemd.unit=rescue.target
# or
init=/bin/bash

Symptom: kernel panic or oops at boot

The serial console will show a stack trace before it locks up. The pattern:

[   12.345678] BUG: kernel NULL pointer dereference, address: 0000000000000018
[   12.345699] #PF: supervisor read access in kernel mode
[   12.345712] #PF: error_code(0x0000) - not-present page
[   12.345725] PGD 0 P4D 0
[   12.345738] Oops: 0000 [#1] SMP NOPTI
[   12.345752] CPU: 0 PID: 1234 Comm: systemd Not tainted 5.15.0
[   12.345766] Hardware name: ...
[   12.345780] RIP: 0010:some_function+0x12/0x456 [some_module]
[   12.345795] Code: ...
[   12.345810] RSP: 0018: ...
[   12.345825] Call Trace:
[   12.345838]  <TASK>
[   12.345851]  ? __die+0x123/0x456
[   12.345864]  ? page_fault_oops+0x123/0x456
[   12.345878]  some_other_function+0x123/0x456 [some_module]
[   12.345893]  ...

Read top to bottom:

The BUG: line says what (NULL deref, GPF, etc.)
RIP: says which function and which module — this is the smoking gun
Call Trace: is the path that got there

Save the trace, then search for "linux kernel " in mailing lists or your distro's bug tracker. Most kernel oopses are known issues with documented fixes (often "upgrade to kernel ≥ X.Y").

Symptom: kdump configured but no crash dump generated

# Is kdump enabled?
sudo systemctl status kdump

# Did the crash kernel reserve memory?
cat /proc/cmdline | grep -o crashkernel=\\S*
# crashkernel=512M

# Was there a panic without dump?
sudo journalctl -b -1 | grep -iE 'panic|oops'
# panic was logged, but no /var/crash/<timestamp>/

ls /var/crash/

Common kdump failure causes:

crashkernel= size too small (< 256 M on big-RAM nodes is risky)
Capture target (NFS, local disk) was unwritable
The panic was a hard lockup that didn't trigger the secondary kernel

Set up kdump properly:

sudo apt install linux-crashdump kdump-tools

# /etc/default/kdump-tools
USE_KDUMP=1
KDUMP_KERNEL=/boot/vmlinuz-$(uname -r)
KDUMP_INITRD=/boot/initrd.img-$(uname -r)
KDUMP_COREDIR=/var/crash

sudo systemctl enable --now kdump-tools

# Verify
sudo kdump-config status
# current state: ready to kdump

Test it without a real panic:

echo c | sudo tee /proc/sysrq-trigger
# Triggers a panic. Node will reboot into capture kernel, write to /var/crash/, then reboot normal.

Memory issues

Symptom: OOM killer fires unexpectedly

sudo dmesg -T | grep -B5 -A30 'Out of memory'
# [time] python invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE),...
# [time] Mem-Info:
# [time] active_anon:123456 inactive_anon:23456 ...
# [time] Node 0 active_anon:78901kB ...
# ...
# [time] Tasks state (memory values in pages):
# [time] [ pid ]   uid  tgid total_vm      rss ... name
# [time] [12345]  1000 12345 1234567   234567 ... python
# [time] [23456]   100 23456  234567    23456 ... node-export
# ...
# [time] oom-kill:constraint=CONSTRAINT_NONE,nodemask=...
# [time] Out of memory: Killed process 12345 (python) total-vm:128GB

Read top to bottom:

gfp_mask — what kind of allocation triggered it (GFP_HIGHUSER_MOVABLE = userspace anon)
Mem-Info — system-wide page counts, identifies which kind of memory was exhausted
Node X lines — per-NUMA breakdown
Task list — rss column to see who was big
Out of memory: Killed process — the chosen victim

Common conclusions:

A process truly grew too big → kill expected, fix the workload
constraint=CONSTRAINT_MEMCG → cgroup OOM, not host (see cgroups)
Node 0 active_anon huge, Node 1 nearly empty → NUMA imbalance with vm.zone_reclaim_mode=1
Available memory is high but a "movable" allocation failed → fragmentation

Symptom: OOM message mentions "Out of memory and no killable processes"

Means kernel can't find a candidate to kill — usually because the cgroup OOM has all candidates protected via oom_score_adj=-1000. Check:

for p in /proc/*/oom_score_adj; do
  pid=$(echo $p | awk -F/ '{print $3}')
  adj=$(cat $p 2>/dev/null)
  comm=$(cat /proc/$pid/comm 2>/dev/null)
  echo "$adj $pid $comm"
done | sort -n | head
# -1000 1234 systemd-oomd
# -1000 5678 sshd
# -999 ...

If everything important has -1000, the killer fails silently and the kernel just drops requests. Re-tune some processes' oom_score_adj to 0 so the killer has options.

Symptom: huge memory pressure on a "fine" node

# Per-NUMA buddy allocator state
cat /proc/buddyinfo
# Node 0, zone DMA32 ... 1234 567 89 0 0 ...
# Node 0, zone Normal ...  78  23  12 5 0 ...    <-- low order = bad

# Page allocator stats
cat /proc/zoneinfo | grep -E 'Node|free|min|low|high|managed' | head -40

# Slab memory (kernel object cache)
sudo slabtop -o -s c | head -20
#  OBJS  ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
# 1234567 1234567 100% 0.18K   54321 16 200000K dentry
#  234567  234567 100% 0.50K    5678 32  90000K filp

If slabtop shows dentry or inode cache eating tens of GB, the workload churns through files. vm.vfs_cache_pressure is the relevant sysctl (default 100, raise to 200 for less aggressive caching).

sysrq for live state

If the node is on its way to OOM but still reachable:

# Memory pressure dump to dmesg
echo m | sudo tee /proc/sysrq-trigger

dmesg -T | tail -100
# Active anon, inactive anon, slab, free per zone

Other useful sysrq triggers:

Letter	What it does
`m`	Memory state dump (zone info, free pages)
`t`	Task list dump (every process and its state)
`w`	Like t but only blocked tasks
`l`	Stack trace of every CPU (catches hung CPUs)
`s`	Sync filesystems (before forced reboot)
`u`	Remount all FS read-only
`b`	Immediate hard reboot (data loss possible)
`c`	Trigger a crash (kdump, if configured, will capture)

Enable sysrq if not already:

echo 1 | sudo tee /proc/sys/kernel/sysrq

# Persistent
echo "kernel.sysrq = 1" | sudo tee /etc/sysctl.d/99-sysrq.conf
sudo sysctl --system

Network issues

Symptom: packet drops, retransmits, or "throughput is half what it should be"

# Per-interface stats
ip -s link show ens14f0
# 2: ens14f0: ...
#     RX: bytes  packets  errors  dropped overrun mcast
#     1234567890 12345678 0       1234    0       1234
#     TX: bytes  packets  errors  dropped carrier collsns
#     2345678901 23456789 0       0       0       0

# Drops on RX without errors usually = qdisc backlog too small
# Drops on TX = sock buffer too small or app slow to drain

# More detail per-NIC
ethtool -S ens14f0 | grep -iE 'drop|err|miss|fail' | grep -v ': 0$'
# rx_dropped: 1234567
# rx_missed: 234
# rx_csum_err: 0
# tx_carrier_err: 0
# ...

# Specifically per-RX queue
ethtool -S ens14f0 | grep -E 'rx_queue.*drop|rx[0-9]+_drop'
# Look for asymmetric drops — one queue dropping while others don't

Asymmetric per-queue drops often = IRQ pinning issue. The hot queue's IRQ is pinned to a busy core that can't drain it.

Conntrack pressure

# Is conntrack involved?
lsmod | grep nf_conntrack

# If yes, table state
sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
# nf_conntrack_count = 245678
# nf_conntrack_max = 262144              <-- almost full!

# What's using the table?
sudo conntrack -L 2>/dev/null | awk '{print $3, $4}' | sort | uniq -c | sort -rn | head
# 12345 tcp ESTABLISHED
#  2345 tcp TIME_WAIT
#  1234 udp ESTABLISHED

Bump nf_conntrack_max (see sysctl).

TCP-level deep dive

# Aggregate counters
nstat -az | grep -E 'TcpExt|IpExt' | head -50
# TcpExtSyncookiesSent          0
# TcpExtSyncookiesRecv          0
# TcpExtTCPLossUndo             1234
# TcpExtTCPSackRecovery         5678
# TcpExtTCPRcvCollapsed         12345
# TcpExtTCPRetransFail          234     <-- retransmits that failed entirely
# ...

# Socket states
ss -s
# Total: 1234
# TCP:   234 (estab 89, closed 23, orphaned 0, timewait 122)
# Transport Total IP IPv6
# RAW   0     0  0
# UDP   45    44 1
# TCP   210   200 10
# INET  255   244 11
# FRAG  0     0  0

# Per-socket — look for buffer pressure
ss -tin | grep -B1 'cwnd' | head -10
# rcv_space:262144   <-- if stuck low on a fat-pipe flow, autotune isn't growing
# wmem_alloc:0

TcpExtTCPRetransFail ticking is bad — retransmits exhausted. Either the peer is gone, a middlebox is dropping, or you're on a black-hole MTU path. Test with:

ping -M do -s 8972 <peer>      # 9000 MTU minus 28 IP+ICMP overhead
ping -M do -s 1472 <peer>      # 1500 MTU

If 1472 succeeds but 8972 fails, MTU is wrong somewhere on path. Set net.ipv4.tcp_mtu_probing=1 to let the kernel discover.

PCIe issues

Symptom: "GPU fell off the bus" or PCIe AER messages

sudo dmesg -T | grep -iE 'pcie|aer'
# [time] pcieport 0000:00:01.0: AER: Corrected error received: 0000:17:00.0
# [time] mlx5_core 0000:17:00.0: PCIe link is down
# [time] nvidia-driver 0000:31:00.0: GPU has fallen off the bus

This is the bad one. Walk through:

# 1. Confirm device is gone
lspci -nn | grep -i nvidia
# 31:00.0 3D controller [0302]: NVIDIA Corp ... 
# 31:00.1 ...
# 31:00.2 ...
# 31:00.3 ...
# 31:00.4 ...
# 31:00.5 ...
# 31:00.6 ...
# 31:00.7 ...
# (one missing? device fell off completely)

# 2. Look at PCIe link status
sudo lspci -vvv -s 31:00.0 | grep -E 'LnkSta|LnkCap'
# LnkCap: Port #..., Speed 32GT/s, Width x16, ASPM not supported
# LnkSta: Speed 32GT/s, Width x16
#         (downgraded if Speed/Width != Cap)

# 3. Check for AER counters
sudo lspci -vvv -s 31:00.0 | grep -iE 'aer|err|cnt' | head -20
# Capabilities: [..] Advanced Error Reporting
#         UESta: ...
#         CESta: 1234   <-- correctable error count, non-zero is suspicious

# 4. Sysfs AER counters (cumulative since boot)
ls /sys/bus/pci/devices/0000:31:00.0/aer_dev_correctable
sudo cat /sys/bus/pci/devices/0000:31:00.0/aer_dev_correctable
# Receiver Error 1234
# Bad TLP 5
# Bad DLLP 12
# RELAY_NUM Rollover 0
# Replay Timer Timeout 23
# Advisory Non-Fatal Error 0
# Corrected Internal Error 0
# Header Log Overflow 0
# TOTAL_ERR_COR 1274

If Receiver Error or Bad TLP is rising at hundreds per second, the PCIe link is degrading. Causes (in order of likelihood):

Connector contamination / loose seating — reseat the GPU
Riser cable damage — swap the riser
Thermal cycling damage on the PCIe slot
PSU brown-out (insufficient 12V on PCIe rails)
Motherboard slot damage

If the GPU fell off entirely, check dmesg for thermal:

sudo dmesg -T | grep -iE 'therm|over.?temp|throttl'
# nvidia 0000:31:00.0: GPU temp 95C, throttling
# nvidia 0000:31:00.0: GPU at thermal critical, shutting down

Thermal kills happen when liquid cooling pumps fail or air flow is restricted (dust, blocked vents). Check the cooling subsystem before reseating.

Symptom: PCIe link width or speed downgraded

# Should be x16 Gen4 (16 GT/s) or Gen5 (32 GT/s) for modern GPUs
sudo lspci -vvv -s 31:00.0 | grep LnkSta
# LnkSta: Speed 8GT/s (downgraded), Width x4 (downgraded)   <-- BAD

# Cross-check with nvidia-smi
nvidia-smi -q -i 0 | grep -A5 'PCIe Generation'
# PCIe Generation
#     Max         : 4
#     Current     : 1                  <-- driver agrees, link is at Gen1 instead of Gen4

A persistent downgrade = hardware problem. A transient one (driver just loaded, link negotiating) is normal — read again after 30 seconds.

Reset / unbind / rebind a device

If a device is in a wedged state but not entirely fallen off:

# Find the BDF
BDF=0000:31:00.0

# Unbind from current driver
echo $BDF | sudo tee /sys/bus/pci/drivers/nvidia/unbind

# Rescan
echo 1 | sudo tee /sys/bus/pci/devices/$BDF/remove
sleep 2
echo 1 | sudo tee /sys/bus/pci/rescan

# Re-bind
echo $BDF | sudo tee /sys/bus/pci/drivers/nvidia/bind

For NVIDIA GPUs, nvidia-smi -r -i <id> does a controlled GPU reset (assuming the driver is still responsive). A pci_dev_reset is a sledgehammer that requires no other device on the same root complex be busy.

Disk issues

Symptom: high iowait, slow I/O

See I/O scheduler for the device-specific tuning. The diagnosis chain:

# 1. Confirm iowait
top -bn1 | head -5
# %Cpu(s):  3.4 us,  1.2 sy,  0.0 ni, 0.0 id, 95.1 wa, 0.1 hi, 0.2 si

# 2. Which device is busy?
iostat -xz 1 5
# Device         r/s   w/s   rkB/s    wkB/s   await  %util
# nvme0n1     1234.0   45  234567     2000   45.6    99.0    <-- busy and slow
# nvme1n1        2.0    0     128        0    0.5     0.5

# 3. Which process is doing the I/O?
sudo iotop -oP 5 1

# 4. Block-level latency
sudo blktrace -d /dev/nvme0n1 -w 30 -o nvme0
sudo blkparse -i nvme0 | head -30
sudo btt -i nvme0 -o nvme0.btt
cat nvme0.btt_iops_fp.dat | head     # IOPS over time

Disk health (SMART)

sudo smartctl -a /dev/nvme0n1 | head -40
# Critical Warning:                   0x00
# Temperature:                        45 Celsius
# Available Spare:                    100%       <-- spare capacity remaining
# Available Spare Threshold:          10%
# Percentage Used:                    12%        <-- wear indicator
# Data Units Read:                    123456789 [63.2 TB]
# Data Units Written:                 23456789 [12.0 TB]
# Power Cycles:                       45
# Power On Hours:                     12345
# Unsafe Shutdowns:                   2
# Media and Data Integrity Errors:    0          <-- non-zero = drive is dying
# Error Information Log Entries:      0

Watch:

Percentage Used > 90% — drive is near end of life
Media and Data Integrity Errors > 0 — replace the drive
Unsafe Shutdowns rising — power supply / shutdown sequencing issues

Disk full / filesystem corruption

# Check space + inodes
df -h
df -i

# Filesystem errors in dmesg
sudo dmesg -T | grep -iE 'ext[234]|xfs|btrfs|filesystem' | head -30

# Read-only after error
mount | grep ro
# /dev/nvme0n1p1 on / type ext4 (ro,relatime,errors=remount-ro)   <-- forced RO

# Find the cause
sudo dmesg -T | grep -i 'remount-ro' -B3
# EXT4-fs error (device nvme0n1p1): ext4_mb_generate_buddy:756:...
# Aborting journal on device nvme0n1p1-8.
# EXT4-fs (nvme0n1p1): Remounting filesystem read-only

Once the FS is RO due to error, you need to umount and run fsck. Do this from rescue mode:

sudo umount /                  # won't work if root, need rescue
sudo fsck -y /dev/nvme0n1p1
sudo mount -o rw,remount /dev/nvme0n1p1

For non-root filesystems you can do it live.

System hang / lockup

Symptom: node responsive over network but local commands hang

# What's blocked?
ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/' | head -20
# 1234 D    folio_wait_bit_common         python
# 1235 D    refrigerator                  kworker/u:0
# 1236 D    md_thread                     md0_raid5

Tasks in D (uninterruptible sleep) are stuck waiting for I/O or a lock. The wchan column says where.

# Capture stack of every blocked task
echo w | sudo tee /proc/sysrq-trigger
sudo dmesg -T | tail -200

# Or stack of a specific task
sudo cat /proc/1234/stack
# [<ffff...>] folio_wait_bit_common+0x123/0x456
# [<ffff...>] filemap_fault+0x123/0x456
# [<ffff...>] __do_fault+0x12/0x456
# [<ffff...>] handle_mm_fault+0x123/0x456

Common stuck-task causes:

folio_wait_bit_common / filemap_fault — waiting on storage
cv_wait (ZFS) — ZFS arc / spa state
synchronize_rcu — waiting on RCU grace period
wait_on_buffer — waiting on dirty page writeback

If a Weka or NFS mount is the cause:

# Show all NFS clients and their state
sudo cat /proc/net/rpc/nfs
sudo cat /proc/mounts | grep -E 'nfs|wekafs'
sudo nfsstat -c

Symptom: kernel reports "soft lockup" or "rcu_sched stall"

sudo dmesg -T | grep -iE 'soft lockup|hard lockup|rcu_sched|stall'
# [time] watchdog: BUG: soft lockup - CPU#42 stuck for 23s! [python:12345]
# [time] CPU: 42 PID: 12345 Comm: python
# [time] RIP: 0010:some_function+0x12/0x456
# [time] Call Trace:
# [time]  ? watchdog_timer_fn+0x123/0x456
# [time]  ...

Soft lockup = a CPU spent >20 seconds inside the kernel without yielding. Common causes:

Buggy driver in a loop (NVIDIA driver bugs in old versions)
Storage timeout that holds a spinlock too long
Bad BPF program

Hard lockup (watchdog: BUG: hard lockup) = a CPU was unresponsive to NMI for ~10 s. Almost always hardware (failing CPU, microcode bug) or kernel bug. Capture vmcore via kdump and escalate.

Symptom: load average climbing without obvious culprit

# uptime says 50.0, but `top` doesn't show 50 hot processes
uptime
# load average: 50.23, 45.67, 40.12

# Where is load coming from? D-state count
ps aux | awk '$8 ~ /D/' | wc -l
# 50                                <-- 50 processes stuck in D, that's your load

# What are they?
ps aux | awk '$8 ~ /D/' | head -10
# user 12345 0.0 0.0 ... D ... cmd

# Catch a few stacks
for p in $(ps -eo pid,stat | awk '$2 ~ /D/ {print $1}' | head -10); do
  echo "=== pid $p ===";
  sudo cat /proc/$p/stack
done

If they're all stuck in the same call site (e.g., all in nfs_wait_on_request), the upstream resource is the cause.

perf for hot path identification

When the system is responsive but slow:

# Profile system-wide for 30 seconds
sudo perf record -a -g -F 99 -- sleep 30

# View as flat report
sudo perf report --stdio --no-children | head -30
# Samples: 12K of event 'cycles', Event count: 12345678
# Overhead  Command   Shared Object        Symbol
#   45.67%  python    [kernel.kallsyms]    [k] some_kernel_function
#   12.34%  swapper   [kernel.kallsyms]    [k] cpuidle_enter
#   ...

# Or as flame graph (need to install FlameGraph from brendangregg/FlameGraph)
sudo perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > flame.svg

If the top symbol is something kernel like _raw_spin_lock_irqsave, you have a lock contention issue. Drill into who's calling it.

ftrace for tracing specific events

# Function tracer — record every kernel function call (heavy!)
sudo trace-cmd record -p function_graph -F some_command
sudo trace-cmd report | head

# Or focus on one function
sudo trace-cmd record -p function -l 'mlx5_*' -F some_command

# Or track wakeups
sudo trace-cmd start -e sched:sched_wakeup
sleep 5
sudo trace-cmd extract
sudo trace-cmd report | head

trace-cmd is the user-friendly wrapper around the kernel's /sys/kernel/tracing/ interface. Use it for "what's the kernel actually doing right now".

The diff method — find what drifted

For "yesterday it worked, today it doesn't":

# Take a baseline (when things work)
mkdir /var/lib/baseline
sudo sysctl -a 2>/dev/null > /var/lib/baseline/sysctl.txt
lsmod > /var/lib/baseline/modules.txt
ip link > /var/lib/baseline/links.txt
ip route > /var/lib/baseline/routes.txt
sudo iptables-save > /var/lib/baseline/iptables.txt
ls /etc/sysctl.d/ /etc/modprobe.d/ /etc/modules-load.d/ > /var/lib/baseline/conf.txt
dpkg -l > /var/lib/baseline/packages.txt   # or rpm -qa
sudo dmidecode > /var/lib/baseline/hw.txt

# Later, when it breaks:
diff /var/lib/baseline/sysctl.txt <(sudo sysctl -a 2>/dev/null)
diff /var/lib/baseline/modules.txt <(lsmod)
diff /var/lib/baseline/packages.txt <(dpkg -l)

Anything that changed between baseline and broken is suspicious. Often it's a package upgrade, a config-management agent change, or an "auto-upgrade" service that ran. The diff is the fastest path to the cause.

For network paths specifically:

# Did routes change?
diff /var/lib/baseline/routes.txt <(ip route)

# Did MTU change on an interface?
diff /var/lib/baseline/links.txt <(ip link)

Reading kernel call stacks

Stack traces in dmesg follow a pattern. Example:

[time] Call Trace:
[time]  <TASK>
[time]  ? __die+0x123/0x456                   <-- ? = inlined or unsure
[time]  ? page_fault_oops+0x123/0x456
[time]  do_user_addr_fault+0x123/0x456
[time]  exc_page_fault+0x12/0x34
[time]  asm_exc_page_fault+0x26/0x30
[time]  RIP: 0010:nvidia_driver_function+0x45/0x67 [nvidia]   <-- WHERE THE FAULT HAPPENED
[time]  ...
[time]  some_higher_function+0x12/0x34 [nvidia]
[time]  another_function+0x56/0x78 [nvidia]
[time]  unix_stream_recvmsg+0x12/0x34
[time]  __sys_recvfrom+0x123/0x456
[time]  __x64_sys_recvfrom+0x12/0x34
[time]  do_syscall_64+0x12/0x34
[time]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[time]  </TASK>

Read bottom to top. Bottom = how the syscall got into the kernel. Top = where it failed. The RIP: line is the actual instruction pointer — that function in that module crashed.

The +0x45/0x67 notation means "offset 0x45 within the function, function size 0x67". You can disassemble:

# If you have debug symbols
sudo apt install linux-image-$(uname -r)-dbgsym

addr2line -e /lib/modules/$(uname -r)/kernel/drivers/.../nvidia.ko nvidia_driver_function+0x45
# or
objdump -d /lib/modules/.../nvidia.ko | grep -A20 nvidia_driver_function:

For the NVIDIA proprietary driver, debug symbols aren't shipped — the function names are obfuscated. Bug reports go to NVIDIA with the full stack trace.

Worked example: "GPU fell off the bus" walkthrough

A real diagnosis sequence:

# Symptom: training job died, GPU0 not visible
nvidia-smi
# Unable to determine the device handle for GPU 0000:31:00.0

# Step 1: dmesg for the bus event
sudo dmesg -T | grep -B5 -A20 'fell off' | head -40
# [Mon May 5 14:23:11 2026] nvidia 0000:31:00.0: GPU has fallen off the bus.
# [Mon May 5 14:23:11 2026] NVRM: GPU has fallen off the bus

# Step 2: AER counters before the event?
sudo dmesg -T | grep -B30 'fell off' | grep -i aer
# [Mon May 5 14:21:34 2026] pcieport 0000:00:01.0: AER: Corrected error received: 0000:31:00.0
# [Mon May 5 14:21:42 2026] pcieport 0000:00:01.0: AER: Corrected error received: 0000:31:00.0
# [Mon May 5 14:22:18 2026] pcieport 0000:00:01.0: AER: Corrected error received: 0000:31:00.0
# (escalating frequency leading up to the fault)

# Step 3: thermal?
sudo dmesg -T | grep -B30 'fell off' | grep -iE 'therm|temp'
# [Mon May 5 14:22:55 2026] nvidia 0000:31:00.0: GPU temperature 88C
# [Mon May 5 14:23:08 2026] nvidia 0000:31:00.0: GPU temperature 91C

# Step 4: power?
sudo dmesg -T | grep -B30 'fell off' | grep -iE 'power|volt'
# (none — not a power issue)

# Step 5: PCIe link state
sudo lspci -vvv -s 31:00.0 | grep LnkSta
# (the device is gone, lspci returns nothing)

# Step 6: AER history
cat /sys/bus/pci/devices/0000:31:00.0/aer_dev_correctable 2>/dev/null
# (also gone — sysfs entry removed when device fell off)

# Conclusion: thermal event triggered AER cascade, then bus drop
# Action: cool the chassis (check fans/liquid), reseat the GPU,
#         monitor temperature trends post-restart

The pattern: AER correctables ramping up, then thermal climbing, then bus drop. Order matters — AER first means the link was already degrading from heat before the actual disconnect. Fix the cooling, the AER stops, the link stays up.

Diagnostic bundle — what to capture before reboot

When you must reboot to recover, capture this first so the postmortem can find the cause:

TS=$(date +%Y%m%d-%H%M%S)
DIR=/var/diag/$TS
sudo mkdir -p $DIR

# Kernel state
sudo dmesg -T > $DIR/dmesg.log
sudo journalctl -b 0 --no-pager > $DIR/journal.log

# Configuration
sudo sysctl -a > $DIR/sysctl.log 2>/dev/null
cat /proc/cmdline > $DIR/cmdline
lsmod > $DIR/modules.log

# Hardware
sudo lspci -vvv > $DIR/lspci.log
sudo dmidecode > $DIR/dmi.log

# GPU + RDMA
sudo nvidia-smi -q > $DIR/gpu.log 2>&1
sudo nvidia-smi topo -m > $DIR/topo.log 2>&1
sudo ibv_devinfo -v > $DIR/rdma.log 2>&1

# Network
ip a > $DIR/ip.log
sudo ss -anp > $DIR/ss.log
sudo conntrack -L > $DIR/conntrack.log 2>&1

# Process tree
ps auxf > $DIR/ps.log

# Stuck tasks
echo w | sudo tee /proc/sysrq-trigger
sleep 1
sudo dmesg -T | tail -200 > $DIR/sysrq-w.log

# Block I/O
iostat -xz > $DIR/iostat.log
mount > $DIR/mounts.log

# Memory state
echo m | sudo tee /proc/sysrq-trigger
sleep 1
sudo dmesg -T | tail -100 > $DIR/sysrq-m.log

tar czf /var/diag/$TS.tar.gz -C /var/diag $TS

That tarball is what you attach to the incident ticket. Anything missing from there will be the question someone asks tomorrow.