Linux kernel versions for GPU/HPC: which to run, when to upgrade

The Linux kernel version on a GPU node is not a detail. It determines whether nvidia-peermem builds cleanly, whether your MLNX_OFED rebuilt itself in DKMS, whether cgroup v2 lets your kubelet expose the right device files, and whether NCCL bootstrap completes within the timeout. A kernel point release that lands in unattended-upgrades overnight can take a whole training run down the next morning if you don't have a strategy.

This page is the operator's view: which kernel branches to run, the distro defaults that matter, the upstream changes worth knowing about, the regressions to actively avoid, and the staged upgrade flow that lets you find problems before they hit production.

Why kernel version matters for HPC

A web service can run any kernel that the package manager picks. A GPU node has at least four pieces of software with hard kernel-version dependencies:

1. nvidia.ko (and nvidia-peermem.ko) — DKMS rebuilds it on every kernel upgrade. If the rebuild fails, GPUs come up after reboot but nvidia-smi returns "no devices found" or nvidia-peermem is missing entirely (silent loss of GPUDirect RDMA).
2. MLNX_OFED kernel modules — the Mellanox-patched mlx5_ib, ib_core, etc. The OFED release notes carry an explicit kernel matrix; running outside it means symbols don't resolve and the modules silently don't load.
3. BPF / cgroup v2 features — kubelet's CPU manager, the device-cgroup controller, the network-bandwidth controllers all rely on feature flags that landed at specific kernel versions.
4. RDMA subsystem — ib_uverbs and the verbs ABI evolve. New ConnectX features (DCT, mkey-on-demand) require kernel-side changes that took years to land upstream.
5. Scheduler — EEVDF (replaced CFS in 6.6) changes how isolated cores behave; some 5.x tunables don't exist anymore.

Running the wrong kernel is one of the most common root causes for "the cluster was fine yesterday and now NCCL won't start". The fix is almost always a kernel pin or a rollback, after the actual diagnosis is done.

Distro kernel matrix (2024-2026)

Two notes on RHEL/Rocky:

• The kernel version 5.14.0-XXX.el9 is wildly different from upstream 5.14 — Red Hat backports thousands of patches from later mainline kernels. Treat it as "RHEL 9 kernel" rather than "5.14".
• kernel-modules-extra is needed for some IB modules; not installed by default on minimal installs.

For Ubuntu, the HWE (Hardware Enablement) kernel ships newer mainline branches on top of the 22.04 / 24.04 base — package name linux-image-generic-hwe-22.04. It's separate from the GA kernel; you can have both installed and pick at boot.

LTS branches and the upstream cadence

Long-Term Support kernels live for several years and receive backported fixes. The active LTS branches as of 2026:

Beyond LTS, the upstream stable cadence is roughly one new mainline every 9-10 weeks (~5.X → 5.X+1, then 6.X → 6.X+1). Each release receives stable fixes for ~12 weeks until the next one, after which only LTS kernels keep getting patches.

For HPC operators, the practical takeaway: pin to an LTS or to a distro's enterprise-supported kernel. Running mainline tip on production GPU nodes means you eat every regression yourself.

Specific kernel notable changes for HPC operators

Not every kernel rev affects HPC; these are the ones where something operator-visible changed.

5.15 — cgroup v2 widely usable

• bpf_cgroup_storage lands properly — kubelet can do per-pod accounting via BPF.
• NVMe TCP improvements (host stack, not just target).
• io_uring becomes broadly stable; significant for high-IOPS workloads.
• IPv6 over IB stable.

This is the floor for most modern Kubernetes deployments. Below 5.15, you'll fight cgroup v2 quirks.

6.0 — io_uring polished, CXL prep

• io_uring IORING_OP_* opcode coverage near-complete.
• CXL 1.1/2.0 framework lands (foundational, not yet useful for most clusters).
• cgroup.kill for fast pod teardown.

6.1 — io_uring close, MGLRU

• IORING_OP_CLOSE finally works correctly with files in non-default file descriptor tables.
• Multi-Generational LRU (MGLRU) — significant memory reclaim improvements; off by default until 6.7.
• MM cgroup performance improvements (less lock contention).

6.5 — nvme polled queues

• nvme.poll_queues=N tunable becomes stable for high-end NVMe.
• More predictable NUMA balancing.
• ConnectX-7 features land in mainline (DCT, dynamic conn).

6.6 — EEVDF replaces CFS

This is a big one. The default scheduler changes from CFS (Completely Fair Scheduler) to EEVDF (Earliest Eligible Virtual Deadline First). For most workloads, no change — but for HPC where you depend on SCHED_OTHER behaving predictably under isolation, the tail-latency profile shifts. Re-baseline cyclictest after upgrading from 5.15/6.1 → 6.6+.

Other 6.6 changes:

• cgroup_no_v1=all works as a more robust gate to force pure v2.
• BPF kfunc improvements relevant to networking offload programs.

6.8 — kthread isolation, scheduler refinements

• Kthreads on isolated CPUs are honored more strictly. Some kernel kthreads that "leaked" onto isolated cores in 6.5 stay on housekeeping cores in 6.8.
• Improvements to RCU offload behavior (rcu_nocbs tighter).
• LSM stacking improvements (multiple LSMs at once: SELinux + AppArmor + Yama).

This is the Ubuntu 24.04 GA kernel.

6.11+ — NUMA balancing improvements

• Less aggressive page migration under NUMA balancing — the 5.x AutoNUMA reputation for stealing performance from steady-state HPC is partially fixed.
• NIC offload features land for 800G hardware.
• DAMON observability improvements.

6.12 (LTS) — current frontier

• TBD; expected to be the next decade's enterprise default.
• BPF arena features for shared maps across programs.

Kernels to avoid for HPC

Some kernel point releases have shipped with regressions specifically affecting GPU clusters. Track these and pin around them.

nvidia-peermem rebuild regression — recent incident pattern

In a recent incident, a kernel point release upgrade caused nvidia-peermem.ko to fail to rebuild via DKMS. The visible symptom: GPUs come up after reboot, nvidia-smi is fine, but multi-node NCCL falls back to host-staged (TCP) instead of GDR. Throughput collapses to single-digit Gbps where it should be 90+.

The detection — check post-reboot:

# Is peermem actually loaded?
lsmod | grep peermem
# nvidia_peermem         16384  0      <-- must be present

# If missing, check DKMS status
sudo dkms status | grep nvidia
# nvidia/535.230.02, 6.8.0-45-generic, x86_64: installed
# nvidia/535.230.02, 6.8.0-50-generic, x86_64: failed     <-- the new kernel didn't build

# Check the build log
sudo cat /var/lib/dkms/nvidia/535.230.02/build/make.log | tail -50
# error: implicit declaration of function 'follow_pte'
# (or similar — symbols moved or signatures changed)

The rollback:

# Option 1: boot back into the prior working kernel
sudo grub-set-default 'gnulinux-advanced-XXX>gnulinux-6.8.0-45-generic-advanced-XXX'
sudo update-grub
sudo reboot

# Option 2: install a newer NVIDIA driver that knows about the new kernel API
sudo apt install nvidia-driver-555      # or whichever is current

# Option 3: pin the old kernel and prevent further upgrades
sudo apt-mark hold linux-image-6.8.0-45-generic linux-headers-6.8.0-45-generic

The lesson: every kernel point upgrade on a GPU node should run through a canary first. Every. Single. One.

mlx5_ib symbol changes requiring OFED rebuild

Roughly once a year, the mlx5_ib code reorganizes. Recent example: in 6.6, several ib_umem_* helpers changed signatures. MLNX_OFED versions older than 24.04 fail to build against 6.6, and you see:

make[2]: *** [.../drivers/infiniband/hw/mlx5/main.o] Error 1
modpost: Symbol info of vmlinux is missing!

Resolution: upgrade MLNX_OFED to a version that supports the kernel, or stay on the older kernel. The MLNX_OFED release notes list explicit kernel support — read them before upgrading:

ofed_info -s
# MLNX_OFED_LINUX-24.10-1.1.4.0:

# Then check the release notes for that version's supported kernels

cgroup v2 device controller regressions

Periodic regressions in the device.allow cgroup controller affect GPU exposure to containers. Pattern: containers that asked for /dev/nvidia0 get permission denied, even though the device file is present. The kubelet log says cgroup setup failed.

# Symptom
kubectl describe pod gpu-pod
# Failed to start container: ... "cgroups: device controller error: ..."

# Investigation
ls -l /dev/nvidia0
# crw-rw-rw- 1 root root 195, 0 ...    <-- device exists
cat /sys/fs/cgroup/system.slice/<pod-cgroup>/cgroup.controllers
# cpu cpuset memory pids       <-- "devices" missing!

Resolution: pin to the prior kernel until a fix lands. This pattern has hit 5.15.x and 6.1.x point releases at different times.

Other "be aware" patterns

• io_uring security disclosures: several CVEs in 2023-2024 led some sites to disable io_uring entirely (kernel.io_uring_disabled=2). Verify your workload doesn't depend on it before disabling.
• MGLRU enabled by default in 6.7+: under heavy memcg pressure, MGLRU's eviction algorithm can cause workloads with large pinned memory (verbs MRs) to behave differently than under classic LRU. Re-baseline after 6.7+.
• EEVDF on isolated cores: rare but documented — some workloads with very tight 1-thread-per-core HPC spin loops see a few % regression. Mitigation: use SCHED_FIFO for the spinning thread.

Upgrade flow — the pattern that doesn't break a tenant

Five steps, in order:

1. Check NVIDIA driver compatibility

DKMS will rebuild nvidia.ko on apt/dnf upgrade. The rebuild can fail. Verify driver version supports the new kernel:

# What driver is installed
nvidia-smi | grep "Driver Version"
# Driver Version: 535.230.02

# What kernel are you upgrading to
apt list --upgradable 2>/dev/null | grep linux-image
# linux-image-generic/jammy-updates 6.8.0-50.51 amd64

# Check NVIDIA's compat matrix
# https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-XXX/
# OR look at recent DKMS rebuild logs from a canary node

If the driver doesn't list support for the new kernel, don't upgrade yet — wait for either a newer driver or the kernel to settle. The cost of "let me just try it" is a 4-hour incident.

2. Check MLNX_OFED compatibility

OFED release notes have a kernel matrix: which OFED versions support which kernels. Find the matrix for your installed OFED:

ofed_info -s
# MLNX_OFED_LINUX-24.10-1.1.4.0:

# Read the release notes for 24.10 — supported kernels list will include something like:
# Ubuntu 22.04 5.15.0, 6.5.0, 6.8.0
# Ubuntu 24.04 6.8.0, 6.11.0

If the new kernel isn't in the matrix, OFED won't rebuild correctly. Either upgrade OFED first, or skip the kernel.

3. Stage on one canary node

Cordon and drain one node; upgrade just that one:

# Pick a canary
CANARY=gpu-01

# Cordon and drain
kubectl cordon $CANARY
kubectl drain $CANARY --ignore-daemonsets --delete-emptydir-data

# SSH to the canary
ssh $CANARY

# Apply the kernel upgrade
sudo apt update
sudo apt install -y linux-image-6.8.0-50-generic linux-headers-6.8.0-50-generic

# Verify DKMS rebuilt cleanly
sudo dkms status | grep nvidia
# nvidia/535.230.02, 6.8.0-50-generic, x86_64: installed

sudo dkms status | grep mlnx
# mlnx-ofed-kernel/24.10.OFED..., 6.8.0-50-generic, x86_64: installed

# Reboot
sudo reboot

If anything in DKMS shows failed instead of installed, stop. Resolve before continuing.

4. Validate the canary

After reboot, run the layered validation:

# nvidia-smi must work
nvidia-smi
# All 8 GPUs listed, no errors

# IB / RoCE links up
ibstat
# State: Active for every port

# Module set loaded
lsmod | grep -E 'ib_|rdma|mlx5|peermem'
# (the full 9-module set from /etc/modules-load.d/rdma.conf)

# DCGM diag
dcgmi diag -r 2
# All checks PASS

# 2-rank intra-node nccl-test
mpirun -np 2 -x NCCL_DEBUG=WARN ./build/all_reduce_perf -b 1G -e 1G -n 5
# busbw should be at expected for the platform (check vs known-good baseline)

# Inter-node check (canary + a known-good node)
mpirun -np 16 --hostfile hosts ./build/all_reduce_perf -b 8G -e 8G -n 5

If any layer fails, roll back the canary and don't proceed. The cluster keeps running on the old kernel.

5. Bake, then roll out

Once the canary passes validation, leave it baking under real workload for 24-72 hours. Watch for:

• Kernel logs (dmesg --since=-72h or journalctl -k) — anything new and weird
• Application metrics: throughput, allreduce time, NCCL retransmits
• DCGM xid events
• HCA error counters

After the bake passes, roll out to the rest of the cluster in waves (10-25% per wave with 30 min between).

# Tag the wave 1 nodes
kubectl label nodes gpu-{02..10} kernel-upgrade-wave=1

# Cordon and drain wave 1
for n in gpu-{02..10}; do
  kubectl cordon $n
  kubectl drain $n --ignore-daemonsets --delete-emptydir-data --force
done

# SSH and upgrade in parallel (e.g. via clush or ansible)
clush -w gpu-{02..10} 'sudo apt install -y linux-image-X.Y.Z-N-generic && sudo reboot'

# Wait for them to come back
clush -w gpu-{02..10} -t 600 'uptime'

# Re-validate
clush -w gpu-{02..10} 'nvidia-smi -L | wc -l'
# 8
# 8
# 8
# (etc)

# Uncordon
for n in gpu-{02..10}; do kubectl uncordon $n; done

If wave 1 has issues, don't proceed to wave 2. Roll back wave 1 and figure it out.

Rollback

The fastest rollback is booting an older kernel that's still installed.

Ubuntu rollback via grub

# List installed kernels
ls /boot/vmlinuz-*
# /boot/vmlinuz-6.8.0-45-generic
# /boot/vmlinuz-6.8.0-50-generic

# Option 1: pick the older kernel for the next boot only
sudo grub-reboot 'Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-45-generic'
sudo reboot

# Option 2: set the older kernel as default permanently
sudo grub-set-default 'Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-45-generic'
sudo update-grub
sudo reboot

# Option 3: remove the bad kernel entirely
sudo apt remove linux-image-6.8.0-50-generic linux-headers-6.8.0-50-generic
sudo update-grub

RHEL rollback via grubby

# List entries
sudo grubby --info=ALL | grep -E '^(index|title|kernel)' 

# Set default to a specific entry by index
sudo grubby --set-default-index=1
sudo reboot

# Or by kernel path
sudo grubby --set-default=/boot/vmlinuz-5.14.0-XXX.el9.x86_64
sudo reboot

Pinning to prevent re-upgrade

# Ubuntu / Debian — apt-mark hold
sudo apt-mark hold linux-image-generic linux-headers-generic
sudo apt-mark hold linux-image-6.8.0-45-generic linux-headers-6.8.0-45-generic

# Verify
apt-mark showhold

# Release the hold when ready
sudo apt-mark unhold linux-image-generic linux-headers-generic

# RHEL / Rocky / Alma — exclude in dnf
sudo dnf install python3-dnf-plugin-versionlock
sudo dnf versionlock add kernel-5.14.0-XXX.el9
sudo dnf versionlock list

# Or globally exclude kernel upgrades
echo 'exclude=kernel*' | sudo tee -a /etc/dnf/dnf.conf

For longer-term pinning (across config reapplies), encode the hold in your config-management:

# ansible
- name: Hold kernel package
  ansible.builtin.dpkg_selections:
    name: linux-image-generic
    selection: hold

Rolling back across multiple installed kernels

# What's installed
dpkg -l | grep ^ii | grep linux-image
# ii  linux-image-6.8.0-45-generic   ...
# ii  linux-image-6.8.0-50-generic   ...

# Remove the bad one (note: requires booted into a different one)
sudo apt remove linux-image-6.8.0-50-generic
sudo apt autoremove        # cleans up old headers

# Verify grub re-generated
ls /boot/vmlinuz-*
sudo update-grub

Reading distro release notes correctly

For each kernel update, read the changelog before deploying. Where to look:

• Ubuntu: /usr/share/doc/linux-image-X.Y.Z-N-generic/changelog.Debian.gz. Or via apt: apt changelog linux-image-generic.
• Subscribe to ubuntu-security-announce: every USN that touches the kernel lands there with the CVE list.
• For deeper info: lwn.net's "Distribution kernel" weekly column tracks what's in each distro's update.
• For upstream context: git log --oneline v6.8.0-45..v6.8.0-50 -- drivers/infiniband/ (or whatever subsystem you care about).

When the changelog mentions any of these, slow down:

• "Updates to drivers/infiniband/" — RDMA stack changed
• "Updates to drivers/nvme/" — storage stack changed
• "Updates to mm/" — memory management changes
• "cgroup" — capacity for kubelet regressions
• "scheduler" — could affect isolation behavior

A clean kernel upgrade has changes only in places you don't touch (filesystems you don't use, drivers for hardware you don't have). When the diff is in your blast radius, canary it.

Reading a kernel oops / regression bug report

When a kernel update breaks a node, the diagnostic data is usually a kernel oops in dmesg or a hung-task report. Reading these is a skill.

Anatomy of a kernel oops

[  234.567890] BUG: kernel NULL pointer dereference, address: 0000000000000018
[  234.567891] #PF: supervisor read access in kernel mode
[  234.567892] #PF: error_code(0x0000) - not-present page
[  234.567893] PGD 0 P4D 0
[  234.567894] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  234.567895] CPU: 12 PID: 1234 Comm: kworker/12:1 Tainted: P     U     OE  6.8.0-50-generic
[  234.567896] Hardware name: Vendor PowerEdge XR8620 ...
[  234.567897] RIP: 0010:mlx5_ib_destroy_qp+0x123/0x4a0 [mlx5_ib]
[  234.567898] Code: 48 89 5d e0 4c 89 65 e8 4c 89 6d f0 4c 89 75 f8 e8 ...
[  234.567899] RSP: 0018:ffffb12345678cf8 EFLAGS: 00010282
[  234.567900] Call Trace:
[  234.567901]  <TASK>
[  234.567902]  ? __die_body+0x1a/0x60
[  234.567903]  ? page_fault_oops+0x163/0x2a0
[  234.567904]  ? exc_page_fault+0x77/0x180
[  234.567905]  mlx5_ib_destroy_qp+0x123/0x4a0 [mlx5_ib]
[  234.567906]  ib_destroy_qp_user+0x80/0x1f0 [ib_core]
[  234.567907]  uverbs_free_qp+0x33/0xb0 [ib_uverbs]
[  234.567908]  ...

What to extract from this:

The Tainted: ... OE flags signal the bug might be in out-of-tree modules (nvidia, MOFED) rather than upstream. Don't file an upstream bug for an out-of-tree crash — file with the vendor.

Searching for prior reports

The fast path:

# Search lkml + bugzilla
# Google: "mlx5_ib_destroy_qp NULL pointer 6.8"

# Or directly:
# https://bugzilla.kernel.org → search subsystem "infiniband"
# https://github.com/Mellanox/... → search issues
# https://github.com/NVIDIA/... → search issues

If your stack trace matches a known bug filed in the last 60 days, the fix is usually waiting in linux-stable or in a vendor patch. Roll back and wait.

Hung-task watchdog

[ 1234.567890] INFO: task python:23456 blocked for more than 120 seconds.
[ 1234.567891]       Tainted: P           OE     6.8.0-50-generic
[ 1234.567892] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1234.567893] task:python   state:D stack:0     pid:23456 ppid:23000 ...
[ 1234.567894] Call Trace:
[ 1234.567895]  __schedule+0x349/0x9d0
[ 1234.567896]  schedule+0x68/0x110
[ 1234.567897]  io_schedule+0x46/0x80
[ 1234.567898]  folio_wait_bit_common+0x131/0x310

Hung-task means a process has been in uninterruptible sleep (state:D) for >120s. For HPC, common causes:

• NFS mount lost the server (hung on nfs_* functions)
• Storage backend (Weka, Ceph) unresponsive (hung on wekafs_* / ceph_*)
• A GPU XID 79 — driver fell off the bus, processes blocked on syscalls

The Call Trace tells you which subsystem. folio_wait_bit_common points to page cache I/O wait — likely storage. mlx5_* would point to RDMA.

Hung tasks aren't crashes — the kernel is still running, but a tenant's job is stuck. Often the right move is kill -9 the stuck process and investigate the underlying I/O subsystem.

Pinning specific kernel — reference patterns

Ubuntu — pin via apt

# /etc/apt/preferences.d/99-pin-kernel
Package: linux-image-6.8.0-45-generic
Pin: version 6.8.0-45*
Pin-Priority: 1001

Package: linux-image-generic
Pin: version 6.8.0.45*
Pin-Priority: 1001

Verify:

sudo apt-cache policy linux-image-generic
# linux-image-generic:
#   Installed: 6.8.0.45.43
#   Candidate: 6.8.0.45.43
#   Version table:
#  *** 6.8.0.45.43 1001
#         100 /var/lib/dpkg/status
#      6.8.0.50.48 500
#         500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages

The candidate stays at 45 even though 50 is available — pin worked.

RHEL — exclude in dnf or use versionlock

# /etc/dnf/dnf.conf
exclude=kernel*

# Or per-package
sudo dnf versionlock add kernel-5.14.0-427.20.1.el9_4

Avoiding unattended kernel upgrades

The unattended-upgrades package on Ubuntu can install kernels overnight:

# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Package-Blacklist {
    "linux-image-.*";
    "linux-headers-.*";
    "linux-modules-.*";
};

For HPC, kernel upgrades are a planned change, not an automated one. Disable.

Subscribe to the right channels

For a healthy upgrade cadence, subscribe to:

• ubuntu-security-announce — every USN affecting Ubuntu
• rhsa-announce — Red Hat security advisories
• lwn.net — kernel news, distro tracking, weekly digest of stable releases
• NVIDIA driver release notes RSS — newest Tesla / data center driver releases
• MLNX_OFED release notes — every OFED release with kernel matrix
• kernel.org stable releases — the upstream LTS/stable feed

The signal-to-noise gets better with each. lwn.net's "Distributions" column is the single most useful free resource for tracking kernel changes that matter to operators.

Common failure modes

"Kernel upgraded but nvidia-smi gone"

nvidia-smi
# NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.

sudo dkms status
# nvidia/535.230.02, 6.8.0-50-generic, x86_64: failed

Cause: NVIDIA driver doesn't support the new kernel. Resolution: rollback or driver upgrade.

"MLNX_OFED kernel modules disappeared"

lsmod | grep mlx5
# (empty)

sudo modprobe mlx5_core
# modprobe: ERROR: could not insert 'mlx5_core': Exec format error

dmesg | grep mlx5
# mlx5_core: version magic '6.8.0-45-generic SMP preempt mod_unload modversions' should be '6.8.0-50-generic SMP preempt mod_unload modversions'

Cause: OFED was built for the old kernel; new kernel won't load it. Resolution: rebuild OFED for the current kernel, or rollback.

sudo dpkg-reconfigure mlnx-ofed-kernel-dkms
# or
sudo /etc/init.d/openibd restart

"Worked yesterday, broken today, no kernel upgrade"

Check last reboot, dpkg.log, dnf history:

last reboot | head -5
# reboot   system boot  6.8.0-50-generic Wed May  1 03:14   still running
# reboot   system boot  6.8.0-45-generic Tue Apr 30 18:00 - 03:13 (09:13)

The new kernel booted at 03:14 — that's likely an unattended upgrade. Re-read /etc/apt/apt.conf.d/50unattended-upgrades and pin.

"DKMS shows installed but module not loading"

sudo dkms status
# nvidia/535.230.02, 6.8.0-50-generic, x86_64: installed       <-- says installed
sudo modprobe nvidia
# modprobe: ERROR: could not insert 'nvidia': Required key not available

Cause: Secure Boot is on, and DKMS-built modules need to be signed with a key the kernel trusts. Either disable Secure Boot in UEFI or generate and enroll a MOK (Machine Owner Key):

sudo update-secureboot-policy --enroll-key
# Walk through the MOK enrollment dialog at next boot

"Kernel and headers mismatched"

sudo apt install linux-image-6.8.0-50-generic
# (forgot headers)

sudo dkms autoinstall
# Error: Cannot find sources for kernel 6.8.0-50-generic

Always install headers + image together:

sudo apt install linux-image-6.8.0-50-generic linux-headers-6.8.0-50-generic

"Older kernel removed; nothing to roll back to"

sudo apt autoremove is aggressive — it removes "unused" packages, including older kernels. Run it explicitly never on a node mid-upgrade:

# DO NOT
sudo apt autoremove -y

# Instead, keep the prior known-good kernel installed
sudo apt-mark hold linux-image-6.8.0-45-generic linux-headers-6.8.0-45-generic

Always keep at least 2 kernels installed: current + prior known-good.

See also

• GRUB cmdline — boot params; how to switch kernels at boot
• Network speed tiers — per-tier kernel feature requirements
• NVIDIA driver — DKMS, secure boot, install/upgrade flow
• MLNX_OFED — kernel-version matrix, persistent module loading
• peermem — module that fails silently when DKMS rebuild misses it
• Driver / firmware mismatch — incident pattern
• Sysctl tuning — runtime knobs that depend on kernel features

External:

• kernel.org/category/releases.html — stable / LTS release feed
• lwn.net — weekly kernel news, distribution column
• ubuntu-security-announce mailing list
• access.redhat.com/articles/3078 — RHEL kernel support policy
• Ubuntu HWE kernel policy: wiki.ubuntu.com/Kernel/LTSEnablementStack
• NVIDIA driver release notes: docs.nvidia.com/datacenter/tesla/
• MLNX_OFED release notes (per-version): docs.nvidia.com/networking/