GRUB cmdline for GPU/HPC nodes

Boot-time kernel parameters every GPU node should ship with — IOMMU, ACS, hugepages, isolcpus, mitigations — plus update-grub vs grubby.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

A handful of kernel parameters cannot be set at runtime. They have to be on the GRUB cmdline before the kernel comes up, because they affect things like IOMMU mode, hugepage reservation from physically-contiguous memory, and CPU isolation.

This page is the canonical list for a single-tenant GPU/HPC node, with the security tradeoffs called out so you know which ones not to copy onto a multi-tenant box.

The default GPU-node cmdline

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash \
  iommu=pt intel_iommu=on \
  pcie_acs_override=downstream,multifunction \
  default_hugepagesz=1G hugepagesz=1G hugepages=64 \
  transparent_hugepage=madvise \
  numa_balancing=disable \
  mitigations=off \
  nvme_core.io_timeout=4294967295"

That's the production-ready string for a typical 8×H100/8×H200 single-tenant compute node. Each token explained below.

For AMD platforms swap intel_iommu=on for amd_iommu=on. Everything else is identical.

Token-by-token

`iommu=pt`

IOMMU passthrough mode. The IOMMU is configured but does not translate DMA for non-VFIO devices — they get direct host-physical addressing. This is the only mode that gives full PCIe peer-to-peer bandwidth. Without pt, every DMA pays an IOMMU translation cost; with iommu=on and no pt, P2P collapses.

Mode	Effect
`iommu=off`	IOMMU disabled entirely. No VFIO. Full P2P. Don't use — you lose all isolation.
`iommu=on`	IOMMU enabled, all DMA translated. Slow.
`iommu=pt`	IOMMU enabled but bypassed for non-VFIO. What you want.

`intel_iommu=on` / `amd_iommu=on`

Activates the Intel VT-d (or AMD-Vi) hardware unit. Pair with iommu=pt to get the right behavior.

`pcie_acs_override=downstream,multifunction`

Disables PCIe ACS source/redirect bits at boot. Required for GPU peer-to-peer over PCIe. Only works on patched kernels (Ubuntu HWE, several distro builds). See the ACS deep-dive.

`default_hugepagesz=1G hugepagesz=1G hugepages=N`

Reserve N × 1 GiB hugepages from physically-contiguous memory at boot. 1 GiB pages must be allocated at boot — fragmentation later makes them impossible. See hugepages.

To request both pools:

default_hugepagesz=1G hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=8192

`transparent_hugepage=madvise`

THP only when madvise(MADV_HUGEPAGE) asks for it. CUDA and jemalloc both ask. Avoids the khugepaged jitter of =always, gives the wins on hot mappings.

Alternatives: =always (more wins, more jitter), =never (off entirely — only choose this for jitter-sensitive RT workloads).

`numa_balancing=disable`

Disables AutoNUMA. AutoNUMA migrates pages between NUMA nodes based on observed access patterns, which is great for misc workloads and terrible for steady-state HPC where you've already pinned things correctly. NCCL collectives in particular develop mysterious latency spikes when AutoNUMA decides to relocate pages mid-step.

`mitigations=off`

Disables Spectre/Meltdown/MDS/L1TF mitigations. The performance recovery is measurable (5-15% on memory-heavy workloads) and the security tradeoff is that a malicious tenant on the same kernel can read kernel/other-process memory.

Use case	`mitigations` setting
Single-tenant bare-metal compute	`off`
Multi-tenant SaaS	leave default (`auto`)
Public-facing edge node	`auto` or `auto,nosmt`

Only set mitigations=off on internal compute clusters where you control every workload that runs.

`nvme_core.io_timeout=4294967295`

Sets the NVMe per-request timeout to 2³² - 1 seconds (effectively never). On busy GPU nodes with many NVMe namespaces and heavy concurrent I/O, the default 30s timeout sometimes fires under load and the device gets reset, taking a training run down with it. Disabling the timeout shifts the failure mode from "kernel resets the device" to "kernel keeps waiting" — which is what you want when the device is healthy and just overloaded.

This is a stability-over-correctness choice. Some sites prefer nvme_core.io_timeout=180 instead.

Extras worth knowing

Param	When to use
`isolcpus=24-47,72-95`	Carve out cores for jitter-sensitive realtime workloads. Kernel scheduler won't put ordinary tasks on these.
`nohz_full=24-47,72-95`	Adaptive ticks — kernel doesn't tick on isolated cores. Pairs with isolcpus for HPC jitter.
`rcu_nocbs=24-47,72-95`	Move RCU callbacks off isolated cores. Same goal.
`processor.max_cstate=1`	Disable deep C-states. Lower wakeup latency, more idle power.
`intel_pstate=disable`	Use acpi_freq governor instead of intel_pstate. Some tuning recipes prefer this.
`pci=realloc`	Force PCIe BAR reallocation at boot. Sometimes needed on systems with wonky firmware.
`acpi_enforce_resources=lax`	Allow IPMI/sensor tools to read protected ranges (some BMCs need this).
`crashkernel=auto`	Reserve memory for kdump. Off on production HPC; pretend a node that crashed is dead.

For most clusters you won't need isolcpus/nohz_full/rcu_nocbs unless you have a specific RT profile. They are mentioned because they show up in tuning guides and people sometimes copy them blindly — don't, unless you measured a need.

Applying the change

Debian / Ubuntu

# Edit /etc/default/grub, set GRUB_CMDLINE_LINUX_DEFAULT="..."
update-grub
# or equivalently:
grub-mkconfig -o /boot/grub/grub.cfg
reboot

RHEL / Rocky / Alma

# /etc/default/grub edits work but grubby is the supported tool
grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on pcie_acs_override=downstream,multifunction"

# Verify the args landed on every kernel entry
grubby --info=ALL | grep -E '^(kernel|args)'
reboot

grubby modifies each kernel entry in /boot/loader/entries/ directly; you don't need to regenerate grub.cfg. Use it when you need to add/remove a single arg without rewriting the whole cmdline.

Removing an arg

# Debian/Ubuntu — edit /etc/default/grub by hand, then update-grub
# RHEL — grubby
grubby --update-kernel=ALL --remove-args="mitigations=off"

Validation after reboot

# What the kernel actually got
cat /proc/cmdline
# BOOT_IMAGE=/boot/vmlinuz-... iommu=pt intel_iommu=on pcie_acs_override=downstream,multifunction default_hugepagesz=1G ...

# Did each setting take effect?
dmesg | grep -i 'IOMMU\|ACS Override\|HugeTLB'
# [    0.024] DMAR: IOMMU enabled
# [    0.025] PCI: ACS override active: downstream multifunction
# [    0.030] HugeTLB: registered 1.00 GiB page size, pre-allocated 64 pages

# Hugepage pool reserved?
grep ^Huge /proc/meminfo

# Mitigations actually off?
grep . /sys/devices/system/cpu/vulnerabilities/* | head
# /sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Vulnerable      <-- mitigations off
# /sys/devices/system/cpu/vulnerabilities/spectre_v1:Vulnerable
# ...

If dmesg doesn't show ACS Override active despite the cmdline having it, your kernel doesn't have the patch — fall back to the runtime setpci method.

Common mistakes

Editing GRUB_CMDLINE_LINUX (no _DEFAULT suffix) when grub uses _DEFAULT, or vice versa. Different distros ship one or the other; check your /etc/default/grub for which is non-empty.
Forgetting to run update-grub / grub-mkconfig after editing on Debian-likes. The grub.cfg has the actual value the bootloader reads.
Putting pcie_acs_override= on a kernel that doesn't have the patch (silent no-op).
Adding iommu=on without pt and watching P2P fall apart.
Reserving 1 GiB hugepages on a 256 GiB node — the kernel can't grow them later.

Reference cmdlines per node profile

The default cmdline at the top of this page is the minimum viable production string. Real nodes layer on isolation, IRQ steering, idle/cstate forcing, and per-NIC-speed considerations. Below are three canonical full cmdlines, annotated.

Profile A — Ubuntu 22.04 LTS GPU node, ConnectX-7 200 Gbps NDR IB

The 200G NDR tier is the current sweet spot for production multi-node training. PCIe gen5 x16 saturates the NIC; isolation is recommended but not strictly mandatory below 4 NICs/node. Skylake / Sapphire Rapids / Genoa CPUs.

BOOT_IMAGE=/boot/vmlinuz-5.15.0-118-generic ro \
  intel_iommu=on iommu=pt \
  pci=realloc=on,assign-busses \
  pcie_acs_override=downstream,multifunction \
  default_hugepagesz=1G hugepagesz=1G hugepages=64 \
  transparent_hugepage=madvise \
  nowatchdog mitigations=off \
  intel_idle.max_cstate=1 processor.max_cstate=1 \
  isolcpus=2-55,58-111 nohz_full=2-55,58-111 rcu_nocbs=2-55,58-111

Token	Purpose at 200G	What breaks if you drop it
`intel_iommu=on iommu=pt`	IOMMU active in passthrough — DMA goes direct without translation	GPU-NIC P2P collapses; GDR throughput drops 50%+
`pci=realloc=on,assign-busses`	Lets kernel resize BARs and renumber buses if firmware did it wrong	Some CX-7 cards expose 0-byte BAR2; mlx5 fails to load
`pcie_acs_override=downstream,multifunction`	Disables ACS so PCIe peer-to-peer is allowed within a switch	GPU↔NIC DMA cannot bypass root complex; nccl-tests busbw caps low
`default_hugepagesz=1G hugepagesz=1G hugepages=64`	Reserve 64 GiB of 1 GiB pages at boot for CUDA/RDMA buffer pools	TLB miss rate climbs; allreduce time jitters; some allocators fall back to 4 KiB pages
`transparent_hugepage=madvise`	THP only when explicitly requested	jemalloc / glibc-malloc lose THP wins on heap; mild perf loss
`nowatchdog mitigations=off`	No periodic watchdog hrtimer; no Spectre/Meltdown overhead	Adds 5-15% syscall overhead and per-core 1ms tick
`intel_idle.max_cstate=1 processor.max_cstate=1`	Disable deep C-states (C3/C6)	Wakeup latency 100 µs+ ruins NCCL latency budget
`isolcpus=2-55,58-111`	Reserve 0-1 + 56-57 for housekeeping; 2-55 + 58-111 for compute	Kernel kworkers land on compute cores → unpredictable jitter
`nohz_full=2-55,58-111`	No timer tick on isolated cores when one task runnable	Per-core 1 ms tick costs ~1 µs each; multiplied by ranks = collective tail latency
`rcu_nocbs=2-55,58-111`	Move RCU callbacks off isolated cores to housekeeping kthreads	nohz_full doesn't actually go tickless; RCU kicks every grace period

For this profile we deliberately do NOT set numa_balancing=disable on cmdline — Ubuntu 22.04 ships with it disabled by default in the inbox kernel. Verify post-boot:

cat /proc/sys/kernel/numa_balancing
# 0     <-- already off

If it shows 1, add numa_balancing=disable to cmdline OR set via /etc/sysctl.d/40-kernel.conf.

Profile B — Ubuntu 24.04 LTS H100 + 400 Gbps NDR

400G NDR is where the kernel TCP path ceases being viable for line rate; you'll be running RDMA / DPDK / io_uring. The cmdline reflects more aggressive isolation, larger hugepage pool, and intel_pstate=passive to let the application drive CPU frequency.

BOOT_IMAGE=/boot/vmlinuz-6.8.0-45-generic ro \
  intel_iommu=on iommu=pt \
  pci=realloc=on,assign-busses \
  pcie_acs_override=downstream,multifunction \
  default_hugepagesz=1G hugepagesz=1G hugepages=128 \
  transparent_hugepage=madvise \
  nowatchdog nosoftlockup mitigations=off \
  intel_idle.max_cstate=1 processor.max_cstate=1 \
  intel_pstate=passive idle=halt \
  numa_balancing=disable \
  isolcpus=managed_irq,domain,4-55,60-111 \
  nohz_full=4-55,60-111 \
  rcu_nocbs=4-55,60-111 rcu_nocb_poll \
  irqaffinity=0-3,56-59 \
  skew_tick=1 nmi_watchdog=0

New / changed token	Why at 400G
`hugepages=128`	DPDK / RDMA buffer pools at 400G need 128 GiB of 1G pages; without them the NIC RX descriptor pool falls back to 4K pages and stalls under burst
`intel_pstate=passive`	Lets governor (e.g. `performance`) drive frequency directly via `cpufreq` instead of HWP autonomy. More predictable for tail-latency workloads
`idle=halt`	Use HLT (cstate 1) instead of MWAIT — no deep cstate entry/exit at all
`numa_balancing=disable`	24.04's 6.8 kernel re-enables AutoNUMA by default — explicit disable is needed
`isolcpus=managed_irq,domain,...`	The 5.4+ form: also exclude managed IRQs and sched domains from isolated cores
`rcu_nocb_poll`	Kthreads poll for RCU callbacks instead of waking via IPI — fewer cross-CPU interrupts at the cost of housekeeping CPU
`irqaffinity=0-3,56-59`	Static cmdline IRQ steering — must agree with isolcpus housekeeping range
`skew_tick=1`	Stagger ticks across cores; on 192-core CPUs the synchronized tick storm shows up in cycle-accurate profiles
`nmi_watchdog=0`	Frees up perf counters that otherwise are owned by the watchdog

The numa_balancing=disable here is critical: AutoNUMA migrates pages mid-job, and at 400G/NIC the per-page migration cost can stall ring buffer refill long enough to drop frames.

Profile C — B200 / 800 Gbps XDR (forward-looking)

ConnectX-8 XDR at 800 Gbps is just shipping in 2025-2026. Most production deployments are on PCIe gen6 x16 or gen5 x32. This profile assumes you have either of those plus NVLink-class internal fabric (B200 + NVL72). Few public references exist; treat this as a working hypothesis.

BOOT_IMAGE=/boot/vmlinuz-6.11.0-X-generic ro \
  intel_iommu=on iommu=pt \
  pci=realloc=on,assign-busses pcie=pcie_bus_perf \
  pcie_acs_override=downstream,multifunction \
  default_hugepagesz=1G hugepagesz=1G hugepages=256 \
  transparent_hugepage=madvise \
  nowatchdog nosoftlockup mitigations=off \
  intel_idle.max_cstate=1 processor.max_cstate=1 \
  intel_pstate=passive idle=halt \
  numa_balancing=disable \
  isolcpus=managed_irq,domain,8-127,136-255 \
  nohz_full=8-127,136-255 \
  rcu_nocbs=8-127,136-255 rcu_nocb_poll \
  irqaffinity=0-7,128-135 \
  skew_tick=1 nmi_watchdog=0 audit=0 \
  workqueue.power_efficient=0 \
  cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1

New / changed token	Why at 800G
`pcie=pcie_bus_perf`	Tells kernel to maximize PCIe `MaxPayloadSize` and `MaxReadRequest` across the topology — at 800G even a 256-byte MPS limit costs measurable bandwidth
`hugepages=256`	DPDK pools at 8x800 = 6.4 Tbps aggregate need 256 GiB of 1 G pages just for the NIC side
`audit=0`	Kernel audit cost per syscall becomes measurable when the application makes 50M syscalls/s polling RDMA CQs
`workqueue.power_efficient=0`	Default workqueues prefer batching; for 800G the workqueue overhead matters less than the latency of waiting for batches
`cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1`	Force pure cgroup v2 — needed for the per-NIC bandwidth controllers and the `cpuset.cpus.partition=isolated` flow to interact correctly with kubelet at this scale

The isolcpus range here assumes a 256-core / 128-physical box (e.g., dual-socket Bergamo / Granite Rapids).

Tuning per network speed — quick matrix

The cmdline above is one slice; here is the broader per-tier guidance. See /docs/kernel-tuning/network-speed-tiers for the deep treatment.

Speed	hugepages (1 GiB)	isolcpus rec.	NIC IRQ count	Min ring buffer	Notes
100G EDR/HDR	32	none required	32-64 per port	4096	Single core can saturate; rare to hit kernel limits
200G NDR	64	recommended	64 per port	8192	RoCE PFC config matters; CPU multi-queue required
400G NDR	128	required	128 per port	16384	DPDK / io_uring on backend FS; PCIe gen5; 1G pages mandatory
800G XDR	256	required	256 per port	32768	NVLink-class internal; PCIe gen6 / gen5 x32; DPU offload normal

The hugepages column is per-node (assuming 8 NICs/node). The "NIC IRQ count" is per NIC port, set via ethtool -L combined N. The "min ring buffer" is the minimum ethtool -G rx N tx N you should run; many CX-7 / CX-8 cards support ring sizes up to 32768.

If you cannot do isolcpus at 400G/800G (e.g. mixed-tenant, opportunistic workload), expect 60-80% of line rate on the kernel-TCP path even with all sysctls maxed. RDMA verbs path is unaffected.

Applying the change — distro specifics

Debian / Ubuntu — `update-grub`

# Edit /etc/default/grub
sudo vi /etc/default/grub

# Regenerate /boot/grub/grub.cfg
sudo update-grub
# or equivalently:
sudo grub-mkconfig -o /boot/grub/grub.cfg

# On EFI systems (which is most of them now), also regenerate the EFI cfg
ls /boot/efi/EFI/ubuntu/grub.cfg && sudo grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg

sudo reboot

After reboot, cat /proc/cmdline will show the active cmdline — it should match what you put in GRUB_CMDLINE_LINUX_DEFAULT. If it doesn't, you edited the wrong file or didn't run update-grub.

RHEL / Rocky / Alma — `grubby`

grubby modifies each kernel entry in /boot/loader/entries/ directly. Don't edit /etc/default/grub and forget — grubby will work but only on the entries it knows about.

# Add args to ALL kernel entries (including future ones via /etc/default/grub)
sudo grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on numa_balancing=disable"

# Verify
sudo grubby --info=ALL | grep -E '^(kernel|args)'

# Remove an arg
sudo grubby --update-kernel=ALL --remove-args="mitigations=auto"

# Set the default kernel by index (after listing them)
sudo grubby --info=ALL | head
sudo grubby --set-default-index=0

sudo reboot

For persistence across kernel installs, also update /etc/default/grub's GRUB_CMDLINE_LINUX= so newly-installed kernels inherit the same args. grubby modifies existing entries; /etc/default/grub is read by grub2-mkconfig for new ones.

Persistence across kernel upgrades

The trap: you set a cmdline arg with grubby on RHEL, then dnf upgrade kernel installs a new kernel. The new kernel entry inherits args from /etc/default/grub's GRUB_CMDLINE_LINUX, not from your previous grubby change. If you only used grubby --update-kernel=ALL, the new entry has the old default args, missing your tuning.

The safe pattern:

# Always update both
sudo grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on"
sudo sed -i 's|^GRUB_CMDLINE_LINUX="|GRUB_CMDLINE_LINUX="iommu=pt intel_iommu=on |' /etc/default/grub
# (or just edit by hand — the goal is /etc/default/grub matches what's on entries)

# After kernel upgrade, verify the new entry has the args
sudo grubby --info=ALL | grep ^args

On Ubuntu, this is simpler — update-grub regenerates from /etc/default/grub every time, so as long as GRUB_CMDLINE_LINUX_DEFAULT is right, every kernel entry inherits it.

Verifying after reboot

The single most important post-boot check is cat /proc/cmdline. If your edits aren't there, nothing else matters.

# What the kernel actually got
cat /proc/cmdline
# BOOT_IMAGE=/boot/vmlinuz-6.8.0-45-generic ro intel_iommu=on iommu=pt ... isolcpus=4-55,60-111 ...

# Same data, with kernel timestamp
dmesg | grep -i 'kernel command line'
# [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-... intel_iommu=on iommu=pt ...

# Hugepage pool actually reserved?
cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
# 128
cat /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages
# 128                  <-- nothing using them yet

grep ^Huge /proc/meminfo
# AnonHugePages:    524288 kB
# ShmemHugePages:        0 kB
# FileHugePages:         0 kB
# HugePages_Total:     128
# HugePages_Free:      128
# Hugepagesize:    1048576 kB
# Hugetlb:       134217728 kB

# IOMMU mode actually passthrough?
dmesg | grep -i 'IOMMU\|DMAR'
# [    0.024] DMAR: IOMMU enabled
# [    0.025] iommu: Default domain type: Passthrough (set via kernel command line)

# ACS override active?
dmesg | grep -i 'ACS Override'
# [    0.030] PCI: ACS override active: downstream multifunction

# Mitigations actually off?
grep . /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null | head
# /sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Vulnerable
# /sys/devices/system/cpu/vulnerabilities/spectre_v1:Vulnerable
# /sys/devices/system/cpu/vulnerabilities/spectre_v2:Vulnerable
# (any "Mitigation: ..." line means mitigations DIDN'T fully disable)

# Isolated cores actually isolated?
cat /sys/devices/system/cpu/isolated
# 4-55,60-111

cat /sys/devices/system/cpu/nohz_full
# 4-55,60-111

# CPU governor on a compute core
cat /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor
# performance

cat /sys/devices/system/cpu/cpu10/cpuidle/state*/disable
# 0 1 1 1            <-- C0 (POLL) active, C1+ disabled (depending on exact policy)

If nr_hugepages shows fewer than you asked for, the kernel could not allocate them at boot — typically because memory is fragmented after recent reboots or because there isn't enough physically-contiguous memory. Reserve them at boot via cmdline, never via runtime echo > nr_hugepages.

Common pitfalls

Quoting issues in `/etc/default/grub`

The cmdline is wrapped in double quotes. If your value itself contains a ", or you have a stray newline, update-grub will silently truncate. Common failure: pasting from a wiki that wraps the line.

# WRONG — broken across two lines
GRUB_CMDLINE_LINUX_DEFAULT="quiet
  iommu=pt"

# CORRECT — use line continuation \
GRUB_CMDLINE_LINUX_DEFAULT="quiet \
  iommu=pt intel_iommu=on \
  default_hugepagesz=1G hugepagesz=1G hugepages=64"

The backslash-newline pattern works because /etc/default/grub is sourced as shell. update-grub will collapse it to a single line in the generated grub.cfg. Verify with grep CMDLINE /boot/grub/grub.cfg.

`GRUB_CMDLINE_LINUX` vs `GRUB_CMDLINE_LINUX_DEFAULT`

Two variables; both are appended to the kernel cmdline, but only one of them is included in the recovery / single-user entry.

Var	Used in normal boot	Used in recovery
`GRUB_CMDLINE_LINUX`	Yes	Yes
`GRUB_CMDLINE_LINUX_DEFAULT`	Yes	No

For HPC tunables you almost always want GRUB_CMDLINE_LINUX_DEFAULT — you don't want mitigations=off in recovery mode, you want the system in its safest config. But for things like intel_iommu=on (which you need even to get into recovery and read disks via passthrough), use GRUB_CMDLINE_LINUX.

Check what your distro ships:

grep -E '^GRUB_CMDLINE_LINUX' /etc/default/grub
# GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
# GRUB_CMDLINE_LINUX=""

The non-empty one is where the distro put its defaults. Ubuntu uses _DEFAULT; many cloud images use GRUB_CMDLINE_LINUX for everything.

EFI-only systems

On modern hardware, /boot/grub/grub.cfg may not be the file the firmware actually reads. The bootloader path is:

firmware → /boot/efi/EFI/ubuntu/grubx64.efi → /boot/efi/EFI/ubuntu/grub.cfg → /boot/grub/grub.cfg

update-grub regenerates /boot/grub/grub.cfg. The EFI shim's grub.cfg typically just sources the main one, so this is fine — but on some custom installs, the EFI cfg has hardcoded paths or its own cmdline. If cat /proc/cmdline shows old args after update-grub, check:

sudo find /boot/efi -name 'grub.cfg' -exec grep -l linux {} \;
sudo cat /boot/efi/EFI/ubuntu/grub.cfg

Some BIOS/UEFI firmwares also let you override the kernel cmdline at boot — ruling that out is part of triage when "I changed grub but cmdline didn't change".

Forgetting `update-grub` after editing

sudo vi /etc/default/grub      # edit
sudo reboot                     # WRONG — cmdline didn't update

Edits to /etc/default/grub only take effect after update-grub (or grub-mkconfig) regenerates /boot/grub/grub.cfg. The bootloader reads grub.cfg, not the source file.

sudo vi /etc/default/grub
sudo update-grub
sudo reboot                     # now works

`pcie_acs_override=` on an unpatched kernel

Some kernel builds drop the ACS override patch. In that case the parameter is silently ignored — dmesg | grep ACS shows nothing — and your GPU P2P bandwidth is reduced. Workaround: runtime via setpci (see ACS). Or pick a kernel where it's patched (Ubuntu HWE, several HPC distros).

Reserving more 1 GiB hugepages than physical memory minus overhead

The kernel needs ~10-20 GiB for itself, the page table, and the buddy allocator. Asking for hugepages=240 on a 256 GiB node will result in a partial allocation (HugePages_Total: 192 or so) and dmesg will show Allocating N huge pages followed by warnings. Cap your hugepage reservation at ~75% of total RAM.

Cmdline length limit

Linux kernel cmdline is capped at COMMAND_LINE_SIZE, which on x86_64 is currently 2048 bytes. Above that the kernel silently truncates. If you have a very long cmdline (multi-NUMA isolcpus + many console= options + custom args), you may hit this. Audit:

wc -c /proc/cmdline
# 1247 /proc/cmdline       <-- under the limit, fine

On the rare occasion this matters, condense isolcpus= ranges (use 2-95 instead of 2,3,4,...,95) and drop redundant args.