Jitter isolation for HPC: isolcpus, nohz_full, RT scheduler

In HPC, the slowest rank in a collective is the speed of the whole collective. If 999 of 1000 ranks finish a 100 ms iteration in 100 ms but one rank takes 105 ms because the kernel decided to run a kworker on that core, the iteration takes 105 ms and you've thrown away 5% of your supercomputer.

Jitter is the variance in how long the same operation takes on different cores or different iterations. Removing it on a compute node is a combination of:

1. Telling the scheduler "don't run anything else on these cores"
2. Telling the kernel "don't even tick the timer on these cores when they're busy"
3. Pinning interrupts away from compute cores
4. Disabling background kernel work that periodically wakes things up

This page covers each layer with the boot params, the runtime checks, and the verification that the noise is actually gone.

Why jitter matters more on HPC than elsewhere

A web server with a 50 µs jitter on one CPU pays nothing — the request takes 5 ms anyway. An HPC job is different:

• All-reduce barriers: collective only finishes when the last rank arrives. Worst-case jitter dominates.
• CPU offload: many HPC apps offload to GPU but spin on CPU during synchronization. CPU stalls show up as GPU idle.
• NCCL: bootstrap, sync, and collective progress run on CPU threads. Latency spikes of even 1 ms cause slowdowns measurable across thousands of GPUs.
• Strong scaling: as you scale up the cluster, the probability that some core somewhere is jittering on each iteration approaches 1. Without isolation, scaling falls off a cliff at 1000+ ranks.

The full HPC GPU-node cmdline (Ubuntu 22.04, 192-core box)

GRUB_CMDLINE_LINUX_DEFAULT="quiet \
  iommu=pt intel_iommu=on \
  default_hugepagesz=1G hugepagesz=1G hugepages=64 \
  transparent_hugepage=madvise \
  numa_balancing=disable \
  mitigations=off \
  isolcpus=managed_irq,domain,8-95,104-191 \
  nohz_full=8-95,104-191 \
  rcu_nocbs=8-95,104-191 \
  rcu_nocb_poll \
  irqaffinity=0-7,96-103 \
  nosoftlockup \
  nowatchdog \
  audit=0 \
  intel_pstate=disable \
  processor.max_cstate=1 \
  intel_idle.max_cstate=0 \
  skew_tick=1 \
  nmi_watchdog=0 \
  nvme_core.io_timeout=4294967295"

This is the production string. It reserves cores 0-7 + 96-103 (16 cores total counting SMT) for the kernel and system services, and isolates 8-95 + 104-191 (176 cores) for the workload.

The reasoning, token by token:

isolcpus=managed_irq,domain,8-95,104-191

The classic CPU isolation parameter. Cores in this list are removed from the scheduler's load balancer. Tasks land there only if explicitly placed (via taskset, cpuset, or scheduler affinity).

The managed_irq,domain flags are modern (kernel 5.4+):

• managed_irq — also exclude these cores from kernel-managed IRQ targets
• domain — exclude from sched-domain load balancing (the "old" behavior)

For HPC, you almost always want both.

Modern style (>= 5.10) actually prefers cpuset-based isolation through cgroup v2's cpuset.cpus.partition=isolated instead of isolcpus= on the cmdline. But isolcpus= is simpler and works on all kernels.

nohz_full=8-95,104-191

Tickless cores. The kernel normally ticks every CPU at HZ (typically 250 or 1000) for accounting and timer wheel. On nohz_full cores, when only one task is runnable, the tick is suppressed entirely — the core runs uninterrupted until something else needs to run.

This is the single biggest jitter improvement on a busy compute node. Without it, every 1 ms (or 4 ms) the timer fires, costs ~1 µs of overhead, and might trigger a scheduler decision. With it, the core ticks at most once per second for housekeeping.

The catch: nohz_full requires rcu_nocbs to cover the same range, otherwise RCU callbacks tick the core anyway.

rcu_nocbs=8-95,104-191 + rcu_nocb_poll

Read-Copy-Update is the kernel's lockless concurrency primitive. By default, every core handles its own RCU callbacks (small bookkeeping at end of grace periods). On rcu_nocbs cores, those callbacks are offloaded to dedicated kthreads (rcuo*) running on housekeeping cores.

rcu_nocb_poll makes those offload kthreads poll instead of waiting for IPIs — slightly more CPU on the housekeeping cores, much less interruption on isolated cores.

irqaffinity=0-7,96-103

Pins all kernel-managed IRQ handlers to cores 0-7 + 96-103 (the 16 housekeeping cores). Without this, IRQs land on whichever CPU was idle, including your compute cores.

This is reinforced at runtime by stopping irqbalance and writing /proc/irq/<n>/smp_affinity directly (see below).

nosoftlockup + nowatchdog + nmi_watchdog=0

Three watchdog disables.

• nosoftlockup — disable the soft-lockup detector that wakes a hrtimer on every core
• nowatchdog — disable the regular watchdog (same)
• nmi_watchdog=0 — disable the hard-lockup NMI watchdog (uses perf counters, takes them away from your profiling)

For dev nodes, leave watchdogs ON. For production HPC, off — you're explicitly trading "would have caught a kernel hang" for "no periodic noise on compute cores".

intel_pstate=disable / processor.max_cstate=1 / intel_idle.max_cstate=0

Disable CPU frequency scaling. Compute cores stay at max P-state (full clock) and shallowest C-state (no deep sleep entry/exit latency).

C-state transitions can take 100 µs+ — long enough to ruin an NCCL latency budget. Force them off.

You can confirm:

# After boot
cat /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor
# performance              <-- not powersave or schedutil

cat /sys/devices/system/cpu/cpu8/cpuidle/state*/disable
# 1 1 1 1 1                 <-- all idle states disabled (or only state0=POLL active)

Note: this consumes ~30 W more idle power per node. Worth it for HPC; not for a generic cloud workload.

skew_tick=1

Stagger the timer ticks across cores so they don't all fire at the same moment (the "tick storm" effect on a many-core box). Mitigates microarchitectural jitter from synchronized cache-line bounces in the timer subsystem.

audit=0

Disable kernel audit subsystem. Useful for security postures, useless for compute, and adds non-zero overhead per syscall.

Stopping irqbalance and pinning IRQs by hand

After boot, irqbalance is the kernel-userspace daemon that periodically reshuffles IRQ targets across cores. It will undo your irqaffinity cmdline. Stop it:

sudo systemctl disable --now irqbalance
sudo systemctl mask irqbalance      # ensure no-one re-enables it

Now pin each IRQ explicitly. The pattern:

# List all IRQs and their current affinity
for irq in /proc/irq/[0-9]*; do
  num=$(basename $irq)
  name=$(cat $irq/../irq/$num/actions 2>/dev/null || tr -d '\n' < $irq/spurious 2>/dev/null)
  aff=$(cat $irq/smp_affinity_list)
  echo "irq=$num aff=$aff $(cat /proc/irq/$num/.. 2>/dev/null)"
done

A practical loop to pin everything to housekeeping cores 0-7:

HOUSEKEEPING="0-7,96-103"

for irq in /proc/irq/[0-9]*; do
  num=$(basename $irq)
  # Skip per-CPU IRQs (their affinity is fixed)
  [ -f $irq/smp_affinity_list ] || continue
  echo $HOUSEKEEPING | sudo tee $irq/smp_affinity_list >/dev/null 2>&1 || true
done

Some IRQs (per-CPU IPIs, NMIs, CPU-local timer) can't be migrated — they live on every core inherently. Those are localtimer, RES, CAL, TLB, etc. in /proc/interrupts. They're tiny — sub-microsecond — and you can't move them.

For high-rate IRQs that can be moved (network NICs especially):

# Example: ConnectX-7 mlx5_comp@0 → core 0 only
for irq in $(grep mlx5_comp /proc/interrupts | awk -F: '{print $1}' | tr -d ' '); do
  echo 0 | sudo tee /proc/irq/$irq/smp_affinity_list
done

# Per-rxq pinning across multiple housekeeping cores
i=0
for irq in $(grep mlx5_comp@0 /proc/interrupts | awk -F: '{print $1}' | tr -d ' '); do
  cpu=$((i % 8))
  echo $cpu | sudo tee /proc/irq/$irq/smp_affinity_list
  i=$((i+1))
done

The result: NIC RX queue interrupts spread across cores 0-7, none touching the compute cores 8-95.

Verifying isolation actually worked

After cmdline + irqbalance stop + IRQ pinning, you need to confirm. Three tests.

Test 1: nohz_full cores are actually tickless

# On a quiescent node, look at how often each core's tick fires
cat /proc/timer_list | head -50

# Or simpler — count timer interrupts before and after a sleep
cat /proc/interrupts | head -1
cat /proc/interrupts | grep -E '^(LOC|TIMER)' | head
sleep 30
cat /proc/interrupts | grep -E '^(LOC|TIMER)' | head
# Subtract: LOC[N] count growth on isolated cores should be ~30-60 (1-2 ticks/sec, housekeeping)
# Compute cores 8-95 should each show ~1 tick total over 30 s (literally tickless)

Or use perf stat per-CPU:

sudo perf stat -C 8 -e irq:irq_handler_entry,irq_vectors:local_timer_entry sleep 10
#   <core 8 over 10 s>
#   1234       irq:irq_handler_entry
#      3       irq_vectors:local_timer_entry

If local_timer_entry is < 10 over 10 seconds, you have nohz_full working. If it's >100, your nohz_full setup didn't take.

Test 2: hwlat detector

The kernel hwlat_detector tracer measures hardware-induced latency (SMI, microcode, etc. — things even the kernel can't see). Run during a maintenance window:

# Enable hwlat on tracing
sudo trace-cmd start -p hwlat
sleep 60
sudo trace-cmd extract
sudo trace-cmd report | head -30
# task-pid CPU# TIMESTAMP FUNCTION
# hwlatd-1234 [000] 1234.567890: latency: 5 us, ...

If you see latencies > 10 µs from hwlat, the underlying hardware (BIOS, microcode, SMI handlers) is the cause — kernel tuning won't fix it. Check BIOS: disable C-states more aggressively, disable SMI sources (USB legacy emulation, etc.).

Test 3: cyclictest (real-time latency)

sudo apt install rt-tests
sudo cyclictest -t 8 -p 99 -i 1000 -l 100000 -m -a 8-15
# T: 0 ( 1234) P:99 I:1000 C: 100000 Min:      2 Act:    3 Avg:    3 Max:      45
# T: 1 ( 1235) P:99 I:1000 C: 100000 Min:      2 Act:    3 Avg:    3 Max:      28
# ...

Max is the worst-case latency in microseconds for a single thread on each isolated core. On a properly isolated node:

• Max < 50 µs typical, often < 20 µs
• Without isolation: Max is hundreds to thousands of µs

A good number to remember: under 50 µs max latency on isolated cores is a healthy compute node.

Test 4: per-core context-switch counts

# Look for context switches per core
mpstat -P ALL 1 5 > /tmp/mpstat.log

# Or via /proc
for cpu in $(seq 0 191); do
  cs=$(awk -v cpu=$cpu '/^cpu/ && $1==("cpu"cpu) {print $14}' /proc/schedstat 2>/dev/null \
       || echo 0)
  echo "cpu$cpu cs=$cs"
done | sort -k2 -t= -n | tail -10

On a quiet compute node with nothing running, isolated cores should be at ~0 context switches per second. Housekeeping cores (0-7) will have hundreds — that's expected.

tuned-adm for the tuned-using

If you're on RHEL/Rocky and don't want to manage GRUB by hand, tuned has profiles:

sudo tuned-adm list
# Available profiles:
# - balanced
# - desktop
# - hpc-compute       <-- this one
# - latency-performance
# - network-latency
# - throughput-performance

sudo tuned-adm profile hpc-compute
sudo tuned-adm active
# Current active profile: hpc-compute

The hpc-compute profile sets:

• governor=performance, all C-states off
• transparent_hugepage=never
• Sysctl bumps (rmem/wmem, dirty ratios)
• isolcpus from /etc/tuned/hpc-compute-variables.conf

It does NOT set nohz_full or rcu_nocbs — for those you still need GRUB. Read /usr/lib/tuned/hpc-compute/tuned.conf to see exactly what it does.

Cluster-wide management: tuned-adm supports remote profiles, so you can centralize.

CPU isolation via cpuset (modern alternative)

Instead of isolcpus= cmdline, you can do it with cpuset cgroup v2:

# Create an isolated partition for compute
sudo mkdir /sys/fs/cgroup/compute
echo "+cpuset +cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo "8-95,104-191" | sudo tee /sys/fs/cgroup/compute/cpuset.cpus
echo "0-1" | sudo tee /sys/fs/cgroup/compute/cpuset.mems
echo "isolated" | sudo tee /sys/fs/cgroup/compute/cpuset.cpus.partition
cat /sys/fs/cgroup/compute/cpuset.cpus.partition
# isolated                   <-- success

Now anything moved into /sys/fs/cgroup/compute/ runs on isolated cores. The "system" cgroup gets the leftover cores 0-7 + 96-103.

Pros over isolcpus=:

• No reboot to change isolation
• Per-job/per-pod isolation possible
• Plays nicely with kubelet's CPU manager

Cons:

• Doesn't replace nohz_full and rcu_nocbs — still need those on cmdline
• Requires kernel 5.4+ and cgroup v2

Watching for stragglers

Once isolation is set, the question becomes: did anything sneak onto your compute cores?

# Per-core process count
for cpu in $(seq 8 95); do
  count=$(ps -eL -o psr= | awk -v c=$cpu '$1==c' | wc -l)
  echo "cpu$cpu: $count threads"
done | sort -k2 -n -t: | tail
# cpu60: 0 threads
# cpu61: 0 threads
# cpu44: 1 threads     <-- something is here!

Find the offender:

ps -eL -o psr,pid,comm | awk '$1==44'
# 44 12345  irq/47-mlx5_comp

That's an IRQ thread that didn't get re-pinned. Update your IRQ pinning logic to include it.

Common stragglers:

• Kernel kworker threads with persistent affinity (rare; usually they migrate to least-loaded)
• SystemD services without proper CPUAffinity= config
• Monitoring daemons (node-exporter, dcgm-exporter)
• ksoftirqd (per-CPU, can't move)

For systemd units, set affinity:

# /etc/systemd/system/some-service.service.d/override.conf
[Service]
CPUAffinity=0-7,96-103

Common failure modes

isolcpus set but compute cores still busy

cat /proc/cmdline | grep isolcpus
# isolcpus=managed_irq,domain,8-95,104-191      <-- correct

# But:
ps -o psr,pid,comm -p 1 -P                      <-- check init's affinity

isolcpus removes cores from the scheduler default mask, but doesn't apply to processes already running by the time the param is parsed. init keeps its full mask, and so does anything it forks with taskset 0xff.... Force housekeeping affinity on init:

sudo taskset -p -c 0-7,96-103 1
# pid 1's current affinity list: 0-191
# pid 1's new affinity list: 0-7,96-103

Confirm in /proc/1/status: Cpus_allowed_list: 0-7,96-103.

NCCL still showing CPU-side jitter

# Check that compute cores are nohz_full
cat /sys/devices/system/cpu/nohz_full
# 8-95,104-191                                   <-- good

# But check tick rate on a "busy" core during workload
sudo perf stat -C 50 -e local_timer_entry sleep 10
# 12345  local_timer_entry                       <-- TOO HIGH

# If the workload thread on cpu50 has more than one runnable task on
# the core, nohz_full doesn't apply. Confirm:
ps -eLo psr,pid,comm,wchan | awk '$1==50'
# 50  12345 python  futex_wait
# 50  12346 nccl_thread futex_wait               <-- two threads, tick stays

nohz_full only suppresses ticks when exactly one task is runnable on the CPU. Two competing threads on the same core = tick stays on. The fix is more cores per rank (so each thread gets its own core), not more isolation.

Symptom: services fail to start because they want a core that's isolated

journalctl -u some-service
# Failed to set affinity: Operation not permitted

A service has a CPUAffinity=8-15 directive but core 8 is in isolcpus. Either change the service's affinity or remove that core from isolation.

Symptom: kernel says "tickless cpu without nohz_full"

dmesg | grep -i nohz
# NO_HZ: Clearing CPUs: 8-95,104-191
# WARNING: tickless CPUs without nohz_full=cpu...

A core was isolated via isolcpus but not given nohz_full. That works but you're missing the latency win. Update GRUB to add the core to nohz_full=.

cpuset.cpus.partition won't go to isolated

echo "isolated" > /sys/fs/cgroup/compute/cpuset.cpus.partition
# bash: echo: write error: Invalid argument

cat /sys/fs/cgroup/compute/cpuset.cpus.partition
# member invalid (Cpus parent overlap)

Cause: the parent (root cgroup) still has those cores in its cpuset.cpus.effective. To make a sub-cpuset isolated, the parent must release them via cpuset.cpus.exclusive:

echo "8-95,104-191" > /sys/fs/cgroup/cpuset.cpus.exclusive

Or use the isolated partition flag at a lower nest. The kernel docs Documentation/admin-guide/cgroup-v2.rst "CPU Allocations" section is required reading for this.

Cyclictest shows huge Max on one specific core

sudo cyclictest -t 1 -p 99 -i 1000 -l 100000 -m -a 50
# T: 0 ( 1234) P:99 I:1000 C: 100000 Min: 3 Act: 4 Avg: 4 Max: 4500

Max=4500 µs on a supposedly isolated core. Run again with -h 100:

sudo cyclictest -t 1 -p 99 -i 1000 -l 100000 -m -a 50 -h 100

The histogram pinpoints whether it's one big spike (SMI? thermal event?) or a periodic pattern. SMI is invisible to OS — dmesg won't show it. Look for it in BIOS settings: disable USB legacy emulation, disable SMM PCH sleep states.

Symptom: irqbalance keeps re-enabling itself

systemctl disable only stops it from auto-starting; if a config-management agent runs, it might re-enable. systemctl mask is the firmer move — symlinks the unit to /dev/null so enable will fail. Combine with a salt/ansible state that explicitly enforces masked.

See also

• GRUB cmdline — the boot params that drive isolation
• NUMA — pair isolation with NUMA-correct pinning
• cgroups — modern cpuset-based isolation
• Sysctl tuning — kernel.sched_* interacting with isolation
• Kernel troubleshooting playbook — system-hang and stall diagnostics

External:

• Documentation/admin-guide/kernel-parameters.txt — full reference for cmdline params
• Documentation/timers/no_hz.rst (Linux kernel)
• Documentation/RCU/Design/Requirements/Requirements.rst — for the curious about rcu_nocbs
• cyclictest(8) — man cyclictest
• LWN: "Per-CPU runqueue counter" / OS noise series (lwn.net)