cgroups v1 vs v2 for GPU/HPC nodes

cgroups is the kernel's resource accounting layer. Every container, every Slurm job step, every systemd service runs inside a cgroup. On a GPU node, cgroups decide which CPUs the workload can touch, how much memory it can pin, and (with the right hooks) which GPU character devices it can open.

For years there were two cgroup APIs side-by-side. v1 had a separate hierarchy per controller (one for cpu, one for memory, etc.). v2 unified them into a single tree. Most production stacks have crossed over to v2 by now — but the GPU stack, especially older container-toolkit versions and a few Slurm releases, still drag behind.

This page is the operator's view: what to check, when v1/v2 mismatches break things, and how to read the per-pod state.

v1 vs v2 at a glance

The big practical change for GPU folks is that v2 has no devices controller. Older nvidia-container-runtime versions wrote to devices.allow to grant access to /dev/nvidia*. Under v2 that file doesn't exist; the runtime has to install an eBPF program on the cgroup instead. Versions before container-toolkit 1.10 don't know how to do this.

Confirm which mode you're on:

mount | grep cgroup
# v2 only:
# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

# v1 (legacy, mostly gone):
# cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
# cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (...)
# (about 12 lines, one per controller)

# Hybrid mode (Ubuntu 20.04 default before 22.04):
# cgroup2 on /sys/fs/cgroup/unified type cgroup2 (...)   <- v2 mounted alongside
# cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (...)
# (some controllers v1, some v2)

# Or read the kernel directly
stat -fc %T /sys/fs/cgroup/
# cgroup2fs       <-- pure v2
# tmpfs           <-- v1 (the controllers are submounts)

To toggle between v1 and v2, set the kernel cmdline at boot:

# Force v2
systemd.unified_cgroup_hierarchy=1

# Force v1
systemd.unified_cgroup_hierarchy=0

# Hybrid (rare, only if you really need it)
systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=0

Modern distros (Ubuntu 22.04+, RHEL 9+, SLES 15 SP4+) default to v2. Stick with the default unless something forces you off.

The unified hierarchy

Under v2, every cgroup is at /sys/fs/cgroup/<path>/. systemd sets up slices and scopes:

ls /sys/fs/cgroup/
# cgroup.controllers       cgroup.subtree_control  init.scope/   system.slice/
# cgroup.max.depth         cgroup.threads          machine.slice/  user.slice/

cat /sys/fs/cgroup/cgroup.controllers
# cpuset cpu io memory hugetlb pids rdma misc       <-- what's available

cat /sys/fs/cgroup/cgroup.subtree_control
# cpuset cpu io memory pids                          <-- what's enabled for child cgroups

# Look at a specific service's cgroup
cat /sys/fs/cgroup/system.slice/kubelet.service/cgroup.procs
# 12345
# 12346
# ... PIDs of all kubelet threads/children

# What CPUs is it allowed to use?
cat /sys/fs/cgroup/system.slice/kubelet.service/cpuset.cpus.effective
# 0-191                                              <-- all of them, kubelet has no cpuset

The interface files matter for diagnosis. The most common ones:

systemd-cgls / systemd-cgtop

For walking the tree by hand:

# Tree view
systemd-cgls
# Control group /:
# -.slice
# ├─user.slice
# │ └─user-1000.slice
# │   └─session-3.scope
# │     ├─12345 sshd: gustcol [priv]
# │     └─12346 -bash
# ├─init.scope
# │ └─1 /sbin/init
# └─system.slice
#   ├─kubelet.service
#   │ └─12340 /usr/bin/kubelet ...
#   ├─containerd.service
#   │ └─12350 /usr/bin/containerd
#   └─slurmd.service
#     └─12360 /usr/sbin/slurmd

# Live top-style view
systemd-cgtop
# Control Group                    Tasks   %CPU   Memory  Input/s Output/s
# /                                  1234  189.5    420.3G       -        -
# kubepods.slice                      890   88.2    256.7G       -        -
# kubepods.slice/kubepods-burstabl…   234   45.1     12.5G       -        -
# system.slice                         50    1.8     14.2G       -        -
# system.slice/kubelet.service          5    0.5      2.1G       -        -

systemd-cgtop is the right starting point when "the node is loaded but I don't know who's eating CPU/memory". It groups by cgroup, not by process — much easier to reason about in a Kubernetes / Slurm world.

CPU controllers — cpu, cpuset, cpuacct

The two CPU-related controllers do different things:

• cpu — bandwidth (CFS): you get N% of CPU time, throttled if you exceed.
• cpuset — placement: you can only run on these specific CPUs (and these NUMA nodes for memory).

For HPC, you almost always want cpuset (hard core pinning) and rarely want cpu quotas (they cause throttling that ruins NCCL). Slurm and Kubernetes (with TopologyManager) drive cpuset automatically.

# What cpuset is a process bound to?
cat /proc/<pid>/cgroup
# 0::/kubepods.slice/kubepods-burstable.slice/...kubepods-burstable-podabcd.../crio-abc123.scope

cat /sys/fs/cgroup/kubepods.slice/.../crio-abc123.scope/cpuset.cpus.effective
# 16-31                                              <-- pinned to these 16 cores

# Is the kernel actually enforcing it?
taskset -cp <pid>
# pid 12345's current affinity list: 16-31           <-- matches cpuset, good

If taskset reports a wider mask than cpuset.cpus.effective, the cpuset hasn't been propagated to the kernel scheduler — usually a kubelet bug or a missed cpuset.cpus.partition=root enablement on a newer kernel.

cpu.max (CFS bandwidth) — when it bites GPU jobs

# A pod with a CPU limit of 8 cores has:
cat /sys/fs/cgroup/.../cpu.max
# 800000 100000                     <-- 8 CPUs * 100ms quota / 100ms period

# Throttling counters
cat /sys/fs/cgroup/.../cpu.stat
# usage_usec 145623789
# user_usec  140123456
# system_usec 5500333
# nr_periods  234567
# nr_throttled  12345              <-- non-zero = the workload is being CPU-throttled
# throttled_usec 9876543

nr_throttled rising while a GPU job runs is poison — every CFS period the kernel can park the user-space CUDA threads, freezing the GPU pipeline mid-collective. Either remove CPU limits on GPU pods (set requests only, no limits), or use static CPU manager which gives whole pinned cores instead of bandwidth.

Memory controller

cat /sys/fs/cgroup/.../memory.current
# 137438953472                      <-- bytes in use, ~128 GiB

cat /sys/fs/cgroup/.../memory.max
# 274877906944                      <-- limit ~256 GiB

cat /sys/fs/cgroup/.../memory.events
# low 0
# high 1234              <-- soft throttle hits
# max  56                <-- hard-limit hits (about to OOM)
# oom  1                 <-- the cgroup OOM-killer fired
# oom_kill 1             <-- and killed something

cat /sys/fs/cgroup/.../memory.stat
# anon 130000000000
# file 5000000000
# kernel_stack 12345678
# slab 234567890
# sock 0
# anon_thp 100000000000   <-- THP-backed anonymous memory
# file_mapped 4500000000
# file_dirty 67890000
# file_writeback 12345000
# pgfault 12345678
# pgmajfault 1234
# workingset_refault 56789
# workingset_activate 12345
# ...

memory.events is the "did this cgroup OOM" file. oom_kill rising means the kernel killed something inside this cgroup. dmesg | grep 'memory cgroup' correlates the timestamp:

dmesg -T | grep -i 'oom\|memory cgroup'
# [Mon May 5 15:23:11 2026] Memory cgroup out of memory: Killed process 12345 (python) ...
# [Mon May 5 15:23:11 2026] memory: usage 134217728kB, limit 134217728kB, failcnt 234

failcnt ticking with no actual OOM means the workload is right at the wall and being slowed by reclaim. Bump the limit.

Devices controller — v1 vs v2 difference

Under v1, the kernel had a devices cgroup controller. Files were:

# v1 only — does NOT exist on v2
cat /sys/fs/cgroup/devices/<scope>/devices.list
# c 195:* rwm                       <-- char major 195 = nvidia, allowed read/write/mknod
# c 235:* rwm                       <-- nvidia-uvm
# c 1:5 rwm

Under v2, the controller was removed. Device access is now enforced by an eBPF program attached to the cgroup. You can't just read a file to see what's allowed; you have to dump the BPF:

# List BPF programs and their cgroup attachment
sudo bpftool prog list
# 234: cgroup_device  tag abc123def4567890 ...

sudo bpftool cgroup tree
# CgroupPath
# ID       AttachType      AttachFlags     Name
# /sys/fs/cgroup/.../crio-abc.scope
#     234  cgroup_device   multi

The relevant code is the runtime's. runc, crun, and nvidia-container-runtime all install BPF device filters under v2. Older versions (pre-1.10 nvidia-container-toolkit) only knew the v1 devices.allow interface and silently dropped device access on v2 — manifesting as "the container starts, but nvidia-smi returns no devices were found".

The fix path:

# Check toolkit version
nvidia-container-toolkit --version
# NVIDIA Container Toolkit CLI version 1.14.3

# Sub 1.10 needs upgrade. Or you must boot with v1:
# Add to GRUB: systemd.unified_cgroup_hierarchy=0

RDMA controller

The rdma controller limits per-cgroup RDMA resource usage. Useful in multi-tenant boxes (rare on dedicated HPC).

ls /sys/fs/cgroup/.../rdma.*
# rdma.current  rdma.max

cat /sys/fs/cgroup/.../rdma.current
# mlx5_0 hca_handle=12 hca_object=234
# mlx5_1 hca_handle=8  hca_object=187

cat /sys/fs/cgroup/.../rdma.max
# mlx5_0 hca_handle=max hca_object=max         <-- unlimited

hca_handle = open verbs contexts (one per process per device, roughly). hca_object = QPs, CQs, MRs combined. If a job hits a non-max limit, you'll see ENOMEM from ibv_create_qp() — but the system has memory. It's the controller saying no.

Slurm + cgroups

Slurm's cgroup.conf controls how it carves cgroups for each job step.

# /etc/slurm/cgroup.conf
CgroupAutomount=yes
CgroupMountpoint=/sys/fs/cgroup
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
ConstrainDevices=yes
TaskAffinity=no                          # let cgroup do binding, not Slurm
AllowedRAMSpace=100
AllowedSwapSpace=0
MaxRAMPercent=100

# /etc/slurm/slurm.conf — relevant excerpts
ProctrackType=proctrack/cgroup           # tracks PIDs via cgroup, can't escape
TaskPlugin=task/cgroup,task/affinity     # bind cores via cgroup, fall back to taskset
JobAcctGatherType=jobacct_gather/cgroup  # accounting from cgroup counters

The combo task/cgroup + JobAcctGatherType=jobacct_gather/cgroup reads cpu.stat and memory.stat for accounting — accurate even when a job forks workers. Without proctrack/cgroup, a daemonized child can survive scancel.

Verify a running step is correctly in its cgroup:

# As root on the compute node, with job 1234 running
scontrol show job 1234 | grep BatchHost
# BatchHost=node-00

# On node-00:
ls /sys/fs/cgroup/system.slice/slurmstepd.scope/job_1234/
# step_0/  step_batch/  cgroup.procs  cpuset.cpus.effective  memory.max ...

cat /sys/fs/cgroup/.../job_1234/step_0/cgroup.procs
# 23456 23457 23458 ...                <-- the user's processes

cat /sys/fs/cgroup/.../job_1234/step_0/cpuset.cpus.effective
# 0-23                                  <-- the requested CPU binding

Container runtimes — containerd / runc / nvidia-container-runtime

The flow for a Kubernetes GPU pod:

1. kubelet decides cpuset/memory based on TopologyManager
2. kubelet calls containerd via CRI
3. containerd writes the OCI spec
4. containerd invokes runc (or crun)
5. runc creates the cgroup, applies cpuset/memory/etc.
6. runc invokes the nvidia-container-runtime as a runtime.NVIDIAOCIPrestart hook
7. nvidia-container-runtime modifies the OCI spec to add device nodes (/dev/nvidia*) and (under v2) installs the BPF filter
8. runc exec's the container's argv[0]

The OCI spec for the container is at /run/containerd/io.containerd.runtime.v2.task/k8s.io/<containerid>/config.json:

sudo cat /run/containerd/io.containerd.runtime.v2.task/k8s.io/<id>/config.json | jq '.linux.resources'
# {
#   "devices": [
#     { "allow": false, "access": "rwm" },
#     { "allow": true, "type": "c", "major": 195, "minor": -1, "access": "rwm" },
#     { "allow": true, "type": "c", "major": 235, "minor": -1, "access": "rwm" }
#   ],
#   "memory": { "limit": 274877906944 },
#   "cpu": { "shares": 8000, "cpus": "16-31", "mems": "1" }
# }

The devices array is the input to whatever the runtime writes — under v1 to devices.allow, under v2 to the eBPF program. If nvidia-smi inside the container says "no devices", first check:

# Inside the container
ls -la /dev/nvidia*
# crw-rw-rw- 1 root root 195,   0 May  5 14:32 /dev/nvidia0
# crw-rw-rw- 1 root root 195, 254 May  5 14:32 /dev/nvidia-modeset
# ...

# Are the nodes there at all? If not, the runtime didn't add them — toolkit issue.
# Are they there but inaccessible? Then it's the BPF filter blocking them.

To verify the BPF on a v2 host:

sudo bpftool cgroup show /sys/fs/cgroup/.../crio-<containerid>.scope/
# ID     AttachType  AttachFlags  Name
# 234    cgroup_device multi      nvidia_dev_filter

If no cgroup_device is attached, the runtime didn't install one and the kernel's default policy applies. Default v2 policy is "allow everything", so the device is accessible — but then a different layer (typically AppArmor, SELinux, or --security-opt seccomp) is blocking it.

Common failure modes

Pod can't see GPU — "no devices were found"

Walk the layers:

# 1. Is the device node in the container?
kubectl exec -it <pod> -- ls -l /dev/nvidia*
# If missing: nvidia-container-toolkit didn't run. Check container runtime
# config — /etc/containerd/config.toml should have the nvidia runtime.

# 2. Is it accessible?
kubectl exec -it <pod> -- cat /proc/self/status | grep CapEff
# CapEff: 00000000a80425fb
# If too restrictive (no CAP_SYS_ADMIN where required), runtime locked it down.

# 3. Is the host-side cgroup filtering it out?
kubectl exec -it <pod> -- cat /proc/self/cgroup
# 0::/kubepods.slice/.../crio-abc.scope
# Then on the host:
sudo bpftool cgroup show /sys/fs/cgroup/kubepods.slice/.../crio-abc.scope/
# Look for cgroup_device program.

OOMKilled with cgroup memory limit

# kubectl describe shows OOMKilled
kubectl describe pod <pod> | grep -A2 'State:'
# State:        Terminated
#   Reason:     OOMKilled
#   Exit Code:  137

# Confirm it was the cgroup OOM, not host OOM
sudo dmesg -T | grep -i 'memory cgroup out of memory'
# [time] Memory cgroup out of memory: Killed process 12345 (python) ...

# How much was it using vs limit?
sudo cat /sys/fs/cgroup/.../memory.events
# oom_kill 1
sudo cat /sys/fs/cgroup/.../memory.peak     # kernel >= 5.19
# 137438953472                              <-- peak before the kill

If memory.peak == memory.max, the workload genuinely exceeded the limit — bump the limit or fix the leak. If memory.peak < memory.max and OOM still fired, the killer was triggered by memory.high pressure causing the kernel's OOM scoring to favor this cgroup. Less common, usually means another cgroup on the host is also under heavy pressure.

Slurm proctrack failure — slurmd can't kill stragglers

Symptom: scancel <jobid> returns immediately, but ps still shows the user's processes. Or: a new job lands on the node and finds residual processes from the previous one.

# Check proctrack config
grep ProctrackType /etc/slurm/slurm.conf
# ProctrackType=proctrack/cgroup            <-- must be cgroup, not pgid or linuxproc

# Check that the slurmstepd cgroup actually exists and contains the procs
ls /sys/fs/cgroup/system.slice/slurmstepd.scope/

# If empty when a job runs, slurmd lost the cgroup mount or the version
# of Slurm doesn't support v2 (Slurm < 22.05)

Slurm 22.05+ supports cgroups v2. Earlier versions (< 21.08) only do v1. If you've upgraded the OS to v2 but kept Slurm < 22.05, switch to ProctrackType=proctrack/linuxproc (less reliable but works) or upgrade Slurm.

cgroup.procs: Resource busy when moving processes

# Trying to move a PID into a cgroup
echo 12345 > /sys/fs/cgroup/foo/cgroup.procs
# bash: echo: write error: Resource busy

Causes:

• The process is in a cgroup with cgroup.type=threaded and you're trying to put it in a non-threaded cgroup, or vice versa.
• The destination cgroup has cgroup.subtree_control not enabled for the parent, so it's a "domain invalid" state.
• A controller is enabled in the source's parent but not in the destination's parent.

Inspect:

cat /sys/fs/cgroup/foo/cgroup.type
# domain invalid                <-- the cgroup is in a bad state, can't accept procs

# Fix: enable controllers in parent's subtree_control
echo "+cpu +memory +cpuset" > /sys/fs/cgroup/cgroup.subtree_control

Throttled GPU job — cpu.stat nr_throttled rising

# Pod has CPU limit, NCCL allreduce stalls
kubectl exec -it <pod> -- cat /sys/fs/cgroup/cpu.stat
# nr_throttled 1234
# throttled_usec 12345678

# Confirm correlation with NCCL stall:
# Look for nr_throttled growing during the time the bandwidth dropped

Remove the CPU limit on the pod (set requests but not limits), or use cpuManagerPolicy: static so the workload gets exclusive cores instead of bandwidth quotas.

pids.max exhausted in a fork-heavy job

cat /sys/fs/cgroup/.../pids.current
# 4096
cat /sys/fs/cgroup/.../pids.max
# 4096                                <-- at the wall

# Symptom: fork(): Resource temporarily unavailable

In Kubernetes, the limit comes from --pod-max-pids (default 4096). Bump in kubelet config:

podPidsLimit: 65536

PyTorch DataLoaders with num_workers >= 16 plus a forking model (DDP) can hit 4096 PIDs surprisingly fast.

Container can see all of host's memory in /proc/meminfo

This isn't a bug, it's how procfs works — it always shows the host. Use cgroup files instead:

# Inside the container
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max

Modern frameworks (PyTorch >= 2, JAX) read cgroup memory limits when sizing internal buffers. Older code reads /proc/meminfo and over-allocates.

Drop-in: cgroup_enable=memory swapaccount=1 on cmdline

On older kernels (pre-5.4) and some distro configs, the memory controller isn't enabled by default. Symptom: memory.max doesn't exist or cgroup.controllers doesn't list memory. Fix:

# /etc/default/grub
GRUB_CMDLINE_LINUX="... cgroup_enable=memory swapaccount=1 ..."

swapaccount=1 is needed if you want memory.swap.* accounting (rare on HPC where swap is off, but still good practice).

See also

• Sysctl tuning — vm.* works alongside cgroup memory limits
• NUMA — TopologyManager + cpuset deep dive
• Jitter isolation — cpuset for HPC isolation
• Container toolkit — the GPU side of the runtime
• GPU Operator — how K8s wires it all together

External:

• Documentation/admin-guide/cgroup-v2.rst (Linux kernel)
• man 7 cgroups
• systemd.resource-control(5)
• Slurm cgroup.conf(5)