NUMA on GPU nodes — pinning, distances, and the perf cliff

How NUMA topology shapes GPU node performance, how to read numactl/numastat, GPU-to-NUMA pinning, and the Slurm flags that actually work.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

A modern 8-GPU server is not a single machine — it is two NUMA domains glued together by UPI/Infinity Fabric, with four GPUs and four NICs hanging off each socket. If your training process lands on the wrong NUMA node, every memory access traverses the inter-socket link, NCCL falls off a cliff, and you spend three days blaming the network.

This page is about reading the topology, pinning correctly, and the failure modes that show up when you don't.

NUMA basics for the operator

A NUMA node is the unit of memory locality. On x86 it usually corresponds to one CPU socket plus its directly-attached DRAM and PCIe lanes. Memory access from CPU0 to its local DRAM is fast (~80 ns); access to the other socket's DRAM goes over UPI (Intel) or Infinity Fabric (AMD) and costs roughly 1.5-2× as much in latency and a chunk of bandwidth.

Read the topology:

lscpu | grep -i numa
# NUMA node(s):                       2
# NUMA node0 CPU(s):                  0-47,96-143
# NUMA node1 CPU(s):                  48-95,144-191

numactl -H
# available: 2 nodes (0-1)
# node 0 cpus: 0 1 2 ... 47 96 97 ... 143
# node 0 size: 515934 MB
# node 0 free: 511201 MB
# node 1 cpus: 48 49 ... 95 144 ... 191
# node 1 size: 516071 MB
# node 1 free: 510844 MB
# node distances:
# node   0   1
#   0:  10  21
#   1:  21  10

The distance matrix is the only part most operators read. 10 is "local" (a relative unit, not nanoseconds); 21 means roughly 2.1× the local cost. On AMD EPYC with chiplets, you can see four NUMA nodes per socket and distances like 10/12/32, where 12 is on-socket-but-different-CCD and 32 is cross-socket.

How memory ends up on the wrong node

The Linux memory policy default is first-touch: a page is allocated on whichever NUMA node ran the thread that first wrote to it. If your program forks and the child thread migrates to the other socket before its first malloc, every subsequent access becomes remote.

numastat -p <pid> shows where a process's pages actually live:

numastat -p 12345
# Per-node process memory usage (in MBs)
# PID                Node 0          Node 1           Total
# -----------  --------------  --------------  --------------
# 12345              4823.51         12047.83        16871.34

If your process is logically pinned to node 0 but Node 1 has more pages, you have a memory-locality bug — most likely a worker thread spawned without numa_run_on_node() being honored.

/proc/<pid>/numa_maps is the line-level view:

cat /proc/12345/numa_maps | head
# 7f9c00000000 default file=/lib/x86_64-linux-gnu/libc.so.6 mapped=512 N0=512 kernelpagesize_kB=4
# 7f9c40000000 default heap anon=131072 dirty=131072 N0=65536 N1=65536 kernelpagesize_kB=4

N0=65536 N1=65536 on the heap means the process is using both nodes for the same anonymous mapping — either an interleave policy or a thread-locality bug.

Which GPU is on which NUMA node

Two sources, both reliable, both you should learn to read.

nvidia-smi topo -m
#         GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  CPU Affinity  NUMA Affinity
# GPU0    X     NV18  NV18  NV18  NV18  NV18  NV18  NV18  0-47,96-143    0
# GPU1    NV18  X     NV18  NV18  NV18  NV18  NV18  NV18  0-47,96-143    0
# GPU2    NV18  NV18  X     NV18  NV18  NV18  NV18  NV18  0-47,96-143    0
# GPU3    NV18  NV18  NV18  X     NV18  NV18  NV18  NV18  0-47,96-143    0
# GPU4    NV18  NV18  NV18  NV18  X     NV18  NV18  NV18  48-95,144-191  1
# GPU5    NV18  NV18  NV18  NV18  NV18  X     NV18  NV18  48-95,144-191  1
# GPU6    NV18  NV18  NV18  NV18  NV18  NV18  X     NV18  48-95,144-191  1
# GPU7    NV18  NV18  NV18  NV18  NV18  NV18  NV18  X     48-95,144-191  1

The right two columns are what you want. GPU0-3 belong to NUMA node 0; GPU4-7 to NUMA node 1. The CPU affinity column tells you which logical CPUs the kernel thinks are local to that GPU.

The same info, lower-level, via sysfs:

for i in 0 1 2 3 4 5 6 7; do
  bdf=$(nvidia-smi -i $i --query-gpu=pci.bus_id --format=csv,noheader | tr 'A-F' 'a-f')
  bdf=${bdf#00000000:}    # strip domain prefix
  echo "GPU$i $bdf NUMA=$(cat /sys/bus/pci/devices/0000:${bdf,,}/numa_node)"
done
# GPU0 17:00.0 NUMA=0
# GPU1 31:00.0 NUMA=0
# GPU2 4b:00.0 NUMA=0
# GPU3 65:00.0 NUMA=0
# GPU4 97:00.0 NUMA=1
# ...

If numa_node reads -1, the BIOS didn't expose NUMA topology to the OS — usually a "NUMA per socket = disabled" / "Cluster-on-Die = off" BIOS setting. Fix in BIOS, not the OS.

NICs matter too

The NIC that talks RDMA on behalf of GPU0 must also be on NUMA node 0, otherwise GPUDirect RDMA pays a cross-socket hop on the NIC's PCIe lanes — which nvidia-smi topo -m will show as SYS (traverses system fabric) instead of PIX or PXB.

nvidia-smi topo -m | head -1
ibdev2netdev -v
# mlx5_0 port 1 ==> ens14f0np0 (Up)
cat /sys/class/net/ens14f0np0/device/numa_node
# 0

Pair the GPU with the NIC on the same NUMA node. NCCL with NCCL_IB_HCA set to a misaligned NIC drops 30-60% of bandwidth.

Pinning processes the right way

`numactl` (single process)

# Run on node 0, allocate memory only on node 0, use only its CPUs.
numactl --cpunodebind=0 --membind=0 ./my_training_step

# Pin to specific cores within a node:
numactl --physcpubind=0-23 --membind=0 ./my_training_step

--membind is strict: an out-of-memory will OOM rather than spill to the other node. --preferred=0 is softer (try node 0 first, fall back).

`taskset` (cores only, doesn't bind memory)

taskset -c 0-23 ./my_training_step

Use taskset only when you've already taken care of memory affinity in the application. For HPC workloads, prefer numactl.

Inside Python / PyTorch

PyTorch DDP launches one process per GPU. The launcher (torchrun, accelerate launch) does not pin NUMA by default. Wrap each rank in numactl:

# Each rank gets its own NUMA-correct cpuset
numactl --cpunodebind=$((LOCAL_RANK / 4)) --membind=$((LOCAL_RANK / 4)) \
  python -u train.py

Or use the helper from NCCL:

import os
os.environ.setdefault("NCCL_IGNORE_CPU_AFFINITY", "0")  # Let NCCL set affinity

Slurm + NUMA

Slurm has the bookkeeping but you have to ask for it.

# Bind each task to its own NUMA-local cores
srun --cpu-bind=cores --gpus-per-task=1 --gpu-bind=closest ./train

# Bind explicitly to a NUMA node
srun --cpu-bind=ldoms --gpu-bind=map_gpu:0,1,2,3 ./train

# Map cores to ranks 1:1 with verbose verification
srun --cpu-bind=verbose,cores ./train
# task 0 cpu_bind=MASK - host123, mask 0x000000ff
# task 1 cpu_bind=MASK - host123, mask 0x0000ff00
# ...

The relevant flags:

Flag	What it does
`--cpu-bind=cores`	One Slurm task per core; OS scheduler can't migrate across cores
`--cpu-bind=ldoms`	One task per NUMA node ("locality domain")
`--cpu-bind=verbose,...`	Print final binding mask, useful for debugging
`--gpu-bind=closest`	Each task gets the GPU on its NUMA node
`--gpu-bind=map_gpu:0,1,2,3`	Explicit GPU-to-task map
`--mem-bind=local`	Memory allocations stay on the task's NUMA node

For multi-node NCCL training, the recipe that almost always works:

sbatch <<'EOF'
#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=24
#SBATCH --gpu-bind=closest
#SBATCH --cpu-bind=cores
#SBATCH --mem-bind=local

srun --container-image=./train.sqsh ./run_ddp.sh
EOF

The classic NUMA failure mode

A multi-node NCCL training job where bandwidth tests in isolation are healthy but real training collapses. Symptoms:

nccl-tests/all_reduce_perf standalone: 380 GB/s busbw on 8 GPUs, expected
Real training job: 90 GB/s, 4× slowdown
nvidia-smi shows GPU util 30-50% with sawtooth pattern
top shows 100% CPU on a few cores on the opposite socket from the GPU

What happened: the training launcher started worker threads (PyTorch DataLoader workers, dataloader pinning, NCCL communicator threads) without NUMA binding. Those threads got scheduled to whichever core was idle, often on the wrong socket. Their memory allocations followed first-touch and ended up on the wrong NUMA node. Each NCCL collective now has to copy host buffers across UPI before it even hits PCIe.

The fix is the numactl wrap above plus NCCL_SOCKET_IFNAME set to the NIC on the same NUMA node as the GPUs in that worker.

Common debug commands

# Where are this process's pages?
numastat -p <pid>

# How much cross-NUMA traffic is the kernel seeing right now?
numastat
#                          node0           node1
# numa_hit              891234567        891345678
# numa_miss               1234567          1234890     # <-- wanted node X, got Y
# numa_foreign            1234890          1234567     # <-- pages allocated for the other node
# interleave_hit            45678            45123
# local_node            889999999        890099999
# other_node              1234568          1245679

# Per-process NUMA maps with allocation counts
grep -E 'N0|N1' /proc/<pid>/numa_maps | head

# kernel.numa_balancing — usually OFF for HPC, ON helps generic workloads
sysctl kernel.numa_balancing

A non-zero numa_miss rate growing during your job is the smoking gun — work is being scheduled away from where its memory lives.

Detecting cross-NUMA traffic in flight

numastat is the system view; it answers "did the kernel hit the right node?" perf c2c is the hardware-counter view; it answers "are CPU cores stealing cache lines from each other across the interconnect?" Both matter.

`numastat -m` for the whole-system view

numastat -m
#
# Per-node system memory usage (in MBs):
#                          Node 0          Node 1           Total
#                 --------------- --------------- ---------------
# MemTotal              515934.42       516071.50      1032005.92
# MemFree               388234.11       501212.34       889446.45
# MemUsed               127700.31        14859.16       142559.47   <-- skewed
# Active               104982.10        12345.67       117327.77
# Inactive              22318.21         2513.49        24831.70
# Active(anon)          98765.43         9876.54       108641.97
# Inactive(anon)         1234.56          123.45         1358.01
# Active(file)           6216.67         2469.13         8685.80
# AnonPages            100000.00        10000.00       110000.00
# Mapped                  500.00           50.00          550.00
# AnonHugePages         98000.00         9800.00       107800.00   <-- THP heavy on N0
# HugePages_Total           0.00            0.00            0.00
# HugePages_Free            0.00            0.00            0.00

If MemUsed is 100 GB on node 0 and 15 GB on node 1, your workload is using almost only one socket's memory. If AnonHugePages is heavily skewed in the same direction, THP is amplifying the imbalance — every minor faulted-in 2 MiB page came from local. That is fine if your job runs only on node 0; it's bad if you also have threads on node 1 reaching across UPI for those pages.

`perf c2c` for cache-line ping-pong

perf c2c (cache-to-cache) profiles HITM events — when a core takes a modified cache line from another core's L1/L2, possibly across the socket boundary. Painful for HPC because every HITM is a few hundred cycles of stall.

# Record system-wide for 30 seconds
sudo perf c2c record -a sleep 30

# Display the hot lines
sudo perf c2c report --stdio | head -80
# =================================================
#       Trace Event Information
# =================================================
#   Total records                     :    1245634
#   Locked Load/Store Operations      :        342
#   Load Operations                   :     623145
#   Loads - no mapping                :       1234
#   Load Fill Buffer Hit              :     123456
#   Load L1D hit                      :     456789
#   Load L2D hit                      :      45678
#   Load LLC hit                      :       4567
#   Load Local HITM                   :        234   <-- intra-socket HITM
#   Load Remote HITM                  :       1234   <-- ACROSS socket. BAD.
#   ...

A high Load Remote HITM count under a steady-state workload means cores on different sockets are actively fighting over cache lines. The fix is almost always pinning — split the work so each thread group only writes lines that stay in its own socket's LLC.

The report --stdio output then walks line-by-line. The most contended lines look like:

=================================================
        Shared Data Cache Line Table
=================================================
#
#                              Total      Tot  ----- LLC Load Hitm -----  Total
# Index           Cacheline   records  loads          Total  LclHitm  RmtHitm  stores
#     0  0x7fb234567000             45      32             14        2       12       0
#     1  0x7fb234568040             89      67             32        8       24       3

The line at 0x7fb234568040 had 24 remote HITMs in 30 s — that line is being passed across the socket. Annotate to find the source:

sudo perf c2c report --stdio --full-symbols 2>/dev/null | grep -A5 0x7fb234568040

Usually a global counter, mutex, or shared queue head. The fix is per-socket sharding (e.g., per-NUMA work stealing pools).

`perf stat` for the quick check

# Run while load is steady
sudo perf stat -a -e \
  cache-misses,cache-references,LLC-loads,LLC-load-misses,\
  node-loads,node-load-misses,node-stores,node-store-misses \
  sleep 10
#
#  Performance counter stats for 'system wide':
#
#    8,123,456,789      cache-misses
#   45,678,901,234      cache-references
#    2,345,678,901      LLC-loads
#    1,234,567,890      LLC-load-misses               #  52.6% of all LL-cache accesses
#    9,876,543,210      node-loads
#      234,567,890      node-load-misses              #   2.3% of all node-loads
#                                                       (raw remote-DRAM accesses)

node-load-misses / node-loads > 5% under steady state means the kernel is fetching from the remote DRAM more often than it should. Pair with numastat -p on your hot processes to find the culprit.

libnuma vs numactl in code

numactl is a shell wrapper. For programs that fork worker threads after startup, the binding from numactl is inherited but doesn't pin newly-created threads to a specific node — Linux's NUMA policy is per memory area, not per thread. The thread runs wherever the scheduler likes; only the CPU mask is inherited.

For libraries that spawn threads dynamically (PyTorch DataLoader workers, OpenMP teams, NCCL plugins), the only reliable approach is in-code binding:

#include <numa.h>
#include <numaif.h>

if (numa_available() < 0) abort();

// Bind THIS thread to node 0
numa_run_on_node(0);             // CPU affinity
numa_set_membind_compat(numa_parse_nodestring("0"));  // memory affinity

// Bind a region of memory explicitly
void *buf = malloc(SIZE);
numa_tonode_memory(buf, SIZE, 0);

For Python:

import ctypes, os
libnuma = ctypes.CDLL("libnuma.so.1", use_errno=True)
libnuma.numa_available.restype = ctypes.c_int
assert libnuma.numa_available() >= 0

# Bind this thread to node 0
libnuma.numa_run_on_node(ctypes.c_int(0))

In a PyTorch DataLoader, the worker_init_fn is the right hook:

def worker_init_fn(worker_id):
    import os, ctypes
    libnuma = ctypes.CDLL("libnuma.so.1")
    # Map worker_id to a NUMA node: even -> 0, odd -> 1
    node = (worker_id // 4) % 2
    libnuma.numa_run_on_node(ctypes.c_int(node))

DataLoader(..., num_workers=8, worker_init_fn=worker_init_fn)

Without this, the workers spread across all cores; with it, they stay socket-local and the prefetched batches end up in local DRAM, ready for the GPU on that socket to pull via cudaMemcpy.

Slurm + NUMA — the modern flags

Older recipes use --cpu-bind=ldoms. On modern Slurm (>= 22.05) the cleaner flags are:

# Hint the scheduler that the job is memory-bandwidth-bound (will prefer
# spreading across NUMA nodes if it has the choice)
srun --hint=memory_bound ./train

# Hint that it's compute-bound (prefer packing within a NUMA node)
srun --hint=compute_bound ./train

# Explicit memory binding to closest-to-CPU
srun --mem-bind=local --cpu-bind=cores ./train

# Print binding masks for every task — what you want during shakedown
srun --cpu-bind=verbose,cores --mem-bind=verbose,local ./train
# task 0 cpu_bind=MASK - host123, mask 0x000000000000ffff
# task 0 mem_bind=NODE - host123, nid 0
# task 1 cpu_bind=MASK - host123, mask 0x00000000ffff0000
# task 1 mem_bind=NODE - host123, nid 0
# ...

The verification step is non-negotiable. Multiple times per year, a Slurm config change (a new cgroup.conf, a new gres.conf) silently regresses GPU pinning, and the only way to catch it before the user complains is verbose,cores.

For container jobs (srun --container-image=), the binding propagates through enroot/pyxis automatically — the container sees only its assigned cores. Inside the container, lscpu will show only the bound cores.

Kubernetes + NUMA: TopologyManager

K8s does NUMA via the kubelet TopologyManager. Three policies you need to know:

Policy	Behavior
`none`	(default) No alignment. CPU manager and Device manager pick independently. Bad.
`best-effort`	Try to align CPU, memory, and devices to one NUMA node; if it can't, schedule anyway.
`restricted`	Try to align; if it can't, fail the admission of the pod (Pending state).
`single-numa-node`	All resources MUST be on one NUMA node, or the pod fails admission. Production.

Configured per-kubelet:

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
memoryManagerPolicy: Static
topologyManagerPolicy: single-numa-node
topologyManagerScope: container
reservedSystemCPUs: "0-3,96-99"      # for kubelet/system, off the workload

Restart the kubelet (and on RKE2: systemctl restart rke2-agent).

Verify on a running pod:

# Find the container ID
crictl ps -a | grep <pod-name>
crictl inspect <containerid> | jq '.info.runtimeSpec.linux.resources.cpu'
# {
#   "cpus": "0-23",                  <-- bound cpuset
#   ...
# }

# What does the GPU plugin think?
kubectl describe node <node> | grep -A20 'Capacity:'
# nvidia.com/gpu: 8
# nvidia.com/gpu-memory: 80GB

# Cross-reference with nvidia-smi topo -m to confirm GPUs match cpuset NUMA

For GPU jobs, the typical issue is Pending pods with the event:

Warning  TopologyAffinityError  ... cannot satisfy single-numa-node policy

This means the pod requested 5 GPUs, but each NUMA node only has 4 — the constraint is unsatisfiable. Either drop to 4 GPUs or relax to restricted/best-effort.

Reading `nvidia-smi topo -m` and translating to a pinning recipe

Real H100 8-GPU node example:

        GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0  NIC1  NIC2  NIC3  CPU Affinity  NUMA
GPU0     X    NV18  NV18  NV18  NV18  NV18  NV18  NV18  PIX   NODE  SYS   SYS   0-47,96-143    0
GPU1    NV18   X    NV18  NV18  NV18  NV18  NV18  NV18  NODE  PIX   SYS   SYS   0-47,96-143    0
GPU2    NV18  NV18   X    NV18  NV18  NV18  NV18  NV18  NODE  NODE  SYS   SYS   0-47,96-143    0
GPU3    NV18  NV18  NV18   X    NV18  NV18  NV18  NV18  NODE  NODE  SYS   SYS   0-47,96-143    0
GPU4    NV18  NV18  NV18  NV18   X    NV18  NV18  NV18  SYS   SYS   PIX   NODE  48-95,144-191  1
GPU5    NV18  NV18  NV18  NV18  NV18   X    NV18  NV18  SYS   SYS   NODE  PIX   48-95,144-191  1
GPU6    NV18  NV18  NV18  NV18  NV18  NV18   X    NV18  SYS   SYS   NODE  NODE  48-95,144-191  1
GPU7    NV18  NV18  NV18  NV18  NV18  NV18  NV18   X    SYS   SYS   NODE  NODE  48-95,144-191  1

What the columns say:

GPU-to-GPU: all NV18 (NVLink 4.0 18-lane, 900 GB/s). Doesn't matter for pinning.
GPU-to-NIC: PIX = same PCIe switch, ideal for GPUDirect RDMA. NODE = same NUMA but different switch, OK. SYS = crosses sockets, bad.
NUMA column: GPU0-3 → node 0, GPU4-7 → node 1.
CPU Affinity: 0-47,96-143 for node 0 (cores 0-47 + their SMT siblings 96-143).

The pinning recipe drops out:

# Per local rank: which NUMA node, which NIC
declare -A RANK_NUMA=( [0]=0 [1]=0 [2]=0 [3]=0 [4]=1 [5]=1 [6]=1 [7]=1 )
declare -A RANK_NIC=( [0]=mlx5_0 [1]=mlx5_1 [2]=mlx5_0 [3]=mlx5_1
                      [4]=mlx5_2 [5]=mlx5_3 [6]=mlx5_2 [7]=mlx5_3 )

LR=$LOCAL_RANK
NODE=${RANK_NUMA[$LR]}
NIC=${RANK_NIC[$LR]}

NCCL_IB_HCA=$NIC \
NCCL_SOCKET_IFNAME=$(ibdev2netdev | awk -v dev=$NIC '$1==dev {print $5}') \
numactl --cpunodebind=$NODE --membind=$NODE \
  python -u train.py

Each rank gets:

Its own NUMA-local cores
Its own NUMA-local DRAM
The RDMA NIC on the same PCIe switch as its GPU (PIX, not SYS)

That's the difference between 380 GB/s and 90 GB/s on nccl-tests/all_reduce_perf.

Troubleshooting cross-NUMA penalties

Symptom: collective bandwidth halves when you add a second node

# On every node, capture during the run
mpstat -P ALL 1 5 > /tmp/cpu.log 2>&1 &
numastat -p $(pgrep -d, python) 1 5 > /tmp/numa.log 2>&1 &

# Look for: high CPU on the wrong socket relative to the GPUs
# Look for: numa_miss growing

If numa_miss is climbing during steady-state, your workers aren't NUMA-pinned.

Symptom: GPU visible to the wrong NUMA root

# /sys says one thing
cat /sys/bus/pci/devices/0000:17:00.0/numa_node
# 0

# But nvidia-smi says another
nvidia-smi topo -m | head -2
# ... CPU Affinity   NUMA Affinity
# GPU0  ...           -1                 <-- /sys disagrees!

This is a BIOS / ACPI SLIT table problem. The kernel populated /sys from one source and the NVIDIA driver read another. Common on systems where "NUMA per socket" was just toggled in BIOS and a reboot didn't propagate fully. Cold-boot the chassis (full power cycle, not just reboot) to force ACPI re-enumeration.

Symptom: container with cpuset crossing NUMA

# Inside the container
cat /sys/fs/cgroup/cpuset.cpus.effective
# 16-31,48-63             <-- crosses node 0 (16-47) and node 1 (48-95)

# Check which NUMA each core belongs to
for c in 16 31 48 63; do
  echo "core $c -> NUMA $(cat /sys/devices/system/cpu/cpu$c/topology/physical_package_id)"
done

The pod was scheduled with a CPU request that the kubelet split across sockets. Either:

Use TopologyManager single-numa-node (above) so admission would fail
Reduce CPU request to fit in one node
Use a static cpuset assignment in the pod spec

Symptom: `numa_balancing` still shows movement despite `kernel.numa_balancing=0`

# Check both sources
sysctl kernel.numa_balancing
cat /proc/cmdline | grep -o numa_balancing=\\S*

The kernel cmdline numa_balancing=enable overrides the sysctl. If you see numa_balancing=enable (or =1) on the cmdline, edit GRUB, regenerate, reboot. The sysctl can't override boot args.

Symptom: `numa_node` reads `-1` for one NIC, not GPUs

Some single-port BlueField / ConnectX cards expose -1 for NUMA on certain firmware versions. Workaround:

# Look up the actual PCIe root port
lspci -tv | grep -B3 mlx5
# Find the parent that DOES have a NUMA assignment
cat /sys/bus/pci/devices/0000:00:1f.0/numa_node    # the parent root
# 0

You can pin manually via numactl based on the parent's NUMA. Or update the NIC firmware — Mellanox/NVIDIA fixed this in mlx5 firmware ~2023 builds.