Slurm + SUNK install + GPU best practices

Running Slurm at scale on an H100/H200 fleet is not hard. Running it correctly — where "correctly" means GPUs are fully utilized, tenants are isolated, jobs don't leak into the next tenant's cgroup, and an operator at 03:00 can diagnose a failing node without tribal knowledge — requires deliberate choices at every layer. This page documents those choices and the reasoning behind them.

Why Slurm, not pure Kubernetes

Kubernetes scheduling is optimized for microservices: many small, fast-starting workloads, eviction is acceptable, co-scheduling is an afterthought. Training jobs are the opposite: large, long-running, sensitive to preemption, require gang-scheduling (all nodes or none), and users come from an HPC background where sbatch is muscle memory.

Slurm's fair-share, backfill, QoS, reservations, and sacct are purpose-built for exactly this. NCCL + MPI jobs have been running on Slurm for a decade; the failure modes are known.

The reason you still want K8s underneath is operational: you're probably already running K8s for everything else (inference endpoints, tooling, CI). SUNK gives you one control plane with two schedulers. The GPU nodes are K8s Node objects; slurmd runs as a privileged DaemonSet; and Slurm jobs are co-scheduled with K8s workloads through NodeSet reservations that exclude non-Slurm pods from the training fleet.

Alternatives:

• Kueue alone: better for K8s-native teams with no HPC background. Lacks fair-share tree, multi-level limits, sacct-style accounting.
• Volcano: richer gang scheduling than default K8s; still no fair-share decay, no reservations in the HPC sense.
• Bare Slurm without K8s: operationally simpler; you lose container portability and must manage node images manually.
• SUNK community (SlinkyProject/slurm-operator): same wire format, open source, less supported. Suitable if you don't run CoreWeave infrastructure.

Cluster topology assumed

This guide targets a topology that recurs at most GPU operators:

controller nodes (2) — slurmctld active+backup
                       slurmdbd + MariaDB
login nodes (2-4)    — StatefulSet / bare-metal, user SSH entry points
compute nodes        — gpu-01..gpu-N (H100/H200, 8 GPUs each)
                       slurmd DaemonSet, direct host access
management network   — 10.0.0.0/24, all nodes
IB / RoCE fabric     — separate VLAN / subnet, compute only
storage              — Weka FS, mounted on compute + login nodes

The diagram below shows how these roles wire together at runtime. Read the arrows as RPC paths; a broken arrow means that daemon goes down, not just that flow stops.

The same layout in plain ASCII (for terminal-friendly viewing):

  ┌──────────────────────────────────────────────────────────────────────┐
  │  Management network  10.0.0.0/24                                     │
  │                                                                      │
  │  ┌─────────────────────┐     ┌──────────────────────────────────┐    │
  │  │ slurmctld (active)  │────►│ slurmdbd  10.0.0.12:6819         │    │
  │  │ cp-01  10.0.0.10    │     │ (accounting writes, fair-share)  │    │
  │  └─────────┬───────────┘     └─────────────┬────────────────────┘    │
  │            │  heartbeat/                   │                         │
  │            │  state sync                   ▼                         │
  │  ┌─────────▼───────────┐     ┌─────────────────────────────────┐     │
  │  │ slurmctld (backup)  │     │ MariaDB  10.0.0.13:3306          │     │
  │  │ cp-02  10.0.0.11    │     │ (slurm_acct_db, Galera or single)│     │
  │  └─────────────────────┘     └─────────────────────────────────┘     │
  │            │                                                         │
  │    job dispatch (RPC, munge-authenticated)                           │
  │            │                                                         │
  │   ┌────────┼──────────────────────────┐                              │
  │   ▼        ▼                          ▼                              │
  │ ┌──────┐ ┌──────┐              ┌─────────────────────────────────┐   │
  │ │login-│ │login-│  ssh/sbatch  │ compute fleet  gpu-01..gpu-128  │   │
  │ │pod-01│ │pod-02│─────────────►│   slurmd + munge + enroot/pyxis │   │
  │ └──────┘ └──────┘              │   /dev/nvidia[0-7], IB HCAs     │   │
  │                                └─────────────────────────────────┘   │
  │                                                                      │
  │  munge.key fanned out to every node/pod via K8s Secret (SUNK)        │
  │  StateSaveLocation on shared RWX PVC — survives slurmctld restart    │
  └──────────────────────────────────────────────────────────────────────┘

Failure domains: losing slurmctld primary hands off to backup via BackupController; losing slurmdbd freezes fair-share but leaves scheduling intact; losing a slurmd pod marks only that node DOWN.

The controller nodes are intentionally not GPU nodes. Mixing the scheduler and GPU workloads creates OOM contention and makes firmware reboots tricky.

OS prerequisites

Kernel command line

# /etc/default/grub — add to GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="... cgroup_memory=1 cgroup.memory=nokmem \
  systemd.unified_cgroup_hierarchy=1 \
  iommu=pt intel_iommu=on \
  hugepagesz=1G hugepages=160 \
  transparent_hugepage=never \
  isolcpus=nohz,domain:1-7,9-15,17-23,25-31 \
  nohz_full=1-7,9-15,17-23,25-31 \
  rcu_nocbs=1-7,9-15,17-23,25-31 \
  numa_balancing=disable \
  pci=realloc=off"

cgroup_memory=1 is required for Slurm's cgroup/v2 plugin to enforce memory limits. Without it, jobs can blow past their --mem= allocation and kill neighboring jobs. systemd.unified_cgroup_hierarchy=1 enables v2 unified hierarchy — Slurm's cgroup/v2 plugin requires it since Slurm 23.11.

isolcpus removes cores from the general kernel scheduler so user tasks have uninterrupted access. Adjust the mask to your NUMA topology — the example leaves cores 0, 8, 16, 24 for the OS on a dual-socket 128-core machine.

transparent_hugepage=never prevents page defragmentation stalls mid-job. Large training jobs run better on explicit --mem-per-cpu + mlock than on the kernel doing it behind their back.

ulimits

# /etc/security/limits.d/slurm.conf
*        soft  nofile      1048576
*        hard  nofile      1048576
*        soft  memlock     unlimited
*        hard  memlock     unlimited
*        soft  stack       unlimited
*        hard  stack       unlimited
*        soft  nproc       unlimited
*        hard  nproc       unlimited

memlock=unlimited is required for RDMA (GDR). Without it, ibv_reg_mr fails with ENOMEM and NCCL falls back to host memory, cutting effective bandwidth by 30-40%.

Time synchronization

Munge requires clocks within 300 seconds (default) across all nodes. Use chrony:

# /etc/chrony.conf — compute nodes
server 10.0.0.1 iburst
maxdistance 1.0
makestep 1.0 3
rtcsync

Set MungeKeyTimeout=1800 in /etc/munge/munge.conf if you're in a stretched datacenter. Do not rely on the default; 300s is tight if NTP drift accumulates during a power event.

cgroup v2

Slurm's cgroup/v2 plugin (shipped in 23.11+) requires:

# Verify
stat -fc %T /sys/fs/cgroup
# must print: cgroup2fs

# Enable if not already
grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1"
# reboot required

pam_slurm_adopt

pam_slurm_adopt intercepts SSH logins to compute nodes and moves the SSH session into the user's existing job cgroup. Without it, a user SSHing directly to a GPU node bypasses all resource limits. This matters because users with a long-running job sometimes SSH in to run nvidia-smi and accidentally start a second Python process that consumes GPU memory outside any quota.

# /etc/pam.d/sshd — on compute nodes only
# Add BEFORE pam_systemd_home or equivalent session line:
session  required  pam_slurm_adopt.so

The PAM module is part of the Slurm distribution. You also need UsePAM yes in /etc/ssh/sshd_config and the node must be a Slurm compute node (not login or controller). Incoming SSH on a node with no matching job → SSH is rejected. That is intentional.

# /etc/slurm/slurm.conf — enable SSH enforcement
PrologFlags=contain

With PrologFlags=contain, slurmd creates the cgroup hierarchy before the job starts. pam_slurm_adopt then finds the cgroup and adopts the SSH session into it. If the job doesn't exist (NodeNotInPartition, no_job_found), the user is denied entry.

Munge key distribution and rotation

Munge is Slurm's authentication layer. Every daemon-to-daemon RPC is wrapped in a munge credential. Any node or pod with a different key cannot authenticate.

# Generate once on the slurmctld node
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
chmod 400 /etc/munge/munge.key
chown munge:munge /etc/munge/munge.key

# Distribute to all nodes
for node in $(scontrol show hostnames gpu-[01-16]); do
    scp /etc/munge/munge.key root@${node}:/etc/munge/munge.key
    ssh root@${node} "chmod 400 /etc/munge/munge.key && chown munge:munge /etc/munge/munge.key && systemctl restart munge"
done

In SUNK, the key is stored in a Kubernetes Secret and mounted into all pods:

apiVersion: v1
kind: Secret
metadata:
  name: munge-key
  namespace: tenant-foo
type: Opaque
data:
  munge.key: <base64-encoded key>

Rotation procedure: Generate new key → update the Secret → roll all slurmd/slurmctld/slurmdbd pods with a brief overlap window. Because rolling happens pod-by-pod, briefly have two keys in flight. Solutions: (a) use MungeKeyTimeout > pod rollout time, (b) deploy the new key as a second Secret and rotate atomically via a Job that updates the Secret and triggers a rollout.

slurm.conf for GPU clusters

The canonical fields that matter most. Full reference: man slurm.conf.

# /etc/slurm/slurm.conf

ClusterName=production
SlurmctldHost=cp-01(10.0.0.10)   # primary
SlurmctldHost=cp-02(10.0.0.11)   # backup
SlurmctldPort=6817
SlurmdPort=6818
SlurmUser=slurm

StateSaveLocation=/var/spool/slurm/state
SlurmdSpoolDir=/var/spool/slurm/slurmd

AuthType=auth/munge
MungeSocketPath=/var/run/munge/munge.socket.2

CryptoType=crypto/munge
JobAcctGatherType=jobacct_gather/cgroup
JobAcctGatherFrequency=30
AcctGatherEnergyType=acct_gather_energy/ipmi

# Accounting — point at slurmdbd
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=10.0.0.12
AccountingStoragePort=6819
AccountingStorageTRES=gres/gpu,gres/nvme
AccountingStorageEnforce=associations,limits,qos,safe

# TRES billing — charge GPU-hours to accounts
TRESBillingWeights="CPU=0.1,Mem=0.005G,GRES/gpu=100,GRES/nvme=0.5"

# Scheduler
SchedulerType=sched/backfill
SchedulerParameters=bf_interval=30,bf_window=4320,bf_resolution=600,bf_max_job_test=500,bf_max_job_user=20,bf_continue,default_queue_depth=200,partition_job_depth=50

# Priority
PriorityType=priority/multifactor
PriorityDecayHalfLife=14-0
PriorityCalcPeriod=5
PriorityFlags=ACCRUE_ALWAYS,FAIR_TREE,SMALL_RELATIVE_TO_TIME
PriorityWeightAge=10000
PriorityMaxAge=7-0
PriorityWeightFairshare=100000
PriorityWeightQOS=50000
PriorityWeightTRES=CPU=0,Mem=0,GRES/gpu=10000

# GRES — declare GPU and NVMe as generic resources
GresTypes=gpu,nvme

# Process tracking — cgroup, not pid
ProctrackType=proctrack/cgroup
TaskPlugin=task/affinity,task/cgroup
TaskPluginParam=SlurmdOffSpec

# Cgroup config reference
CgroupPlugin=cgroup/v2

# Prolog/Epilog
Prolog=/etc/slurm/prolog.sh
Epilog=/etc/slurm/epilog.sh
PrologFlags=contain,nohold
EpilogMsgTime=30
PrologSlurmctld=/etc/slurm/prolog.slurmctld.sh
EpilogSlurmctld=/etc/slurm/epilog.slurmctld.sh

# Select: topology-aware TRES packing
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory,CR_CORE_DEFAULT_DIST_BLOCK,CR_ONE_TASK_PER_CORE

# Timeouts
SlurmctldTimeout=120
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
WaitTime=0

# Nodes
NodeName=gpu-[01-16] \
    Sockets=2 CoresPerSocket=64 ThreadsPerCore=2 \
    RealMemory=2000000 \
    Gres=gpu:h100:8,nvme:1 \
    Feature=h100,ib,weka \
    State=UNKNOWN

# Partitions
PartitionName=training \
    Nodes=gpu-[01-12] \
    Default=YES \
    MaxTime=7-00:00:00 \
    DefaultTime=1-00:00:00 \
    State=UP \
    OverSubscribe=NO \
    PriorityTier=1 \
    TRESBillingWeights="GRES/gpu=100"

PartitionName=inference \
    Nodes=gpu-[13-16] \
    MaxTime=infinite \
    DefaultTime=4:00:00 \
    State=UP \
    OverSubscribe=YES:4 \
    PriorityTier=2 \
    TRESBillingWeights="GRES/gpu=50"

PartitionName=interactive \
    Nodes=gpu-[01-16] \
    MaxTime=4:00:00 \
    DefaultTime=1:00:00 \
    State=UP \
    OverSubscribe=NO \
    PriorityTier=3

Key decisions:

• SelectType=cons_tres — topology-aware packing. cons_res is legacy; cons_tres is required for GRES-aware scheduling.
• TaskPlugin=task/affinity,task/cgroup — both required. affinity binds tasks to NUMA-local cores; cgroup enforces the binding in the kernel.
• AccountingStorageEnforce=safe — blocks submission when slurmdbd is unreachable rather than silently dropping accounting records. Strict but correct for multi-tenant billing.
• PriorityFlags=ACCRUE_ALWAYS — pending jobs accumulate age priority even before they become eligible (e.g., while waiting for a reservation start time). Without this, a job gated behind a future reservation has zero age when the reservation opens and gets stuck behind freshly submitted jobs.
• SMALL_RELATIVE_TO_TIME — normalizes job size factor so a 1-GPU/1-hour job and an 8-GPU/8-hour job carry equal priority contribution. Prevents large jobs from always losing the size lottery.
• OverSubscribe=NO on training, YES:4 on inference — training jobs need exclusive GPU access. Inference pods can share via MPS (see GPU best practices section). Setting OverSubscribe=YES:4 allows up to 4 jobs per node; actual GPU sharing enforcement is at the MPS layer.

gres.conf for H100/H200 with NVLink topology

# /etc/slurm/gres.conf — on every compute node
# H100 SXM 80GB, 8 GPUs per node
# NVLink topology: GPUs 0-3 form one NVSwitch domain, GPUs 4-7 another
# IB HCAs: mlx5_0 (rail 0), mlx5_1 (rail 1), mlx5_2 (rail 2), mlx5_3 (rail 3)
# Each rail HCA is closest to 2 GPUs

Name=gpu Type=h100 File=/dev/nvidia0 Cores=0-31   Links=0:1:2:3:NVL:NVL:NVL:NVL
Name=gpu Type=h100 File=/dev/nvidia1 Cores=0-31   Links=1:0:2:3:NVL:NVL:NVL:NVL
Name=gpu Type=h100 File=/dev/nvidia2 Cores=0-31   Links=2:3:0:1:NVL:NVL:NVL:NVL
Name=gpu Type=h100 File=/dev/nvidia3 Cores=0-31   Links=3:2:1:0:NVL:NVL:NVL:NVL
Name=gpu Type=h100 File=/dev/nvidia4 Cores=32-63  Links=NVL:NVL:NVL:NVL:0:1:2:3
Name=gpu Type=h100 File=/dev/nvidia5 Cores=32-63  Links=NVL:NVL:NVL:NVL:1:0:2:3
Name=gpu Type=h100 File=/dev/nvidia6 Cores=32-63  Links=NVL:NVL:NVL:NVL:2:3:0:1
Name=gpu Type=h100 File=/dev/nvidia7 Cores=32-63  Links=NVL:NVL:NVL:NVL:3:2:1:0

# NVMe — declare per physical device
Name=nvme Type=local File=/dev/nvme0n1

# IB HCA GRES (optional; lets you constrain jobs to specific rails)
# Name=ib Type=mlx5 File=/dev/infiniband/uverbs0 Cores=0-31
# Name=ib Type=mlx5 File=/dev/infiniband/uverbs1 Cores=0-31
# Name=ib Type=mlx5 File=/dev/infiniband/uverbs2 Cores=32-63
# Name=ib Type=mlx5 File=/dev/infiniband/uverbs3 Cores=32-63

The diagram below maps the eight GPU entries to their NUMA node and the IB HCA rail that carries their collective traffic off-node. Slurm uses Cores= to keep CPU affinity within the same socket as the GPU.

ASCII equivalent:

  ┌──────────────────────────────────────────────────────────────────┐
  │  H100 SXM 8-GPU node  (dual socket, 2 NVSwitch domains)         │
  │                                                                  │
  │  NUMA 0  (CPUs 0-63)              NUMA 1  (CPUs 32-63)           │
  │  ┌───────────────────────────┐    ┌───────────────────────────┐  │
  │  │ GPU0 /dev/nvidia0         │    │ GPU4 /dev/nvidia4         │  │
  │  │  Cores=0-31  Links=NVSwA  │    │  Cores=32-63 Links=NVSwB  │  │
  │  │  ↕ NVLink (NVSwitch A)    │    │  ↕ NVLink (NVSwitch B)    │  │
  │  │ GPU1 /dev/nvidia1         │    │ GPU5 /dev/nvidia5         │  │
  │  │  Cores=0-31  Links=NVSwA  │    │  Cores=32-63 Links=NVSwB  │  │
  │  │  ↕ NVLink                 │    │  ↕ NVLink                 │  │
  │  │ GPU2 /dev/nvidia2         │    │ GPU6 /dev/nvidia6         │  │
  │  │  Cores=0-31  Links=NVSwA  │    │  Cores=32-63 Links=NVSwB  │  │
  │  │  ↕ NVLink                 │    │  ↕ NVLink                 │  │
  │  │ GPU3 /dev/nvidia3         │    │ GPU7 /dev/nvidia7         │  │
  │  │  Cores=0-31  Links=NVSwA  │    │  Cores=32-63 Links=NVSwB  │  │
  │  └───────────┬───────────────┘    └───────────┬───────────────┘  │
  │              │ PCIe                            │ PCIe             │
  │     ┌────────┴──────────┐           ┌──────────┴─────────┐       │
  │     │ mlx5_0  mlx5_1   │           │ mlx5_2   mlx5_3   │       │
  │     │ IB rail 0, rail 1 │           │ IB rail 2, rail 3  │       │
  │     └────────┬──────────┘           └──────────┬─────────┘       │
  │              └──────────────┬──────────────────┘                 │
  │                             ▼                                     │
  │                     IB / RoCE fabric                             │
  └──────────────────────────────────────────────────────────────────┘

  gres.conf consequence (cross-domain 4-GPU job WITHOUT Links= field):
    Possible assignment: GPU0, GPU2, GPU5, GPU7 — crosses NVSwitch domains
    Collective latency penalty: up to 2× vs intra-domain

  gres.conf consequence (WITH Links= field):
    Slurm prefers GPU0+GPU1+GPU2+GPU3 (same NVSwitch A domain)
    or   GPU4+GPU5+GPU6+GPU7 (same NVSwitch B domain)

The Links= field is the NVLink topology bitmask. NVL means NVLink-connected; an integer is the link count to that peer. Slurm uses this to prefer allocating GPUs that share NVSwitch domain when a job requests fewer than 8. Without it, a 4-GPU job might get GPUs 2, 3, 5, 7 — crossing NVSwitch domains and reducing effective all-reduce bandwidth by up to 50% for inter-domain transfers.

Generate the correct Links= for your hardware:

nvidia-smi topo -m
# Read the NVL column — GPUs on the same NVSwitch show NVL for each other

For H200 SXM: topology is identical to H100 SXM at the software layer. Same gres.conf format, Type=h200 if you want to differentiate.

slurmdbd + MariaDB bootstrap

MariaDB setup

# Install on db node (RHEL/Rocky 9 example)
dnf install -y mariadb-server mariadb-devel

# Sizing: innodb_buffer_pool_size = 70-80% of DB node RAM
# For a 64 GB DB node: 45G
cat >> /etc/my.cnf.d/slurm.cnf <<'EOF'
[mysqld]
innodb_buffer_pool_size = 45G
innodb_log_file_size    = 1G
innodb_lock_wait_timeout = 900
innodb_log_buffer_size  = 64M
max_allowed_packet      = 64M

# Tuning for slurmdbd write patterns
innodb_flush_log_at_trx_commit = 2
sync_binlog = 0
EOF

systemctl enable --now mariadb

# mysql_secure_installation: remove anonymous user, remove test database,
# disallow remote root login. Do NOT skip this step.
mysql_secure_installation

The mysql_secure_installation step that new operators skip: when prompted "Remove test database and access to it?", answer Y. When prompted "Disallow root login remotely?", answer Y. The test database is a well-known attack surface; remote root lets any network-visible host brute-force the DB.

-- Create slurmdbd user and database
CREATE DATABASE slurm_acct_db;
CREATE USER 'slurm'@'10.0.0.12' IDENTIFIED BY 'strong-password-here';
GRANT ALL ON slurm_acct_db.* TO 'slurm'@'10.0.0.12';
FLUSH PRIVILEGES;

innodb_lock_wait_timeout = 900 — at scale (>500 running jobs), slurmdbd fires concurrent writes to the job and step tables. At the default timeout (50s), lock contention during peak submission windows produces slurmdbd log spam and occasional timeout errors that bubble up to slurmctld as "accounting unavailable". 900s is conservative; tune down once you know your contention profile.

slurmdbd.conf

# /etc/slurm/slurmdbd.conf
AuthType=auth/munge
AuthInfo=/var/run/munge/munge.socket.2
DbdHost=10.0.0.12
DbdPort=6819
SlurmUser=slurm

# Storage
StorageType=accounting_storage/mysql
StorageHost=10.0.0.12
StorageUser=slurm
StoragePass=strong-password-here
StorageLoc=slurm_acct_db

# Archiving (optional; reduces table size at cost of query complexity)
ArchiveEvents=yes
ArchiveJobs=yes
ArchiveSteps=no
ArchiveSuspend=no
ArchiveResv=yes
ArchiveUsage=no
PurgeEventAfter=12months
PurgeJobAfter=24months

LogFile=/var/log/slurm/slurmdbd.log
DebugLevel=info

Bootstrap the schema:

systemctl enable --now slurmdbd
# First start creates schema automatically.
# Check:
journalctl -u slurmdbd -n 50
# Look for "accounting_storage/mysql plugin loaded successfully"

Add the cluster name to accounting:

sacctmgr -i add cluster production
sacctmgr -i add account root \
    Description="Root account" \
    Organization=platform \
    Cluster=production

enroot + pyxis install

Enroot is a rootless container runtime. Pyxis is the Slurm SPANK plugin that intercepts job spawning and calls enroot. Together they let users run sbatch --container-image=nvcr.io/nvidia/pytorch:25.04-py3 train.sh without any container daemon running on the compute node.

# enroot — install from NVIDIA GitHub releases
# https://github.com/NVIDIA/enroot/releases
dnf install -y fuse-overlayfs squashfuse libcap

# Download and verify enroot RPM
ENROOT_VERSION=3.5.0
rpm -i https://github.com/NVIDIA/enroot/releases/download/v${ENROOT_VERSION}/enroot-${ENROOT_VERSION}-1.el9.x86_64.rpm
rpm -i https://github.com/NVIDIA/enroot/releases/download/v${ENROOT_VERSION}/enroot+caps-${ENROOT_VERSION}-1.el9.x86_64.rpm

# pyxis SPANK plugin
dnf install -y slurm-spank-pyxis
# or build from source: https://github.com/NVIDIA/pyxis

enroot.conf — cache layout decision

This is the choice that bites you later if you get it wrong.

# /etc/enroot/enroot.conf

# Option A: tmpfs cache (fast pull, lost on reboot, fits in RAM)
# Use when: nodes have 512+ GB RAM, containers are <50 GB, users pull each run
# ENROOT_CACHE_PATH=/run/enroot/cache

# Option B: local NVMe (fast pull, survives reboot, bounded by disk)
# Use when: nodes have fast NVMe, pull-once-run-many workflow
ENROOT_CACHE_PATH=/mnt/nvme/enroot/cache

# Option C: Weka shared FS (pull once across all nodes, slow first pull, FS pressure)
# Use when: container images are large (>50 GB NVCR containers), low NVMe
# ENROOT_CACHE_PATH=/weka/enroot/cache

# Common settings regardless of cache location
ENROOT_DATA_PATH=/run/enroot/data
ENROOT_TEMP_PATH=/run/enroot/tmp
ENROOT_SQUASH_OPTIONS="-noI -noX -noF -noD -processors 16"

# NVIDIA NVCR credentials (if pulling from NGC)
# ENROOT_LOGIN_SERVER=nvcr.io
# ENROOT_LOGIN_USER=$oauthtoken
# ENROOT_LOGIN_PASSWORD=<ngc-api-key>

The job dispatch path through enroot/pyxis is worth tracing once so you know where each failure mode lives. Steps after the dashed line are per-job; everything above it is shared and reused across jobs on the same node.

Plain-text reference:

  sbatch train.sh  (login pod)
        │
        ▼  Slurm RPC
  slurmctld  ──dispatch──►  slurmd  (compute node)
                                 │
                                 ▼  prolog.sh runs first (as root)
                          cgroup hierarchy created
                          nvidia-fabricmanager check
                          container pre-pull (if prolog configured)
                                 │
                      ─ ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
                    (reused across jobs on same node, if image cached)
                                 ▼
                          pyxis SPANK hook fires
                                 │
                    ┌────────────▼────────────────────────────┐
                    │ ENROOT_CACHE_PATH  (NVMe or tmpfs)      │
                    │   pytorch-25.04-py3.sqsh   ◄── reused   │
                    │   (squashfs blob, read-only mount)       │
                    └────────────┬────────────────────────────┘
                                 │  enroot start (per-job)
                                 ▼
                    ┌────────────────────────────────────────┐
                    │ ENROOT_DATA_PATH  (tmpfs, per-job)     │
                    │   /run/enroot/data/<jobid>/            │
                    │   overlay upper layer (writable)       │
                    │   /home/user   ← mount_home=1 bind     │
                    │   /weka/...    ← container-mounts      │
                    └────────────┬───────────────────────────┘
                                 │
                                 ▼
                          user process runs (CUDA app)
                          GPU access via /dev/nvidia[0-7]
                          IB access via /dev/infiniband/uverbs*
                                 │
                                 ▼  epilog.sh (as root)
                          DATA_PATH overlay torn down
                          NVMe sqsh blob kept → reused next job

The trade-off:

• tmpfs: zero I/O overhead for execution, but the 512 GB limit is exhausted by ~6 concurrent large container pulls. On a 16-node job, all 16 nodes pull simultaneously. That's a container stampede.
• local NVMe: survives reboots and survives cache stampedes if images are pre-staged. Use a pre-pull prolog script.
• Weka: one pull for all nodes via the shared FS, but puts FS load onto the same path as training checkpoints. Acceptable if your Weka cluster is sized for it; painful if it isn't.

In practice: NVMe cache + a prolog that pre-pulls containers before the job starts is the most operationally predictable.

# /etc/slurm/prolog.sh (partial — container pre-pull)
#!/bin/bash
if [ -n "${ENROOT_IMAGE}" ]; then
    enroot import --output ${ENROOT_CACHE_PATH}/$(echo ${ENROOT_IMAGE} | tr '/:' '_').sqsh \
        docker://${ENROOT_IMAGE} 2>/dev/null || true
fi

pyxis plugin config

# /etc/slurm/plugstack.conf
required /usr/lib64/slurm/spank_pyxis.so  \
    container_scope=job \
    mount_home=1 \
    remap_root=0 \
    sbatch_support=1 \
    srun_support=1 \
    container_writable=1

remap_root=0 — do not remap the container root to a non-root UID. The container runs as the submitting user. mount_home=1 — mount the user's home directory into the container, so job scripts and datasets at known paths are accessible.

pam_slurm_adopt

Covered in the OS prereqs section above, but the full config:

# /etc/pam.d/sshd — compute nodes
auth     required  pam_unix.so
auth     required  pam_listfile.so item=user sense=deny file=/etc/security/deny_login onerr=fail

account  required  pam_nologin.so
account  required  pam_unix.so
account  required  pam_slurm_adopt.so log_level=debug  # see logs during testing, drop after

session  required  pam_limits.so
session  required  pam_unix.so
session  required  pam_slurm_adopt.so   # cgroup adoption happens here

password required  pam_unix.so

Test before enabling widely:

# From a login node, submit a job that sleeps
srun --gres=gpu:1 -N 1 sleep 300 &

# From another terminal, try to SSH to the assigned compute node
ssh gpu-01  # should succeed and land in the job's cgroup

# Verify: check /proc/$$/cgroup on the compute node
cat /proc/$$/cgroup
# Should show: 0::/system.slice/slurm-job-<jobid>.scope

Prolog and epilog scripts

The prolog runs as root before the job starts. The epilog runs after. Both are synchronous from slurmd's perspective — a non-zero exit halts the job before it begins (prolog) or leaves the node in a bad state (epilog). Keep them fast; 30 seconds is the limit before slurmctld starts worrying.

#!/bin/bash
# /etc/slurm/prolog.sh

set -euo pipefail
LOG=/var/log/slurm/prolog-${SLURM_JOB_ID}.log
exec >> "${LOG}" 2>&1

echo "=== prolog start $(date -Iseconds) ==="
echo "  job=${SLURM_JOB_ID} user=${SLURM_JOB_USER} nodes=${SLURM_NODELIST}"

# 1. Enable persistence mode
nvidia-smi -pm 1 || { echo "WARN: persistence mode failed"; }

# 2. Reset MIG geometry to default (full GPU) before each job
# Prevents a previous job's MIG config from leaking
nvidia-smi mig -cgip 2>/dev/null || true

# 3. Set GPU clocks for training workloads
# Lock to max memory clock, boost graphics clock
nvidia-smi -lgc 1980,1980 --scope job 2>/dev/null || nvidia-smi --gpu-reset-applications-clocks

# 4. Fabric Manager liveness check — required for NVLink on multi-GPU nodes
if ! systemctl is-active --quiet nvidia-fabricmanager; then
    echo "ERROR: nvidia-fabricmanager not running — starting"
    systemctl start nvidia-fabricmanager
    sleep 5
    if ! systemctl is-active --quiet nvidia-fabricmanager; then
        echo "FATAL: fabric-manager failed to start"
        exit 1
    fi
fi

# 5. Wipe /tmp — jobs should not inherit previous job's temp files
find /tmp -maxdepth 1 -not -name '.' -not -name '..' -delete 2>/dev/null || true

# 6. Verify ECC mode (training = on, inference nodes may differ)
PARTITION=${SLURM_JOB_PARTITION:-unknown}
if [[ "${PARTITION}" == "training" ]]; then
    nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader | grep -q "Enabled" || {
        echo "WARN: ECC not enabled on training partition node"
    }
fi

echo "=== prolog complete $(date -Iseconds) ==="
exit 0

#!/bin/bash
# /etc/slurm/epilog.sh

set -euo pipefail
LOG=/var/log/slurm/epilog-${SLURM_JOB_ID}.log
exec >> "${LOG}" 2>&1

echo "=== epilog start $(date -Iseconds) ==="

# 1. Kill any stray processes (belt-and-suspenders; cgroup teardown should handle this)
# slurmd with ProctrackType=cgroup does this automatically, but log it
for gpu in $(seq 0 7); do
    gpu_pids=$(nvidia-smi -i ${gpu} --query-compute-apps=pid --format=csv,noheader 2>/dev/null || echo "")
    if [ -n "${gpu_pids}" ]; then
        echo "WARN: stray GPU ${gpu} pids after job ${SLURM_JOB_ID}: ${gpu_pids}"
    fi
done

# 2. Reset clock locks set in prolog
nvidia-smi --reset-gpu-clocks 2>/dev/null || true

# 3. Wipe NCCL shared memory segments if any leaked
ls /dev/shm/ | grep -E "nccl|sem\." | xargs -I{} rm -f /dev/shm/{} 2>/dev/null || true

# 4. Verify fabric-manager still running (crash during a job is worth alerting on)
if ! systemctl is-active --quiet nvidia-fabricmanager; then
    echo "ERROR: fabric-manager died during job ${SLURM_JOB_ID}"
    systemctl start nvidia-fabricmanager || true
fi

echo "=== epilog complete $(date -Iseconds) ==="
exit 0

SUNK install

Prerequisites

SUNK requires:

• Kubernetes 1.27+ with a working GPU Operator (NVIDIA drivers on all compute nodes)
• cert-manager installed in the cluster
• A StorageClass for the controller's StateSaveLocation (ReadWriteMany — Weka CSI or NFS)
• A MariaDB instance reachable from the namespace (can be the SUNK-managed one)

Helm install (CoreWeave SUNK)

# Add SUNK Helm repository
helm repo add sunk https://helm.coreweave.com/sunk
helm repo update

# Install the operator into its own namespace
helm install sunk-operator sunk/sunk-operator \
    --namespace sunk-system \
    --create-namespace \
    --set controller.replicaCount=2 \
    --set controller.leaderElection.enabled=true \
    --wait

# Verify
kubectl -n sunk-system get pods
# sunk-operator-controller-manager-xxx   2/2   Running

Controller HA

The SUNK operator uses Kubernetes leader election (lease.coordination.k8s.io). Two replicas run; one holds the leader lease. The other is a hot standby that takes over within the LeaseDuration (default 15s) when the leader becomes unreachable.

The slurmctld pods themselves also run as replicas: 2 with Slurm's own BackupController mechanism:

# slurm.conf (generated by SUNK, but useful to know)
SlurmctldHost=slurmctld-0.slurmctld.tenant-foo.svc.cluster.local(10.0.0.20)
SlurmctldHost=slurmctld-1.slurmctld.tenant-foo.svc.cluster.local(10.0.0.21)

The state migration path is what determines recovery time. Both replicas mount the same RWX PVC; primary writes continuously, backup reads on takeover.

ASCII reference:

  ┌──────────────────────────┐      ┌──────────────────────────┐
  │  slurmctld-0 (PRIMARY)   │      │  slurmctld-1 (BACKUP)    │
  │  10.0.0.20:6817          │      │  10.0.0.21:6817          │
  │                          │      │                          │
  │  ● accepts RPCs          │      │  ● idle — polls primary  │
  │  ● writes state files    │      │    via SlurmctldTimeout  │
  │    every checkpoint      │      │    (default 120s)        │
  │    interval              │      │  ● waits for lock file   │
  └────────────┬─────────────┘      └──────────────┬───────────┘
               │  write                             │  read on failover
               ▼                                   ▼
  ┌────────────────────────────────────────────────────────────┐
  │  StateSaveLocation  (shared RWX PVC — Weka CSI / NFS)      │
  │  /var/spool/slurm/state/                                   │
  │    job_state       — all pending/running job records       │
  │    node_state      — node UP/DOWN/DRAIN flags              │
  │    part_state      — partition definitions                 │
  │    resv_state      — active reservations                   │
  └────────────────────────────────────────────────────────────┘

On primary loss, backup reads the state files and resumes scheduling within SlurmctldTimeout (120s default). Jobs that completed while primary was down have their accounting written when slurmdbd connection is re-established.

StateSaveLocation must be on a shared RWX PVC that both replicas can mount. SUNK uses a PersistentVolumeClaim with accessModes: [ReadWriteMany].

ReservationBinding

A ReservationBinding maps a Slurm reservation to a set of K8s nodes that should be tainted for exclusive Slurm use:

apiVersion: slurm.coreweave.com/v1alpha1
kind: ReservationBinding
metadata:
  name: tenant-foo-training
  namespace: tenant-foo
spec:
  clusterRef:
    name: cluster-tenant-foo
  nodeSelectorTerms:
    - matchExpressions:
        - key: reserved.tenant
          operator: In
          values: [tenant-foo]
        - key: workload-type
          operator: In
          values: [training]
  taint:
    key: reserved.tenant
    value: tenant-foo
    effect: NoSchedule
  reservation:
    name: tenant-foo-training
    startTime: "now"
    duration: "infinite"
    accounts: "tenant-foo"
    users: ""   # empty = all users in the accounts
    partitions: "training"

This taint + toleration pattern ensures that nodes reserved for tenant-foo via Slurm are also protected from K8s-native workloads. If the ReservationBinding controller loses leader election and stops reconciling, the K8s taints remain (they were already applied); new nodes added to the pool will not get the taint until the controller recovers. That is a known gap.

The SUNK controller sits at the boundary between two control planes. The diagram below shows the cross-domain arrows that matter for day-2 ops.

Plain-text reference:

  ┌──────────────────────────────────────────────────────────────────┐
  │  Kubernetes cluster                                              │
  │                                                                  │
  │  ┌─────────────────────────────────────────────────────────┐    │
  │  │  sunk-operator  (Deployment, 2 replicas, leader-elected) │    │
  │  │                                                          │    │
  │  │  watches: SlurmCluster, NodeSet, ReservationBinding CRDs │    │
  │  │  writes:  slurm.conf ConfigMap, munge.key Secret,        │    │
  │  │           Node taints, Pod specs                         │    │
  │  │  exposes: /healthz  /readyz  /metrics                    │    │
  │  │  leader lease: lease.coordination.k8s.io/sunk-operator   │    │
  │  └────────────────────────┬────────────────────────────────┘    │
  │                           │ reconcile (K8s API)                  │
  │           ┌───────────────┼────────────────────┐                │
  │           ▼               ▼                    ▼                │
  │  ┌──────────────┐  ┌─────────────┐   ┌──────────────────────┐  │
  │  │ slurmctld    │  │ slurmdbd    │   │ slurmd DaemonSet      │  │
  │  │ Pod (×2 HA)  │  │ Pod (×1)    │   │ (one pod per K8s node)│  │
  │  │              │  │             │   │                       │  │
  │  │ reads:       │  │ reads:      │   │ reads:                │  │
  │  │  slurm.conf  │  │  slurmdbd   │   │  slurm.conf           │  │
  │  │  ConfigMap   │  │  .conf CM   │   │  ConfigMap            │  │
  │  │  munge.key   │  │  munge.key  │   │  munge.key Secret     │  │
  │  │  Secret      │  │  Secret     │   │                       │  │
  │  └──────┬───────┘  └──────┬──────┘   └──────────┬────────────┘  │
  │         │                 │                      │              │
  └─────────┼─────────────────┼──────────────────────┼──────────────┘
            │  Slurm RPC      │  Slurm DBD RPC       │  Slurm RPC
            │  (munge auth)   │  (munge auth)        │  (munge auth)
            └─────────────────┴──────────────────────┘
                   These arrows are Slurm-native — they survive
                   K8s API unavailability as long as pods are running

slurmctld ↔ slurmd traffic never touches the K8s API. If the operator loses its leader lease, scheduling continues; only CRD reconciliation stops.

What fails first when SUNK loses leader election

1. New NodeSet or SlurmCluster CRD reconciliation stops. Existing deployments continue running unchanged. slurmd pods stay up; scheduling continues.
2. ReservationBinding taints are not applied to newly added nodes. Nodes that join the cluster while the operator is down may accept K8s-native pods even if they should be Slurm-only.
3. Image updates in CRD specs are not applied. A spec.controller.image bump during leader election outage is silently queued.
4. Slurm itself keeps running. slurmctld → slurmd communication is direct (Slurm auth, not K8s API). Jobs that were already running continue.

The priority during SUNK operator downtime: fix the operator. Don't restart slurmctld unless it is itself failing — doing so forces all jobs to requeue.

GPU best practices

Clock locking

# Inference — lock graphics clock for latency stability
# H100 SXM max: 1980 MHz graphics, 2619 MHz memory
nvidia-smi -lgc 1980,1980   # min,max — locks to max, prevents boost variance

# Training — leave clocks unlocked for boost
nvidia-smi --reset-gpu-clocks

Why lock for inference: when serving latency is P99-bound, a GPU that downclocks between requests (due to temperature, power budget, idle states) introduces tail latency. -lgc with min=max eliminates that variance at the cost of slightly higher idle power.

Why leave unlocked for training: training throughput is MFU-bound. The GPU sustains high utilization throughout; clocks stay near max naturally. Locking to max during low-memory-bandwidth phases (optimizer step) burns power without benefit.

Set this in prolog based on partition:

if [[ "${SLURM_JOB_PARTITION}" == "inference" ]]; then
    nvidia-smi -lgc 1980,1980
else
    nvidia-smi --reset-gpu-clocks
fi

MIG configuration for inference partitions

MIG (Multi-Instance GPU) slices an H100 into isolated compute and memory partitions. Use it when inference workloads are small (single-model, low batch size) and you want to pack more tenants onto one GPU.

Valid H100 MIG profiles:

Choose:

• 1g.10gb: packing small embedding models, rerankers, ~7B models that fit in 10 GB at FP8. 7 instances per GPU = 56 per node.
• 3g.40gb: mid-size models (13B–34B at FP8/FP16). 2 per GPU. Good balance if model fits and throughput is adequate.
• 7g.80gb: full GPU, no partitioning. When the model needs the full 80 GB or when single-request latency is the goal.

# Enable MIG on all GPUs in the node
nvidia-smi -mig 1

# Create 3g.40gb profiles on all 8 GPUs
for i in $(seq 0 7); do
    nvidia-smi mig -cgi 9,9 -C -i ${i}   # 9 = 3g.40gb profile ID
done

# Verify
nvidia-smi mig -lgip
nvidia-smi mig -lgi

In gres.conf with MIG: each MIG instance becomes a separate /dev/nvidia-caps/nvidia-cap* device. Update gres.conf accordingly and restart slurmd.

MIG and SUNK: the NVIDIA GPU Operator's MIG Manager handles MIG configuration declaratively via a MIGStrategy configmap. It will reconfigure the GPU when the configmap changes. Do not mix manual nvidia-smi mig commands with the MIG Manager — they will fight.

CUDA MPS for sub-GPU sharing

CUDA Multi-Process Service (MPS) allows multiple processes to share a single GPU with lower context-switch overhead than time-slicing. Use it when inference requests are short (< 1 ms per kernel) and latency-bound, and when MIG is not configured.

Why MPS conflicts with MIG: MIG creates hardware-isolated partitions; MPS shares the full GPU in software. They cannot both be active on the same physical GPU.

# Start MPS server (run as root on the compute node before jobs start)
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
nvidia-smi -pm 1
nvidia-cuda-mps-control -d

# Stop
echo quit | nvidia-cuda-mps-control

In Slurm: launch MPS as part of a job prolog or as a system service scoped to the inference partition. Scope MPS to specific GPUs using CUDA_VISIBLE_DEVICES. MPS is CUDA-process-scoped, not cgroup-scoped — it does not enforce memory limits between tenants. Do not use MPS on a multi-tenant cluster unless tenants are trusted; one process can observe another's kernel launch patterns.

ECC

# Check current ECC state
nvidia-smi --query-gpu=ecc.mode.current --format=csv,noheader

# Enable (requires reboot or device reset)
nvidia-smi --ecc-config=1 -i 0,1,2,3,4,5,6,7

# Disable
nvidia-smi --ecc-config=0 -i 0,1,2,3,4,5,6,7

Training nodes: leave ECC on. A bit flip in the middle of a 7-day training run corrupts the model silently. The compute overhead of ECC is ~2-3% on H100. The operational cost of a corrupted checkpoint 6 days in is unbounded.

Inference nodes: consider disabling if memory pressure forces model evictions. Disabling ECC frees ~1.25 GB per GPU on H100 (64 GB becomes ~65.25 GB addressable). The tradeoff: inference results become slightly less reproducible, and a DRAM error will produce wrong output rather than a corrected value. For most inference use cases, this is acceptable. For regulated workloads (medical imaging, financial models), it is not.

ECC mode change requires a GPU reset (nvidia-smi --gpu-reset or a reboot). Drain the node before changing.

NUMA binding in sbatch templates

Jobs that cross NUMA nodes pay a memory latency penalty. On a dual-socket host with 8 GPUs:

• Socket 0 (NUMA node 0): CPUs 0-63, GPUs 0-3, IB HCAs 0-1
• Socket 1 (NUMA node 1): CPUs 64-127, GPUs 4-7, IB HCAs 2-3

Binding matters most for inference workloads with tight latency budgets:

#!/bin/bash
#SBATCH --job-name=inference-svc
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32
#SBATCH --gres=gpu:h100:4
#SBATCH --partition=inference
#SBATCH --mem=256G

# Determine NUMA node based on which GPUs were assigned
GPU_IDS=$(nvidia-smi --query-gpu=index --format=csv,noheader | tr '\n' ',')
# Heuristic: if first assigned GPU is 0-3, bind NUMA 0
FIRST_GPU=$(echo $GPU_IDS | cut -d, -f1)
if [ "$FIRST_GPU" -lt 4 ]; then
    NUMA_NODE=0
    CPU_RANGE="0-63"
else
    NUMA_NODE=1
    CPU_RANGE="64-127"
fi

numactl --membind=${NUMA_NODE} --cpunodebind=${NUMA_NODE} \
    python -m vllm.entrypoints.openai.api_server \
        --model /weka/models/llama-3-70b \
        --tensor-parallel-size 4

DCGM exporter integration

# daemonset env snippet — metrics to enable
- name: DCGM_EXPORTER_COLLECTORS
  value: "/etc/dcgm-exporter/dcp-metrics-include.csv"

Critical metrics to alert on:

DCGM_FI_DEV_ROW_REMAP_PENDING is the most operationally urgent: a pending remap means a DRAM bank has shown errors and needs remapping at the next GPU reset. The GPU is still functional now, but the next training job on it may produce incorrect results. Drain, reset (nvidia-smi --gpu-reset), and verify the remap applied before returning to service.

# Check row remap status
nvidia-smi --query-remapped-rows=gpu_bus_id,remapped_due.uncorrectable,remapped_due.correctable,pending,failure --format=csv

NCCL environment defaults for SUNK

Set these in the SUNK SlurmCluster configmap or as cluster-wide environment variables pushed to job environments via slurm.conf:

# slurm.conf — propagate to all jobs
PropagateResourceLimitsExcept=MEMLOCK
LaunchParameters=send_gids

# Export defaults for all jobs
# (Set in /etc/environment on compute nodes, or in a site prologue script)

# /etc/profile.d/nccl.sh — on compute nodes
export NCCL_DEBUG=WARN              # INFO floods; WARN shows real errors
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3   # explicitly specify HCAs
export NCCL_SOCKET_IFNAME=^lo,docker0   # exclude loopback and docker bridge
export NCCL_IB_GID_INDEX=3          # RoCE v2: use GID index 3 (routable)
export NCCL_IB_TC=106               # DSCP ECN-capable transport
export NCCL_IB_TIMEOUT=23           # IB timeout: 23 = ~1.67s; default is too low for large clusters
export NCCL_IB_RETRY_CNT=7
export NCCL_NET_GDR_LEVEL=3         # Enable GPUDirect RDMA: 3 = PHB (PCIe Host Bridge)
export NCCL_TOPO_FILE=/etc/nccl/topo-h100-dgx.xml   # topology XML for accurate ring selection
export NCCL_ALGO=Ring               # Ring or Tree; Ring is default and correct for most cases
export NCCL_PROTO=Simple            # Simple/LL/LL128; Simple is most robust

Generate the topology XML once per node type:

nvidia-smi topo -x > /etc/nccl/topo-h100-dgx.xml

NCCL uses this file to build its own topology graph for ring/tree construction. Without it, NCCL probes the system at job start — adding ~30s to each job's startup on a 16-node run.

Job submission templates

Multi-node training (IB, GDR):

#!/bin/bash
#SBATCH --job-name=train-llm
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8       # 1 task per GPU
#SBATCH --gres=gpu:h100:8
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=8G
#SBATCH --partition=training
#SBATCH --time=7-00:00:00
#SBATCH --account=tenant-foo
#SBATCH --output=/weka/jobs/%j/stdout.log
#SBATCH --error=/weka/jobs/%j/stderr.log

# NCCL tuning
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
export NCCL_SOCKET_IFNAME=^lo
export NCCL_DEBUG=WARN

# Distributed launch via SRUN
srun --container-image=nvcr.io/nvidia/pytorch:25.04-py3 \
     --container-mounts=/weka/models:/models,/weka/datasets:/data \
     torchrun \
         --nnodes=${SLURM_NNODES} \
         --nproc_per_node=8 \
         --rdzv_id=${SLURM_JOB_ID} \
         --rdzv_backend=c10d \
         --rdzv_endpoint=${SLURM_NODELIST%%,*}:29500 \
         train.py \
         --model-path /models/llama-3-405b \
         --data-path /data/fineweb

Single-node inference with MPS:

#!/bin/bash
#SBATCH --job-name=inference-vllm
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:h100:8
#SBATCH --cpus-per-task=128
#SBATCH --mem=800G
#SBATCH --partition=inference
#SBATCH --time=infinite
#SBATCH --account=tenant-foo

# Start MPS for sub-GPU sharing
nvidia-cuda-mps-control -d
sleep 1

# Launch inference server
srun --container-image=vllm/vllm-openai:v0.6.0 \
     --container-mounts=/weka/models:/models \
     --container-env=CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50 \
     python -m vllm.entrypoints.openai.api_server \
         --model /models/llama-3-70b \
         --tensor-parallel-size 8 \
         --max-model-len 32768 \
         --port 8000

# Stop MPS on exit
echo quit | nvidia-cuda-mps-control

Interactive session:

# Allocate 1 GPU, 16 CPUs, 128 GB for 4 hours
srun --gres=gpu:h100:1 \
     --cpus-per-task=16 \
     --mem=128G \
     --partition=interactive \
     --time=4:00:00 \
     --account=tenant-foo \
     --pty bash

Reservations vs partitions for tenant isolation

Both let you dedicate nodes to a tenant. The operational implications differ significantly.

The practical recommendation:

• Use partitions for permanent topology separation (training nodes, inference nodes, interactive nodes).
• Use reservations for time-boxed commitments: "tenant-foo gets all 32 GPUs for 48 hours for a benchmark run."
• Avoid using reservations as a substitute for partitions — reservations that run indefinitely are a leak waiting to happen.

Common operator gotchas

Typed vs untyped GRES:

# In a mixed fleet (H100 + H200 nodes)

# This matches ANY GPU, regardless of model:
#SBATCH --gres=gpu:8

# This matches only H100 GPUs:
#SBATCH --gres=gpu:h100:8

# This matches only H200 GPUs:
#SBATCH --gres=gpu:h200:8

If your fleet has both H100 and H200 nodes and you don't use typed GRES in job submissions, jobs land on whichever GPU type is free. Training runs started on H100 that get preempted and requeued might land on H200 nodes — with different memory bandwidth and topology. The job may produce different throughput numbers, confusing the ML team.

The fix: use typed GRES in all production job templates. Set DefaultGRES=gpu:h100:0 in the partition definition to require explicit GPU type selection — a job that omits the type gets zero GPUs and fails with an informative error rather than silently landing on the wrong hardware.

Draining a node:

# Soft drain — wait for current jobs to finish before going DRAINED
scontrol update NodeName=gpu-01 State=DRAIN Reason="planned maintenance $(date -Iseconds)"

# Check what's still running
squeue -w gpu-01

# Wait for queue to clear, or hard drain:
# Hard drain — cancel all running jobs immediately, then DRAIN
scontrol update NodeName=gpu-01 State=DRAIN Reason="emergency $(date -Iseconds)"
scancel --state=R --nodelist=gpu-01

# Verify drained
sinfo -n gpu-01
# Expected: State=drained

# Return to service
scontrol update NodeName=gpu-01 State=RESUME

Soft drain is almost always correct. Hard drain should be reserved for hardware emergencies (XID errors, fabric-manager crash). Cancelling jobs without advance warning is the primary source of trust loss between operators and tenants.

Cordoning a node in K8s while keeping it in Slurm:

In SUNK, slurmd pods run as a DaemonSet. If you kubectl cordon gpu-01, the DaemonSet pod is not evicted (it was already scheduled). New K8s pods are not placed on gpu-01, but the existing slurmd pod stays, and Slurm can still schedule jobs to it.

# Cordon for K8s-native workloads, keep for Slurm
kubectl cordon gpu-01
# gpu-01 now has: SchedulingDisabled
# slurmd pod on gpu-01: still Running
# Slurm sinfo: still shows IDLE (or ALLOC when jobs are running)

# To keep K8s workloads away but also drain from Slurm:
kubectl cordon gpu-01
scontrol update NodeName=gpu-01 State=DRAIN Reason="K8s+Slurm drain"

The reverse — keeping K8s workloads away while Slurm schedules — is what the ReservationBinding taint does. The taint repels K8s-native pods (NoSchedule) but Slurm's slurmd DaemonSet pod tolerates it (the operator adds the toleration automatically).

The gap: if you taint the node in K8s without using a ReservationBinding, slurmd may lose its node registration if the DaemonSet pod is evicted by a conflicting taint. Always use ReservationBinding to manage taints on Slurm nodes.

See also

• SUNK overview — daemon topology, CRDs, common failure modes
• Slurm scheduling — backfill, fair-share, QoS
• Multi-tenant Slurm — account hierarchy, limits, isolation
• Slurm job failures — triage by exit code and state
• enroot + pyxis — container execution in detail
• Day-2 runbook — operational failure scenarios with fixes
• DCGM monitoring — full DCGM metric reference
• MIG partitioning — MIG profiles, operators, gres.conf detail
• NCCL multi-node — NCCL tuning, ring topology, debug