SUNK — Slurm on Kubernetes, what it is and how it runs

SUNK (Slurm-on-Kubernetes) is a Kubernetes operator that runs a complete Slurm cluster as workloads in your K8s cluster. The Slurm controller, accounting database, login nodes, and the slurmd worker agents all become pods. From the user's perspective, it is a normal Slurm: they ssh into a login node, run sbatch, and their job runs on GPU nodes.

The win is that you get Slurm's batch-scheduler, fair-share, accounting, and reservation semantics on top of a Kubernetes cluster you were already running for orchestration. The cost is operational: you now have two control planes that need to agree.

What SUNK deploys

Plus a couple of CRDs:

• SlurmCluster — top-level. Defines a Slurm cluster's components (controller, db, login, etc.).
• NodeSet — a homogeneous group of compute nodes (e.g., "8x H100 nodes for tenant-foo"). Replaces a Slurm partition definition.

How a job ends up running

1. User SSHes to login-0 (login pod, has sbatch client + Slurm config)
2. User runs: sbatch train.sh
3. login-0's sbatch sends the job to slurmctld
4. slurmctld picks an idle node from a NodeSet matching the job's resources
5. slurmctld signals slurmd on that node (gpu-01) to spawn the job
6. slurmd (running as a DaemonSet pod with privileged access) runs the user's
   script — usually wrapped via pyxis/enroot in a container
7. Job stdio is streamed back to login-0 (or written to the requested file)
8. slurmd reports completion → slurmctld → slurmdbd records accounting

The interesting bit is step 6: slurmd runs as a privileged DaemonSet pod, so it can nsenter into the host's PID namespace and spawn user processes that have direct access to GPUs, NICs, and host network. This is how you get bare-metal-equivalent performance from a job that was scheduled by Kubernetes.

SlurmCluster CRD example

apiVersion: slurm.coreweave.com/v1alpha1
kind: SlurmCluster
metadata:
  name: cluster-tenant-foo
  namespace: tenant-foo
spec:
  clusterName: tenant-foo
  controller:
    replicas: 1
    image: ghcr.io/coreweave/sunk-slurmctld:23.11.7
    resources:
      requests: { cpu: 2, memory: 8Gi }
  database:
    enabled: true
    storageClass: weka-csi-fast
    size: 50Gi
  login:
    replicas: 2
    image: ghcr.io/coreweave/sunk-login:23.11.7
    sshKeysSecret: tenant-foo-ssh-keys
  authConfig:
    type: ldap
    ldapHost: authentik-ldap.auth.svc.cluster.local
  prologEpilog:
    enabled: true
    configMap: tenant-foo-hooks

NodeSet CRD example

apiVersion: slurm.coreweave.com/v1alpha1
kind: NodeSet
metadata:
  name: gpu-h100-nodes
  namespace: tenant-foo
spec:
  clusterRef:
    name: cluster-tenant-foo
  partition: gpu
  nodeSelector:
    matchLabels:
      reserved.tenant: tenant-foo
      nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
  resources:
    cpu: 192
    memory: 2Ti
    gres:
      - type: gpu
        count: 8
        model: h100
  features: ["h100", "ib", "weka"]
  weight: 100

The operator translates this into Slurm config: slurm.conf gets a NodeName=gpu-h100-[01-08] CPUs=192 RealMemory=2097152 Gres=gpu:h100:8 Feature=h100,ib,weka line, and slurmd on each matching K8s node is started with that identity.

Login pods and tenant access

Tenants don't touch K8s. They SSH:

ssh tenant-user@login.tenant-foo.example.internal
$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpu*         up   infinite      8   idle gpu-h100-[01-08]

$ sbatch --gres=gpu:8 -N 4 train.sh
Submitted batch job 12345

$ squeue
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
12345       gpu    train  alice  R       0:30      4 gpu-h100-[01-04]

Login pods run an SSH server keyed off a Secret synced from Authentik (or whatever IdP). PAM is wired to the LDAP outpost so getent passwd alice resolves through the IdP. See authentik for that integration.

Common failure modes

slurmdbd crash-loops — usually MariaDB is the culprit:

• Disk full on the DB PV (/var/lib/mysql)
• Schema migration partially applied after a Slurm version bump
• DB credentials changed and slurmdbd's Secret wasn't updated

kubectl -n tenant-foo logs slurmdbd-0 --previous | tail -50
# error: mysql_real_connect failed: ...

Fix the DB or restore from snapshot. While slurmdbd is down, slurmctld will keep running but accounting writes accumulate locally and eventually back-pressure the controller.

Prolog/epilog hooks failing — silent in sinfo but visible in slurmd logs:

kubectl -n tenant-foo logs -l app=slurmd --tail=200 | grep -i "prolog\|epilog"
# slurmd: error: prolog failed for job 12345 on gpu-01: exit 127

Causes:

• Hook script not executable
• Hook script tries to run a binary that's not on the slurmd's PATH
• Hook tries to write to a path that doesn't exist on the host (slurmd runs as a privileged container but with a specific filesystem view)

Missing accounts in slurmdbd — submitting from a new user fails with:

sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

Account list is in slurmdbd, not slurmctld. Add via sacctmgr:

kubectl -n tenant-foo exec login-0 -- sacctmgr add account tenant-foo Description="tenant-foo"
kubectl -n tenant-foo exec login-0 -- sacctmgr add user alice Account=tenant-foo

slurmd shows nodes as DOWN — slurmctld can't reach a worker. Causes:

• Network: cluster CNI dropped the pod-to-pod path. kubectl exec slurmctld-0 -- nslookup gpu-01.slurmd.svc
• The slurmd pod crashed and isn't restarting (image pull error, OOM)
• Time skew: Slurm uses MUNGE for auth, which requires close clock alignment. NTP misalignment > 5 min → MUNGE rejects.

kubectl -n tenant-foo exec slurmctld-0 -- scontrol show node gpu-01
# State=DOWN, Reason=Not responding [slurm@2026-05-04T...]

scontrol update NodeName=gpu-01 State=RESUME after fixing.

Jobs schedule but pods don't start on the host — slurmd is fine, but the job is wrapped via pyxis/enroot and the container image can't be pulled. See the enroot/pyxis page — usually a credentials issue.

Day-2 ops

# Slurm-side state
kubectl -n tenant-foo exec login-0 -- sinfo -lN
kubectl -n tenant-foo exec login-0 -- squeue -A tenant-foo
kubectl -n tenant-foo exec login-0 -- sacctmgr list assoc

# K8s-side state
kubectl -n tenant-foo get slurmcluster,nodeset
kubectl -n tenant-foo get pods -l app=slurmd

# Drain a node for maintenance
kubectl -n tenant-foo exec login-0 -- scontrol update NodeName=gpu-01 State=DRAIN Reason="firmware update"

# Resume after maintenance
kubectl -n tenant-foo exec login-0 -- scontrol update NodeName=gpu-01 State=RESUME

Upgrading Slurm via SUNK

The operator handles version pins:

spec:
  controller:
    image: ghcr.io/coreweave/sunk-slurmctld:23.11.10  # bumped patch

Operator drains slurmctld → upgrades → starts → reconciles config. Workers (slurmd DaemonSet) roll one node at a time, with running jobs draining gracefully via scontrol shutdown.

Major-version bumps (23.x → 24.x) require a slurmdbd schema migration. The operator runs it automatically; back up the DB first.

When SUNK is not the right answer

• Single-tenant cluster where everyone uses kubectl. Skip Slurm entirely; use Job/Volcano/Kueue.
• Sub-second job dispatch latency is critical (Slurm scheduling cycle is ~30s by default; can be tuned but adds load).
• You want very fine-grained pod-level fair-share. Kueue is a better fit.

SUNK shines when tenants come from an HPC background and expect sbatch, when you need fair-share / QoS / reservations, and when you're already running Slurm somewhere else and want one job-submit experience across both bare-metal and K8s.

Daemon topology in detail

Architecture overview:

                       ┌────────────────────────────────────────┐
                       │           Kubernetes cluster           │
                       │                                        │
   user ── ssh ──►  ┌──┴──────┐                                 │
                    │  login  │                                 │
                    │   pod   │── srun/sbatch ──►               │
                    └─────────┘                ▼                │
                                       ┌──────────────┐         │
                                       │  slurmctld   │ Pod     │
                                       │  (active +   │         │
                                       │   backup)    │         │
                                       └──────┬───────┘         │
                                              │                 │
                              ┌───────────────┼───────────────┐ │
                              ▼               ▼               ▼ │
                       ┌────────────┐  ┌────────────┐  ┌────────┴──┐
                       │  slurmdbd  │  │   munge    │  │  slurmd   │
                       │   Pod      │  │  (sidecar  │  │  Pod per  │
                       │            │  │  per pod)  │  │  GPU node │
                       └─────┬──────┘  └────────────┘  └─────┬─────┘
                             ▼                               │
                       ┌────────────┐                        ▼
                       │  MariaDB   │              ┌──────────────┐
                       │ (StateSet) │              │ pyxis SPANK  │
                       └────────────┘              │  + enroot    │
                                                   └──────┬───────┘
                                                          ▼
                                                   OCI container
                                                   (CUDA app)

Four daemons, three CRDs, one database. Each has a different failure mode:

SUNK CRDs in depth

SlurmCluster

Cluster-scoped or namespace-scoped (operator install option). One per Slurm cluster. The operator reconciles it into:

• slurmctld Deployment (often 2 replicas — active + backup with shared StateSaveLocation PVC, RWX)
• slurmdbd Deployment (1 replica) + MariaDB StatefulSet (or external DB connection)
• Service that login pods point SLURM_CONF at
• ConfigMap with rendered slurm.conf
• Secret with munge.key

A minimal definition:

apiVersion: slurm.coreweave.com/v1alpha1
kind: SlurmCluster
metadata:
  name: cluster-tenant-foo
spec:
  clusterName: tenant-foo
  controller:
    replicas: 2                     # active + backup
    stateSaveLocation:
      pvc:
        storageClassName: weka-rwx
        size: 50Gi
  database:
    enabled: true
    storageClass: weka-rwx
    size: 100Gi
  login:
    replicas: 2
    sshKeysSecret: tenant-foo-ssh-keys
  authConfig:
    type: ldap
    ldapHost: authentik-ldap.auth.svc.cluster.local

NodeSet

The compute pool. 1:1 mapping to physical Kubernetes nodes — closer in spirit to a DaemonSet than a StatefulSet, with explicit replica count and node-selection logic.

A NodeSet says: "for every K8s node matching this selector, run a slurmd Pod with these resources, this image, these tolerations, and register it into Slurm partition gpu-h100." The operator creates Pod-per-node and a slurm-syncer updates slurm.conf (or uses Slurm's REST API) so slurmctld learns the node, its GRES, and its partition.

The 1:1 binding is critical: slurmd needs to see real GPUs (via the GPU device plugin), real IB devices (via the network operator / RDMA plugin), and real local NVMe — that means hostPath mounts, nvidia.com/gpu requests, and pinned scheduling.

Helper CRDs

• SlurmClusterAccount / SlurmClusterUser — declarative sacctmgr (account/user creation through GitOps)
• Login — login pod definition (separable from SlurmCluster so you can scale them independently)
• SlurmConfig — declarative slurm.conf fragments merged into the rendered ConfigMap

Node lifecycle: power-save and cloud-bursting

Slurm has a built-in mechanism for nodes that don't exist all the time: SuspendProgram / ResumeProgram.

# slurm.conf
SuspendProgram=/etc/slurm/suspend.sh
ResumeProgram=/etc/slurm/resume.sh
SuspendTimeout=600
ResumeTimeout=600
SuspendTime=900               # idle this long → suspend
SuspendRate=10                # max suspends per minute
ResumeRate=10
SuspendExcNodes=login[1-2]
SuspendExcParts=interactive

SuspendProgram runs when a node is idle for SuspendTime seconds. In a SUNK context, a typical implementation calls the K8s API to scale a NodeSet down (or evict the slurmd Pod), or calls a cloud API to power off the underlying VM.

ResumeProgram runs when slurmctld decides to start a job on a POWERED_DOWN node — it must bring the hardware back up and register the node as IDLE again. Failure here puts the node in DOWN~ state (powered-down + drained) and jobs queue forever.

For a multi-tenant on-prem GPU cluster, you probably don't want suspend (the GPUs are paid-for whether they idle or not). For burst scenarios — extra capacity from a cloud provider — this is the hook.

Slurm DB schema basics

Everything in Slurm accounting hangs off the same five tables. Knowing them turns "why can't this user submit?" from guesswork into a SQL query.

                ┌──────────────┐
                │   cluster    │  (one row per Slurm cluster)
                └──────┬───────┘
                       │
                ┌──────▼───────┐
                │   account    │  hierarchical: root → tenant-foo → ml-team
                └──────┬───────┘
                       │
                ┌──────▼────────────────────┐
                │       association         │  (user, account, partition)
                │   limits live here:       │  GrpTRESMins, MaxJobs, MaxWall
                └──────┬────────────────────┘
                       │
              ┌────────┴────────┐
              ▼                 ▼
        ┌─────────┐       ┌─────────┐
        │   qos   │       │  user   │
        └─────────┘       └─────────┘

                ┌──────────────┐    ┌──────────────┐
                │     job      │    │ reservation  │
                └──────────────┘    └──────────────┘

• cluster — one row per Slurm cluster. SUNK installs typically have one.
• account — hierarchical bucket for fair-share and limits. Jobs charge usage to an account.
• user — Linux user, typically synced from LDAP. Has a default account and one or more allowed accounts.
• association — (user, account, cluster, partition) tuple. Limits live here: GrpTRESMins, MaxJobs, MaxWall, Priority, Fairshare. AssocGrpJobsLimit reasons mean an association row hit a cap.
• qos — Quality of Service. Cross-cuts associations; see scheduling.

sacctmgr show association where user=alice tree walks the tree.

Direct DB access (read-only) when you need it:

kubectl exec -n slurm slurmdbd-0 -- mysql -u slurm -p slurm_acct_db -e \
  "SELECT id_assoc, user, acct, partition, grp_jobs FROM ${CLUSTER}_assoc_table WHERE user='alice';"

Never UPDATE the slurmdb directly. Always go through sacctmgr — the daemon caches associations and an out-of-band update produces drift that surfaces hours later as "limits aren't enforced" or "user can't submit".

GRES: how Slurm sees GPUs

Slurm doesn't natively know what a GPU is. You declare it as a Generic Resource (GRES) in gres.conf and slurm.conf:

# slurm.conf
GresTypes=gpu,nvme
NodeName=gpu[01-16] Gres=gpu:h100:8,nvme:1 CPUs=128 Sockets=2 \
         CoresPerSocket=64 ThreadsPerCore=1 RealMemory=2000000

# gres.conf — sits on every compute node, identifies actual hardware
Name=gpu Type=h100 File=/dev/nvidia0 Cores=0-31
Name=gpu Type=h100 File=/dev/nvidia1 Cores=0-31
Name=gpu Type=h100 File=/dev/nvidia2 Cores=32-63
Name=gpu Type=h100 File=/dev/nvidia3 Cores=32-63
Name=gpu Type=h100 File=/dev/nvidia4 Cores=64-95
Name=gpu Type=h100 File=/dev/nvidia5 Cores=64-95
Name=gpu Type=h100 File=/dev/nvidia6 Cores=96-127
Name=gpu Type=h100 File=/dev/nvidia7 Cores=96-127
Name=nvme Type=local File=/dev/nvme0n1

The Cores= field is critical — it pins each GPU to its NUMA-local CPU cores so when a user requests --gres=gpu:4 --cpus-per-task=32, Slurm picks four GPUs whose cores are all reachable. Get the topology wrong and you'll see PCIe-level latency where you expected NVLink.

Type=h100 lets users request a specific GPU class: --gres=gpu:h100:1. Useful in mixed clusters.

In SUNK the NodeName=... line is generated by the slurm-syncer based on the NodeSet spec and the K8s Node object's allocatable resources. Don't hand-edit it.

Things you must internalize before debugging

1. slurm.conf is not the source of truth at runtime. scontrol reconfigure reloads most of it, but partition definitions, node definitions, and SchedulerType require a slurmctld restart. SUNK does this by rolling the slurmctld Deployment.

2. Auth is munge, not Kubernetes. Two pods with different munge.key cannot talk, even if they're in the same namespace.

3. StateSaveLocation must be a shared volume (PVC with ReadWriteMany, typically Weka or NFS) accessible to all slurmctld replicas. Lose it and you lose every running job.

4. slurmdbd going down does not stop scheduling, but AccountingStorageEnforce=associations,limits,qos,safe blocks submission when slurmdbd is down. With AccountingStorageEnforce=safe, slurmctld buffers — beware buffer overflow. Choose carefully.

5. GPUs are GRES, not native. A pod that has 8 nvidia.com/gpu allocated by Kubernetes but where gres.conf says 4 — Slurm will only schedule 4-GPU jobs. The slurm-syncer is supposed to keep this aligned; when it doesn't, you get capacity drift.

See also

• Slurm scheduling: backfill, fair-share, QoS — how scheduling decisions are actually made
• Slurm job failure modes — triage guide
• Multi-tenant Slurm — accounts, QoS, isolation
• SUNK troubleshooting — when the K8s side breaks
• enroot + pyxis — how containers actually run
• Login pods — the user-facing surface
• Reservations — how SUNK tenants get their nodes
• Authentik — LDAP integration for Slurm PAM/auth

External:

• Slurm slurm.conf reference
• CoreWeave SUNK docs
• Slurm SLUG23: SUNK design
• SlinkyProject/slurm-operator — open-source variant