SUNK + Slurm-on-Kubernetes troubleshooting

When the K8s side breaks Slurm: slurmctld pod CrashLoopBackOff, slurmdbd DB issues, NodeSet not registering, prolog/epilog inside Pods, munge auth across pods, and SUNK upgrade gotchas.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

Most "Slurm is broken" incidents in a SUNK cluster are Kubernetes incidents wearing a Slurm hat. A login pod can't reach slurmctld because its Service IP changed; a node won't register because its DaemonSet Pod can't read the host's GPU device files; jobs all fail with auth errors after a Secret rotation. This page is the Pod-and-CRD-side triage.

slurmctld Pod in CrashLoopBackOff

slurmctld is the brain. CrashLoopBackOff means the brain won't even start. Order of likely causes:

1. `StateSaveLocation` mismatch / unavailable

slurmctld writes its in-memory state (job queue, node states) to StateSaveLocation. This is a PVC, mounted RWX, shared between active and backup slurmctld Pods. On startup it reads the prior state.

If the PVC isn't mounted, or the directory is on a stale Weka mount, or the new Pod can't write to it (permissions changed):

slurmctld: error: Cannot create directory /var/spool/slurmctld: Read-only file system
slurmctld: error: Unable to recover state, exiting

Triage:

kubectl logs -n slurm deploy/slurmctld -c slurmctld --previous | tail -50

# Check the PVC binds
kubectl get pvc -n slurm slurmctld-state
# STATUS should be Bound

# Mount inside (debugging container)
kubectl debug -n slurm deploy/slurmctld -it --image=busybox -- ls -la /var/spool/slurmctld/

Fixes:

PVC pending: storage class problem (Weka CSI, NFS provisioner). See Weka troubleshooting.
Mount stale: scale slurmctld to 0, scale back to 1, kubelet remounts.
Permissions wrong: chown slurm:slurm /var/spool/slurmctld — though the SUNK image's entrypoint should do this. If it doesn't, the underlying volume's Group ownership doesn't match fsGroup in the Pod spec.

2. `slurm.conf` parse error

A bad slurm.conf (typically pushed via the SUNK ConfigMap) makes slurmctld exit before listening:

slurmctld: fatal: Invalid SchedulerType: sched/foobar
slurmctld: fatal: Could not parse PartitionName line at /etc/slurm/slurm.conf:42

This is your most common cause. Triage:

kubectl logs -n slurm deploy/slurmctld -c slurmctld --previous | grep -E "fatal|error"

# Diff against last-known-good
kubectl get configmap -n slurm slurm-conf -o jsonpath='{.data.slurm\.conf}' > /tmp/now.conf
git diff /path/to/last-known/slurm.conf /tmp/now.conf

# Validate syntax outside the cluster
slurmctld -C -f /tmp/now.conf       # parses, exits

Fix: revert the ConfigMap change (the SlurmConfig CRD or whatever pushed it), then restart slurmctld.

A subtle case: a NodeName= line refers to a NodeSet that was deleted. slurmctld won't start because the partition references a non-existent node. Either remove the partition reference or recreate the NodeSet.

3. slurmdbd unreachable at startup

If AccountingStorageEnforce=associations,limits,qos,safe and AccountingStorageType=accounting_storage/slurmdbd, slurmctld refuses to start when slurmdbd is down. Log:

slurmctld: error: slurmdbd: Sending PersistInit msg: Connection refused
slurmctld: error: Could not connect to slurmdbd. Trying again in 5 seconds.
slurmctld: fatal: Unable to load assoc_mgr state, exit

Triage:

# Is slurmdbd up?
kubectl get pods -n slurm -l app=slurmdbd
# Status: Running, Ready: 1/1?

# Can slurmctld reach it?
kubectl exec -n slurm deploy/slurmctld -c slurmctld -- nc -zv slurmdbd 6819

# slurmdbd's own log
kubectl logs -n slurm deploy/slurmdbd

4. munge socket missing or mismatched

slurmctld connects to munged (sidecar or shared socket) and requires the same munge.key as the rest of the cluster:

slurmctld: fatal: Munge encode failed: Invalid credential

Fix in the next section.

slurmdbd Pod failures

slurmdbd is a thin layer over MariaDB. Most failures are DB-side.

Database connection refused

slurmdbd: error: mysql_real_connect failed: Can't connect to MySQL server on 'mariadb' (111)

Triage:

# Is MariaDB up?
kubectl get pods -n slurm -l app=mariadb
kubectl logs -n slurm sts/mariadb

# Can slurmdbd reach it?
kubectl exec -n slurm deploy/slurmdbd -- nc -zv mariadb 3306

# Disk full on the DB volume?
kubectl exec -n slurm sts/mariadb -- df -h /var/lib/mysql

Common: PVC for MariaDB filled up. The slurmdb grows steadily — without archive runs it's unbounded. Fix: sacctmgr archive dump, then truncate old job rows.

Schema migration on restart

When you bump Slurm major version, slurmdbd migrates the schema on startup. The migration can take minutes-to-hours on large DBs. During that time, slurmdbd appears unresponsive but is not crashed:

slurmdbd: Converting old job table...
slurmdbd: 1000000 of 5000000 rows converted

If the readiness probe times out before migration completes, K8s loops the Pod and the migration restarts from scratch. Fix:

# In SUNK SlurmCluster spec or the slurmdbd Deployment override
livenessProbe:
  initialDelaySeconds: 600         # give 10 min for migration
  failureThreshold: 30
readinessProbe:
  initialDelaySeconds: 600

Better: take a DB snapshot, run the migration manually in a one-off Pod with the new image, then roll the Deployment.

Credentials drift

The slurm MySQL user's password lives in a Secret. If that Secret gets rotated but slurmdbd's Pod isn't restarted, slurmdbd keeps using the old password until reconnect — at which point it fails:

slurmdbd: error: mysql_real_connect failed: Access denied for user 'slurm'

Fix: kubectl rollout restart deploy/slurmdbd after Secret changes.

The user runs sbatch, gets a JobID back, but squeue -j <id> shows PD with reason Resources even though sinfo shows IDLE nodes. The disconnect is on the slurmd side.

slurmd Pod missing on target node

Every K8s node should have a corresponding slurmd Pod (one per node, NodeSet-driven). If the Pod is missing or unhealthy:

# All slurmd Pods, group by Ready
kubectl get pods -n slurm -l app=slurmd -o wide

# Slurm's view of the same nodes
kubectl exec -n slurm deploy/slurmctld -- sinfo -N

# Cross-reference: nodes where K8s pod is Ready but Slurm says DOWN

Common: the slurmd Pod is on a node that doesn't have GPUs (selector mismatch), so the Pod runs but registers 0 GPUs and Slurm rejects the GPU job. Or the Pod is Pending because the node has a taint the Pod doesn't tolerate.

GRES not registered

slurmd announces its GRES at startup. If gres.conf is missing, wrong, or the device files don't exist:

slurmd: error: Setting node X state to DRAIN: Low GRES count: gpu (0 < 8)

Check gres.conf inside the Pod:

kubectl exec -n slurm <slurmd-pod> -- cat /etc/slurm/gres.conf
# Name=gpu Type=h100 File=/dev/nvidia0 Cores=0-31
# ...

# Are the device files actually present and visible?
kubectl exec -n slurm <slurmd-pod> -- ls -l /dev/nvidia*

If /dev/nvidiaN is missing inside the slurmd Pod: the GPU device plugin didn't allocate them, or hostPath/CDI for /dev isn't passing through. See GPU Operator.

If gres.conf is missing: the SUNK config sync didn't propagate. kubectl rollout restart the NodeSet's slurmd Pods.

Partition `STATE=DOWN`

Less common but easy to miss. scontrol show partition shows STATE=DOWN:

PartitionName=gpu-h100
   ...
   State=DOWN

Reason: typically a config error, sometimes manual scontrol update Partition=X State=DOWN for maintenance. Fix:

scontrol update PartitionName=gpu-h100 State=UP

`sinfo -R` review

The single most useful one-liner when triaging "jobs stuck":

kubectl exec -n slurm deploy/slurmctld -- sinfo -R
# REASON                       USER      TIMESTAMP           NODELIST
# kernel: GPU Xid              ops       2026-05-04T10:23   gpu-03
# Low GRES count               slurm     2026-05-04T11:01   gpu-07
# unkillable job 12345         slurm     2026-05-04T09:14   gpu-12

Each line is a node out of the IDLE pool. Fix the node, scontrol update Node=X State=RESUME, capacity returns.

NodeSet not registering nodes

The NodeSet is created, the Pods are running, but sinfo doesn't show the nodes. This is the SUNK syncer's job — it observes NodeSet reconciliation and updates Slurm via REST API or by re-rendering the ConfigMap.

Taint mismatch

The K8s nodes have taints (typically reserved=tenant-foo:NoSchedule) that the slurmd Pod doesn't tolerate. The Pods stay Pending:

kubectl get pods -n slurm -l app=slurmd -o wide | grep Pending
kubectl describe pod -n slurm <pending-pod> | grep -A 5 Events
# 0/16 nodes are available: 16 had taint {reserved: tenant-foo}, that the pod didn't tolerate.

Fix: add toleration to the NodeSet template. See reservations.

Reservation/ReservationBinding misalignment

If your tenant uses CoreWeave-style Reservations, the slurmd Pod must match the ReservationBinding's pod selector to get its nodeAffinity injected:

kubectl get reservationbinding -n slurm
# Should select app=slurmd or similar

kubectl get pods -n slurm -l app=slurmd -o yaml | grep -A 10 nodeAffinity
# Should show reserved.tenant: tenant-foo

If nodeAffinity is missing, the admission webhook didn't see the Pod (selector mismatch). Fix the ReservationBinding to match the slurmd Pod labels.

Kubelet device plugin not exposing GPUs

slurmd Pod is Running but Slurm sees 0 GPUs:

kubectl describe node gpu-01 | grep -E "nvidia.com|allocatable"
# nvidia.com/gpu: 8     <-- this should be 8

kubectl exec -n slurm <slurmd-pod-on-gpu-01> -- ls /dev/nvidia*
# /dev/nvidia0 ... /dev/nvidia7   <-- this should be all 8

If the node has 8 in Allocatable but the slurmd Pod only sees fewer device files: another Pod is consuming GPUs (kubectl get pods --all-namespaces -o jsonpath to find consumers). The device plugin allocates GPUs to one Pod at a time; a slurmd that wants all 8 needs nvidia.com/gpu: 8 in its resource request and no other Pod holding any.

If Allocatable shows 0: the GPU device plugin DaemonSet isn't running on this node, or NVIDIA driver isn't loaded. See GPU Operator.

"Invalid account" errors

sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified

Two causes:

The user/account doesn't exist in slurmdbd. Run sacctmgr show user alice and sacctmgr show association where user=alice. Add via multi-tenant onboarding.
The account exists but isn't allowed in the partition. scontrol show partition gpu-h100 | grep AllowAccounts. Add the account or set AllowAccounts=ALL.

Subtle: in SUNK with declarative SlurmClusterAccount CRDs, the CRD has been applied but the operator hasn't reconciled it yet (or hit an error). Check the CRD's status:

kubectl get slurmclusteraccount -n slurm tenant-foo -o yaml | yq '.status'

Prolog/epilog failures specific to SUNK

Prologs and epilogs run inside the slurmd Pod's container, not on the host. This changes their behavior in ways that surprise people coming from bare-metal Slurm.

PATH is the slurmd Pod's PATH

Hooks that call nvidia-smi need /usr/bin/nvidia-smi to exist inside the slurmd container. The SUNK slurmd image bundles it via the NVIDIA Container Toolkit; if you're using a custom image, you need to handle that yourself.

kubectl exec -n slurm <slurmd-pod> -- which nvidia-smi
# /usr/bin/nvidia-smi   <-- if missing, prolog with `nvidia-smi -pm 1` fails

root vs slurm user

Prolog= runs as root within the slurmd pod. TaskProlog= runs as the user. If your prolog uses setpci to disable ACS, root inside the container is not the same as root on the host — the container's user namespace remaps. Without privileged mode or specific capabilities, the prolog can't touch /sys/bus/pci.

The SUNK slurmd Pod spec typically requests:

securityContext:
  privileged: true                   # needed for cgroup, devices, NVML
  capabilities:
    add: [SYS_ADMIN, NET_ADMIN, SYS_RESOURCE, IPC_LOCK]
hostPID: true                        # see all processes
hostNetwork: true                    # for IB / NCCL

If privileged: false, your ACS-disable prolog will silently fail (or noisily, with Operation not permitted).

Common prolog/epilog patterns

#!/bin/bash
# /etc/slurm/prolog.sh — runs inside slurmd Pod context
set -e
JOBID="${SLURM_JOB_ID}"
USER="${SLURM_JOB_USER}"
NODE="$(hostname -s)"

# Disable PCIe ACS (re-disabled on every job since reboots re-enable it)
for dev in $(lspci -d 1000: | awk '{print $1}'); do
    setpci -s "$dev" ECAP_ACS+0x6.w=0
done

# Persistent mode + clocks
nvidia-smi -pm 1

# Ensure scratch dir exists
mkdir -p "/scratch/${USER}/${JOBID}"
chown "${USER}" "/scratch/${USER}/${JOBID}"

# Drop pagecache from previous job
sync; echo 3 > /proc/sys/vm/drop_caches

exit 0

#!/bin/bash
# /etc/slurm/epilog.sh
JOBID="${SLURM_JOB_ID}"
USER="${SLURM_JOB_USER}"

# Capture Xid errors that happened during the job
dmesg -T | grep "Xid" | tail -50 > "/var/log/slurm/xid-${JOBID}.log"

# Reset MIG mode if changed
nvidia-smi --reset-applications-clocks

# Cleanup scratch (user may have left junk)
rm -rf "/scratch/${USER}/${JOBID}"

exit 0

Failures show up in slurmd logs and in sinfo -R:

kubectl logs -n slurm <slurmd-pod> | grep -iE "prolog|epilog"
# slurmd: error: prolog failed for job 12345 on gpu-01: exit 127
# slurmd: error: epilog failed for job 12346 on gpu-02: exit 1, draining node

munge auth issues

munge requires three things to work across pods: the same munge.key, clocks within ~5 minutes of each other, and a reachable munged socket.

Symptoms

slurmctld: error: Munge decode failed: Rewound credential
# (clock skew: source clock is ahead of dest clock)

slurmctld: error: Munge decode failed: Invalid credential
# (key mismatch)

slurmctld: error: Connection refused: /var/run/munge/munge.socket.2
# (munged not running or socket not mounted)

Triage

# Same key in every Pod?
for pod in $(kubectl get pods -n slurm -l 'app in (slurmctld,slurmdbd,slurmd,login)' -o name); do
    echo "$pod"
    kubectl exec -n slurm "$pod" -- md5sum /etc/munge/munge.key
done
# All hashes should be identical.

# Clock skew
for pod in $(kubectl get pods -n slurm -l 'app in (slurmctld,slurmd)' -o name | head -10); do
    echo "$pod $(kubectl exec -n slurm "$pod" -- date -u)"
done
# All within seconds of each other.

# munged running?
kubectl exec -n slurm deploy/slurmctld -c munge -- pgrep munged

Fixes

Key mismatch: ensure every Pod mounts the same munge.key Secret. After a rotation, kubectl rollout restart every Slurm Deployment + the slurmd DaemonSet/NodeSet.
Clock skew: ensure all nodes run NTP. K8s typically inherits host clocks, so this is a host-OS problem. chronyc sources -v on the host.
Socket missing: munge runs as a sidecar; check the Pod has a munge container running and the volume /var/run/munge is shared between munge and slurmctld containers.

SUNK upgrade gotchas

Bumping Slurm via SUNK (spec.controller.image: ...:24.05.3 to :24.05.4) is normally a rolling upgrade. The cases that go wrong:

Major-version bumps need slurmdbd schema migration

23.x → 24.x changes the DB schema. The first slurmdbd on the new version runs the migration. As covered above, this can take a long time on large DBs and the readiness probe needs a long enough initialDelaySeconds.

Always snapshot the DB before a major upgrade.

kubectl exec -n slurm sts/mariadb -- \
    mysqldump --single-transaction --quick --routines slurm_acct_db | \
    gzip > slurm_acct_db_pre_upgrade.sql.gz

Version skew between slurmctld and slurmd

Slurm tolerates +/- 2 minor versions between slurmctld and slurmd, but only one major. During a rolling upgrade you'll briefly have slurmctld 24.05 talking to slurmd 23.11 — that's fine. slurmctld 24.05 talking to slurmd 22.05 — not fine, RPCs reject.

Strategy: upgrade slurmctld + slurmdbd first, then roll the slurmd NodeSet.

State migration

StateSaveLocation on the new slurmctld reads state written by the old one. Generally backward-compatible within a major. If the on-disk format changed, slurmctld emits a one-time conversion message:

slurmctld: notice: Converting state from 23.11.7 to 24.05.3

If conversion fails: roll back to the old image, slurmctld will read its own state, and you investigate. Don't rm -rf /var/spool/slurmctld in panic — you'd lose every running job.

slurm.conf incompatibilities

24.x removed deprecated options. If your slurm.conf has them (e.g., FastSchedule=, CacheGroups=), the new slurmctld refuses to start. Check release notes; clean the conf in advance.

# Before upgrading, validate the conf against the new image
docker run --rm -v $(pwd)/slurm.conf:/etc/slurm/slurm.conf \
    ghcr.io/coreweave/sunk-slurmctld:24.05.3 \
    slurmctld -C

Daily operational quick reference

# Cluster health one-liner
kubectl get pods -n slurm
kubectl exec -n slurm deploy/slurmctld -- sinfo -R         # any drained nodes?
kubectl exec -n slurm deploy/slurmctld -- scontrol diag    # scheduler health
kubectl exec -n slurm deploy/slurmctld -- sacctmgr show stats  # slurmdbd queue depth

# Drain a node for hardware work
kubectl exec -n slurm deploy/slurmctld -- scontrol update NodeName=gpu-03 State=DRAIN Reason="ECC errors investigation"

# After fix
kubectl exec -n slurm deploy/slurmctld -- scontrol update NodeName=gpu-03 State=RESUME

# Force re-read of slurm.conf (not all changes pick up — see slurm-sunk/scheduling)
kubectl exec -n slurm deploy/slurmctld -- scontrol reconfigure

# Hard restart slurmctld (use sparingly; pending jobs survive via StateSaveLocation)
kubectl rollout restart deployment -n slurm slurmctld

slurmctld Pod in CrashLoopBackOff

1. StateSaveLocation mismatch / unavailable

2. slurm.conf parse error

3. slurmdbd unreachable at startup

4. munge socket missing or mismatched

slurmdbd Pod failures

Database connection refused

Schema migration on restart

Credentials drift

Login Pod can submit but jobs are stuck

slurmd Pod missing on target node

GRES not registered

Partition STATE=DOWN

sinfo -R review

NodeSet not registering nodes

Taint mismatch

Reservation/ReservationBinding misalignment

Kubelet device plugin not exposing GPUs

"Invalid account" errors

Prolog/epilog failures specific to SUNK

PATH is the slurmd Pod's PATH

root vs slurm user

Common prolog/epilog patterns

munge auth issues

Symptoms

Triage

Fixes

SUNK upgrade gotchas

Major-version bumps need slurmdbd schema migration

Version skew between slurmctld and slurmd

State migration

slurm.conf incompatibilities

Daily operational quick reference

See also

1. `StateSaveLocation` mismatch / unavailable

2. `slurm.conf` parse error

Partition `STATE=DOWN`

`sinfo -R` review