Slurm + SUNK day-2 runbook

Symptom-cause-fix runbook for Slurm/SUNK operations: slurmctld failover, dbd locks, NodeFail recovery, reservation leaks, and more.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

Production Slurm clusters fail in predictable ways. The failure modes recur across clusters, across operators, across sites. This runbook covers the ones that wake people up, in the format that's useful at 03:00: symptom first, then cause, then the commands to fix it.

Read Slurm + SUNK install first if you're not familiar with the cluster topology assumed here.

slurmctld failover — controller crashes

Symptom

Jobs stop accepting new submissions. Existing running jobs continue for a short window, then start failing. sbatch returns:

sbatch: error: Batch job submission failed: Unable to connect to Slurm daemon (slurmd) - check slurm.conf

Or in SUNK:

kubectl -n tenant-foo get pods | grep slurmctld
# slurmctld-0   0/1   CrashLoopBackOff

Cause

slurmctld is the single point of dispatch. If both primary and backup are down (crash, OOM, node eviction), no new jobs are accepted. Running jobs survive only until their SlurmdTimeout elapses without a controller heartbeat — default 300s.

Common causes:

OOM kill: slurmctld memory grows proportional to job history queue depth. At 50K+ pending jobs, RSS can exceed the pod's limits.memory.
State corruption: StateSaveLocation PVC full or filesystem error mid-write.
Munge auth failure after key rotation (see munge rotation).
In SUNK: operator pod lost leader lease; slurmctld pods stopped being reconciled, then an unrelated eviction hit both replicas.

The timer sequence below shows what happens from primary loss to backup taking over. The window between T+0 and T+120 is where jobs stop accepting new work.

ASCII timeline (terminal-friendly):

  T+0        Primary slurmctld stops responding (OOM, crash, eviction)
  │
  │          slurmd nodes stop receiving heartbeat from primary
  │          (they have SlurmdTimeout=300s before marking themselves DOWN)
  │
  T+0..120   Backup polls primary at SlurmctldTimeout interval (120s)
  │          Backup: "primary unreachable for 120s — acquiring controller role"
  │
  T+120      Backup reads StateSaveLocation (shared RWX PVC):
  │            job_state  → restores pending/running job queue
  │            node_state → restores IDLE/ALLOC/DRAIN per node
  │            resv_state → restores active reservations
  │          Backup: "Running as primary controller"
  │
  T+120+     Backup begins dispatching new jobs
  │          Existing slurmd nodes re-register within SlurmdTimeout (300s)
  │          Jobs that were running continue (slurmd held them)
  │
  T+300      Any slurmd that hasn't re-registered → node marked DOWN
             Jobs on those nodes → NodeFail state (requeue or cancel)

  ─────────────────────────────────────────────────────────────────────
  Key config (slurm.conf defaults):
    SlurmctldTimeout=120   — how long backup waits before taking over
    SlurmdTimeout=300      — how long slurmd runs without a controller
    BackupController=cp-02 — must be set or there is no failover

If BackupController= is not set, the cluster has no automatic failover. Manual recovery is Step 3 below.

Fix

Step 1: Check which replica is live

kubectl -n tenant-foo get pods -l app=slurmctld -o wide
kubectl -n tenant-foo logs slurmctld-0 --previous | tail -50
kubectl -n tenant-foo logs slurmctld-1 --previous | tail -50

Step 2: Check StateSaveLocation

kubectl -n tenant-foo exec slurmctld-0 -- df -h /var/spool/slurm/state
# If disk full: identify and purge old state files
kubectl -n tenant-foo exec slurmctld-0 -- ls -lh /var/spool/slurm/state/
# Files: job_state, node_state, part_state, resv_state — each can grow large

Step 3: Restart and watch

kubectl -n tenant-foo rollout restart deployment/slurmctld
kubectl -n tenant-foo rollout status deployment/slurmctld --timeout=120s

# Follow startup logs
kubectl -n tenant-foo logs -f slurmctld-0
# Look for: "slurmctld version ... started on cluster production"
# Look for: "Running as primary controller"
# Danger sign: "error: Recovered 12345 jobs from state save" — journal replay is normal;
#               if it hangs here, state file is corrupt (Step 4)

Step 4: State file corruption

If slurmctld loops on journal replay:

# Back up current state
kubectl -n tenant-foo exec slurmctld-0 -- \
    cp -r /var/spool/slurm/state /var/spool/slurm/state.bak.$(date +%s)

# Clear only the job state (drastic — all pending jobs are lost)
kubectl -n tenant-foo exec slurmctld-0 -- \
    rm /var/spool/slurm/state/job_state

# Restart with clean slate
kubectl -n tenant-foo rollout restart deployment/slurmctld

Clearing job_state drops all pending and running job state. Running jobs will be left abandoned on their nodes — requeue them manually or let users resubmit. Node state and partition state are preserved, so nodes come back as IDLE/ALLOC correctly after slurmd re-registers.

Step 5: Slurmdbd state recovery

If slurmctld was down long enough that slurmdbd's buffer of unwritten accounting records grew too large:

kubectl -n tenant-foo logs slurmdbd-0 | grep -E "error|warning" | tail -30
# Look for: "accounting_storage buffer full" or "dropped N messages"

Records dropped from the buffer are lost permanently. After controller recovery, run sacct -j <jobid> to verify recent job records exist. If jobs are missing, manually add accounting entries:

# Reconstruct a missing job record from job output file timestamps
sacctmgr add jobs jobid=<id> nodes=gpu-01 start=... end=... user=alice account=tenant-foo

This is unusual. The more common path is that slurmdbd's buffer held and replayed cleanly.

slurmdbd MariaDB row locks at scale

Symptom

slurmctld logs show:

slurmdbd: error: mysql_query failed: (1205) Lock wait timeout exceeded; try restarting transaction
slurmdbd: error: job update: mysql_query failed

Job submission latency increases. sacct queries hang. Eventually slurmctld starts logging:

slurmdbd: warning: Trouble communicating with slurmdbd: 6 of last 10 messages unacknowledged

Cause

At scale (>500 concurrent jobs, large step tables), slurmdbd fires many concurrent writes. The job_table and step_table in MariaDB can develop contention: a long-running SELECT in sacct holds a shared lock while a burst of INSERT/UPDATE from job completions waits. The default innodb_lock_wait_timeout (50s) triggers, and the insert fails with the error above.

Other causes:

A long-running sacct --all query by a user (or a reporting tool) scanning millions of rows.
PurgeJobAfter archival running during peak hours, which locks large table segments.
innodb_buffer_pool_size too small: dirty page flushing competes with DML.

The lock graph shows what SHOW PROCESSLIST looks like during the deadlock and which process to kill to unblock the others.

ASCII reference:

  ┌─────────────────────────────────────────────────────────────────────┐
  │  MariaDB InnoDB — job_table lock contention (simplified)            │
  │                                                                     │
  │  Process A  (sacct --all — user query, long SELECT)                 │
  │  ┌─────────────────────────────────────────────────────────┐        │
  │  │ State: "Sending data"  Time: 420s                       │        │
  │  │ Query: SELECT * FROM job_table WHERE ...                │        │
  │  │ Holds: SHARED lock on job_table rows 1..500000          │────┐   │
  │  └─────────────────────────────────────────────────────────┘    │   │
  │                                                                  │   │
  │  Process B  (slurmdbd — job completion INSERT)                   │   │
  │  ┌─────────────────────────────────────────────────────────┐    │   │
  │  │ State: "Waiting for table metadata lock"  Time: 51s     │    │   │
  │  │ Query: INSERT INTO job_table ...                        │    │   │
  │  │ Waiting for: EXCLUSIVE lock on job_table ──────────────►│◄───┘   │
  │  └─────────────────────────────────────────────────────────┘        │
  │                                                                     │
  │  Process C  (slurmdbd — step table UPDATE)                          │
  │  ┌─────────────────────────────────────────────────────────┐        │
  │  │ State: "Waiting for table metadata lock"  Time: 51s     │        │
  │  │ Query: UPDATE step_table SET ...                        │        │
  │  │ Waiting for: EXCLUSIVE lock on step_table ─────────────►│◄───────┤
  │  └─────────────────────────────────────────────────────────┘        │
  │                                      (same shared lock held by A)   │
  │                                                                     │
  │  Fix flow:                                                          │
  │    KILL QUERY <A.id>   →  Process A releases shared lock           │
  │    Process B unblocks  →  INSERT completes                         │
  │    Process C unblocks  →  UPDATE completes                         │
  │    Total recovery: < 1s after the KILL                             │
  └─────────────────────────────────────────────────────────────────────┘

Kill the eldest blocking query (Process A above) — sacct queries are read-only and re-runnable. Never kill Process B or C; they hold accounting data that will be lost if the transaction is rolled back.

Fix

Immediate: kill the blocking query

# From the MariaDB shell on the DB node
kubectl -n tenant-foo exec -it mariadb-0 -- mysql -u root -p slurm_acct_db

MariaDB [slurm_acct_db]> SHOW PROCESSLIST;
-- Find the offending query (usually state = "Waiting for table metadata lock" or long-running "Sending data")
MariaDB [slurm_acct_db]> KILL QUERY <process_id>;

Immediate: abort the blocking sacct

If a user-initiated sacct is the culprit, you'll see it in SHOW PROCESSLIST with User=slurm and a SELECT spanning the full job table. Kill it from outside:

kubectl -n tenant-foo exec login-0 -- pkill -f "sacct --all"

Prevent: tune lock timeout and buffer pool

# /etc/my.cnf.d/slurm.cnf — adjust and restart mariadb
[mysqld]
innodb_lock_wait_timeout = 900        # up from default 50
innodb_buffer_pool_size  = 45G        # 70-80% of DB RAM; if not already set
innodb_flush_log_at_trx_commit = 2    # durability trade-off: safe for slurmdbd

Prevent: schedule archival off-hours

# slurmdbd.conf
ArchiveEvents=yes
ArchiveJobs=yes
PurgeJobAfter=24months
# SUNK: configure via CronJob to run at 02:00 local time, not continuously

Verify lock rate over time

# Monitor InnoDB lock waits
kubectl -n tenant-foo exec mariadb-0 -- \
    mysql -u root -p -e "SHOW STATUS LIKE 'Innodb_row_lock_waits';"
# If this number grows continuously: contention is ongoing; investigate further

Stuck NodeFail state after kernel panic

Symptom

A compute node panicked (or had an OOM kill that corrupted kernel state). Slurm set it NodeFail. After the node rebooted and slurmd restarted, sinfo still shows:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
training     up   7-00:00:00     1   fail gpu-07

scontrol show node gpu-07 shows:

NodeName=gpu-07 ...
   NodeHostName=gpu-07 NodeAddr=gpu-07
   OS=Linux 6.8.0 ...
   State=FAIL+DRAIN Reason=low_socket_conn_count [slurm@...]
   ...

Cause

When slurmd detects that a node is unreachable (RPC timeout), slurmctld sets NodeFail. After recovery, slurmd sends a re-registration RPC, but slurmctld does not automatically clear FAIL+DRAIN — it requires operator action. This is intentional: automatic recovery after a node failure can hide hardware issues that caused the panic in the first place.

Fix

Step 1: Verify the node is actually healthy

Before returning to service, confirm the hardware that caused the panic has been investigated:

# Check dmesg for the panic reason
kubectl -n tenant-foo exec -it slurmctld-0 -- \
    ssh gpu-07 "dmesg | grep -E 'panic|oom|mce|edac|nvidia' | tail -50"

# Verify GPU health post-panic
ssh gpu-07 nvidia-smi
ssh gpu-07 dcgmi diag -r 1   # quick health check; -r 3 for full

# Verify fabric-manager is up
ssh gpu-07 systemctl status nvidia-fabricmanager

Step 2: Resume the node

scontrol update NodeName=gpu-07 State=RESUME
# or in SUNK:
kubectl -n tenant-foo exec login-0 -- scontrol update NodeName=gpu-07 State=RESUME

# Verify
sinfo -n gpu-07
# Expected: State=idle (or alloc if jobs immediately scheduled)

If the node doesn't return to idle after RESUME:

scontrol show node gpu-07
# Check Reason= field for secondary causes
# Check "SlurmdStartTime" — if recent, slurmd just re-registered
# If still DOWN: slurmd may not have re-registered yet
kubectl -n tenant-foo logs -l app=slurmd --field-selector spec.nodeName=gpu-07 | tail -30

Step 3: Handle orphaned jobs

Jobs that were on gpu-07 when it failed are in NF (NodeFail) state:

sacct -j <jobid> -o jobid,state,exitcode,nodelist
# State=NODE_FAIL

# Requeue if job is idempotent (e.g., checkpointed training)
scontrol requeue <jobid>

# Cancel if not requeued automatically
scancel <jobid>

RequeueExit= in slurm.conf can automate this: set RequeueExit=1:15 to auto-requeue on exit codes 1 and 15. Most training frameworks checkpoint and handle SIGTERM (exit 15) gracefully.

gres.conf out of sync with hardware after driver upgrade

Symptom

After a driver upgrade (nvidia-smi --version shows new version), jobs fail at start with:

srun: error: gpu-05: task 0: Exited with exit code 1
slurmd[gpu-05]: error: gres/gpu: unable to find gpu device /dev/nvidia2

Or jobs report fewer GPUs than requested:

srun --gres=gpu:8 nvidia-smi --list-gpus | wc -l
4

Cause

After a driver upgrade (particularly major versions, or when switching from datacenter drivers to runfile installs), device file paths can change. More commonly: the upgrade triggers a gres.conf validation step where slurmd can't find the listed File=/dev/nvidia* paths because the driver module isn't loaded yet.

Less obvious: gres.conf may have been copied from a reference node and has the wrong Cores= assignments after a BIOS or topology change.

Fix

Step 1: Regenerate gres.conf from the hardware

# On the affected node
nvidia-smi topo -m
ls -la /dev/nvidia[0-9]*

# Generate the correct Cores= mapping
nvidia-smi --query-gpu=index,pcie.link.gen.max,pci.bus_id --format=csv,noheader
lstopo --no-io | grep -A2 GPU  # or use lstopo-no-graphics

# Re-derive the gres.conf entries:
# For each GPU device /dev/nvidiaN, identify which CPU cores are closest
# via 'nvidia-smi topo -m' — look for PIX or PXB (PCIe crossings)

Step 2: Push the updated gres.conf

# Copy corrected gres.conf to affected node(s)
scp /etc/slurm/gres.conf.new root@gpu-05:/etc/slurm/gres.conf

# Drain the node first to avoid partial state
scontrol update NodeName=gpu-05 State=DRAIN Reason="gres.conf update"

# Restart slurmd
ssh gpu-05 systemctl restart slurmd

# Resume
scontrol update NodeName=gpu-05 State=RESUME

Step 3: Validate with a test job

srun -w gpu-05 --gres=gpu:8 nvidia-smi --list-gpus
# Must show 8 GPUs
srun -w gpu-05 --gres=gpu:8 nvidia-smi topo -m
# Verify topology is as expected

In SUNK, the slurm-syncer reconciles gres.conf from the NodeSet spec. If hardware changed after a driver upgrade, update the NodeSet GRES counts and let the syncer regenerate. Do not hand-edit /etc/slurm/gres.conf on nodes managed by SUNK unless you understand the sync loop — a subsequent reconciliation will overwrite your changes.

Multi-tenant unfair scheduling — one tenant draining others

Symptom

squeue shows all gpu-[01-16] nodes allocated to tenant-foo. Tenant-bar's jobs are in PD with reason Resources. sprio shows tenant-foo jobs have significantly higher priority:

$ sprio --account=tenant-bar | head -5
   JOBID PARTITION   PRIORITY FAIRSHARE       AGE
   55501     gpu    120345   100000      5340
$ sprio --account=tenant-foo | head -5
   JOBID PARTITION   PRIORITY FAIRSHARE       AGE
   55101     gpu    890234   800000      5340

Cause

Fair-share imbalance. Most likely:

tenant-foo's Fairshare= is much higher than tenant-bar's — they were given disproportionate priority at account setup.
tenant-foo ran very few jobs recently (low usage) while tenant-bar ran a lot — low usage inflates fair-share scores.
A QoS override is in effect: someone granted tenant-foo a QoS with PriorityFactor=10 that nobody remembered.
PriorityDecayHalfLife is too long — past usage isn't decaying fast enough, so a tenant's recent burst of jobs has locked them into low fair-share for too long.

The FairTree walk shows how normalized usage drives priority decay. Read the NormUsage column in sshare -a -l against the tree below.

ASCII reference:

  FairTree root account  (cluster total usage = 1.0 normalized)
  │
  ├── tenant-foo     Fairshare=500  RawUsage=high  NormUsage=0.80
  │   │              → FairShare score ≈ 0.20 (low: over-used)
  │   │              → Priority penalty compounds each 14-day half-life
  │   │
  │   ├── ml-team    Fairshare=300  (sub-account)
  │   └── infra-team Fairshare=200  (sub-account)
  │
  ├── tenant-bar     Fairshare=300  RawUsage=medium  NormUsage=0.15
  │   │              → FairShare score ≈ 0.85 (high: under-used)
  │   │              → Jobs queue at elevated priority
  │   │
  │   └── research   Fairshare=300
  │
  └── tenant-baz     Fairshare=200  RawUsage=low  NormUsage=0.05
                     → FairShare score ≈ 0.95 (very high: nearly idle)

  ─────────────────────────────────────────────────────────────────────
  PriorityDecayHalfLife=14-0   (14 days)

  tenant-foo ran a burst 3 days ago:
    Day 0:  NormUsage=0.80  FairShare=0.20
    Day 7:  NormUsage=0.40  FairShare=0.60  (half-life progress)
    Day 14: NormUsage=0.20  FairShare=0.80  (back near neutral)

  tenant-bar gets scheduling priority until Day 14 catches up.
  Operator levers: shorten PriorityDecayHalfLife, or sacctmgr reset RawUsage.

Reducing PriorityDecayHalfLife (e.g., to 7-0) recovers balance faster but makes burst behavior more volatile. Increasing the Fairshare= weight for tenant-bar permanently is rarely the right fix — it changes their standing relative to all other tenants, not just tenant-foo.

Fix

Step 1: Diagnose

# Show fair-share tree
sshare -a -l
# ACCOUNT         USER  RAW SHARES  RAW USAGE  NORM SHARES  NORM USAGE  FAIR SHARE
# tenant-foo      -     300         0.01       0.60         0.01        0.99
# tenant-bar      -     300         8500.2     0.60         0.85        0.41

# Check QoS assignments
sacctmgr show qos format=Name,Priority,Flags,GrpTRES

# Check partition-level priority tiers
scontrol show partition training | grep PriorityTier

Step 2: Reset normalized usage (if a historical burst is the cause)

# Reset raw usage for the affected account
sacctmgr -i modify account tenant-foo set RawUsage=0

# Force priority recalc
scontrol recalculate_priorities

This is aggressive — it gives tenant-foo a clean slate. Appropriate if a single large job distorted their fair-share score for weeks. Document the action.

Step 3: Adjust Fairshare weights

# If tenant-foo's fairshare is inflated relative to others
sacctmgr -i modify account tenant-foo set Fairshare=300
sacctmgr -i modify account tenant-bar set Fairshare=300
# Equal fairshare for equal priority tenants

Step 4: Place a temporary hold on tenant-foo's pending jobs

This gives tenant-bar a chance to catch up without cancelling tenant-foo's work:

# Hold all pending jobs for tenant-foo
scontrol requeue $(squeue -h -u tenant-foo -t PD -o "%i" | tr '\n' ',')
# or
for jobid in $(squeue -A tenant-foo -t PD -h -o "%i"); do
    scontrol hold $jobid
done

# Release after tenant-bar catches up
for jobid in $(squeue -A tenant-foo -t PD -h -o "%i"); do
    scontrol release $jobid
done

Step 5: Prevent recurrence

# slurm.conf — ensure GrpTRES caps are set per tenant
# (do this in sacctmgr, not slurm.conf — but verify it's actually enforced)

# Verify GrpTRES is set and being enforced
sacctmgr show assoc where account=tenant-foo format=Account,GrpTRES,GrpJobs
# GrpTRES=gres/gpu=128 limits tenant-foo to 128 GPUs concurrent
# If not set, add it:
sacctmgr -i modify account tenant-foo set GrpTRES=gres/gpu=128

Reservation cleanup leaks

Symptom

scontrol show reservation lists reservations that should have ended. Nodes are showing as RESERVED in sinfo even though no jobs are scheduled within them. Some reservations show future end times that look wrong.

scontrol show reservation
# ReservationName=tenant-foo-burn-in StartTime=2026-01-15T10:00:00 EndTime=infinite
# Flags=
# Nodes=gpu-[01-08] NodeCnt=8
# ...

Cause

Reservations with Duration=infinite or very long EndTime that were created manually and never cleaned up. Common scenarios:

Burn-in reservation created for hardware validation, then the hardware went into production but nobody deleted the reservation.
SUNK ReservationBinding with a dangling reference: the Slurm reservation exists but the CRD was deleted.
Automated provisioning scripts that create reservations on resource allocation but don't always clean them up on deallocation (billing system integration, for example).

Fix

Step 1: Audit

scontrol show reservation | grep -E "ReservationName|EndTime|Nodes|Users"
# List all reservations with EndTime in the past or set to "infinite"

# Identify which reservations are truly active (have a purpose)
# vs. stale (no corresponding CRD, no upcoming job, past end time)
kubectl -n tenant-foo get reservationbinding 2>/dev/null

Step 2: Delete stale reservations

# Delete a specific reservation
scontrol delete reservation tenant-foo-burn-in

# Delete all reservations with end time in the past
scontrol show reservation | awk '/ReservationName/{name=$1} /EndTime/{print name, $1}' | \
    while read name endline; do
        endtime=$(echo $endline | cut -d= -f2)
        if [[ "$(date -d "${endtime}" +%s 2>/dev/null)" -lt "$(date +%s)" ]]; then
            echo "Deleting stale: $name (ended $endtime)"
            scontrol delete ${name}
        fi
    done

Step 3: Verify nodes returned to pool

sinfo
# Nodes that were RESERVED should now show IDLE

Step 4: Prevent recurrence

Add explicit EndTime to all reservations. Never use infinite unless the reservation maps to a permanent allocation managed by a SUNK ReservationBinding CRD.

# Good: time-bounded reservation
scontrol create reservation ReservationName=tenant-foo-benchmark \
    StartTime=now Duration=48:00:00 \
    Nodes=gpu-[01-08] \
    Users=alice,bob \
    Accounts=tenant-foo \
    Flags=OVERLAP,MAINT

# Also: add a cleanup cronjob that fires daily and removes expired reservations

pyxis container cache fills disk

Symptom

Jobs start failing with:

pyxis: ERROR: Failed to import docker image: write /mnt/nvme/enroot/cache/...: no space left on device
srun: error: gpu-03: task 0: Exited with exit code 1

df -h /mnt/nvme on the affected node shows 100% utilization.

Cause

Enroot stores container images as squashfs files in ENROOT_CACHE_PATH. With a fast NVMe and many tenants using different image tags, the cache fills up. Old images are not evicted automatically — enroot has no built-in LRU eviction.

Secondary causes:

Multiple tenants pulling the same image tag concurrently, each building their own squashfs (enroot doesn't share in-progress downloads).
A very large image (NVCR PyTorch containers can be 30-50 GB) pulled multiple times on the same node under different user contexts.

Fix

Immediate: free space

# SSH to affected node
ssh gpu-03

# List cache contents by size
ls -lSh /mnt/nvme/enroot/cache/
# nvcr.io+nvidia+pytorch+25.04-py3.sqsh  48G
# nvcr.io+nvidia+pytorch+24.12-py3.sqsh  46G

# Remove old images (confirm with users first that they're not actively needed)
rm /mnt/nvme/enroot/cache/nvcr.io+nvidia+pytorch+24.12-py3.sqsh

# Remove all images older than 14 days
find /mnt/nvme/enroot/cache -name "*.sqsh" -mtime +14 -delete

After cleanup: run any queued jobs

# Requeue jobs that failed due to disk full
scontrol requeue <jobid>

Prevent: automated eviction script

#!/bin/bash
# /etc/cron.daily/enroot-cache-cleanup

CACHE_DIR=/mnt/nvme/enroot/cache
MAX_SIZE_GB=200
RETENTION_DAYS=7

current_gb=$(du -sg ${CACHE_DIR} | awk '{print $1}')
if [ "${current_gb}" -gt "${MAX_SIZE_GB}" ]; then
    echo "Cache at ${current_gb}G, evicting files older than ${RETENTION_DAYS} days"
    find ${CACHE_DIR} -name "*.sqsh" -mtime +${RETENTION_DAYS} -delete
fi

Prevent: shared Weka cache for large shared images

If multiple nodes share the same NGC base images, move those to Weka:

# Pre-stage to shared FS
enroot import --output /weka/enroot/cache/pytorch-25.04.sqsh \
    docker://nvcr.io/nvidia/pytorch:25.04-py3

# Point local cache at Weka for that tenant
# (or configure enroot.conf per-node to check Weka before local NVMe)

Symptom

Login pods become unresponsive. top shows high sshd process count. Tenants report timeouts connecting. kubectl -n tenant-foo top pod login-0 shows high CPU utilization. sshd log shows thousands of connection attempts per minute.

Cause

Two scenarios:

Automation storm: a user's CI/CD pipeline or monitoring tool is issuing many short ssh -c <command> invocations (common with Slurm polling loops, job status checkers, or badly written sbatch wrappers).
User pileup at start of day: 50 users authenticate simultaneously at 09:00, each with a heavy PAM stack (LDAP + pam_slurm_adopt).

The PAM stack on login pods is heavier than on standard SSH servers: LDAP resolution, getpwnam lookups, SSH key validation against the IdP. Each connection spawns a PAM conversation thread.

Fix

Immediate: identify the storm source

kubectl -n tenant-foo exec login-0 -- \
    ss -tn state ESTABLISHED '( dport = :22 )' | awk '{print $5}' | \
    cut -d: -f1 | sort | uniq -c | sort -rn | head -10
# Output: count of TCP connections from each source IP
# A single IP with 200+ connections = automation storm

# If automation, identify the user
kubectl -n tenant-foo exec login-0 -- \
    who | awk '{print $1}' | sort | uniq -c | sort -rn

Immediate: throttle if automation is the cause

# /etc/security/limits.d/maxlogins.conf — on login pods
*  hard  maxlogins  10
# Limits any user to 10 concurrent SSH sessions

For immediate effect without a pod restart:

kubectl -n tenant-foo exec login-0 -- \
    bash -c "echo 'MaxSessions 5' >> /etc/ssh/sshd_config && kill -HUP 1"

Prevent: connection rate limiting in sshd

# /etc/ssh/sshd_config on login pods
MaxStartups 20:30:100    # start throttling at 20 unauthenticated; hard limit at 100
LoginGraceTime 30        # abort authentication after 30s
MaxSessions 5            # per connection

Prevent: pre-authenticate LDAP lookups (nsscachd or sssd)

The PAM LDAP round-trip is the bottleneck. Use sssd with in-memory cache:

# /etc/sssd/sssd.conf
[domain/cluster]
ldap_uri = ldap://authentik-ldap.auth.svc.cluster.local
cache_credentials = True
entry_cache_timeout = 3600    # 1 hour; reduces LDAP queries by >95% for repeat logins

Prevent: scale login pods horizontally

In SUNK, the SlurmCluster spec controls login pod count. If a single login pod handles 100+ simultaneous users during peak hours, add replicas:

spec:
  login:
    replicas: 4   # up from 2

Put a LoadBalancer Service or an external HAProxy in front. kubectl -n tenant-foo get service login-lb should distribute connections. Without a LB, all SSH goes to login-0 (the first pod) — K8s Service round-robin doesn't apply to persistent SSH sessions.

Quick reference — commands by scenario

# --- State inspection ---
sinfo -lN                          # all nodes, long format
sinfo -R                           # reasons for drained/down nodes
squeue -u alice --start            # when will alice's job start?
sprio -j <jobid>                   # priority breakdown for a pending job
sshare -a -l                       # fair-share tree
sacct -j <jobid> -o all            # full job accounting record
scontrol show node gpu-07          # node details including GRES
scontrol show partition training   # partition config
scontrol show reservation          # all reservations

# --- State modification ---
scontrol update NodeName=gpu-07 State=DRAIN Reason="maintenance $(date -Iseconds)"
scontrol update NodeName=gpu-07 State=RESUME
scontrol update NodeName=gpu-07 State=DOWN Reason="hardware failure"
scontrol requeue <jobid>
scontrol hold <jobid>
scontrol release <jobid>
scancel --state=PD --account=tenant-foo    # cancel all pending jobs for tenant
scancel --nodelist=gpu-07 --state=R        # cancel all running jobs on gpu-07

# --- GRES and GPU ---
scontrol show node gpu-07 | grep -i gres   # GRES allocated vs configured
srun -w gpu-07 --gres=gpu:8 nvidia-smi --list-gpus
nvidia-smi --query-remapped-rows=gpu_bus_id,pending --format=csv

# --- SUNK-specific ---
kubectl -n tenant-foo get slurmcluster,nodeset,reservationbinding
kubectl -n tenant-foo exec slurmctld-0 -- scontrol ping   # slurmctld self-check
kubectl -n tenant-foo exec login-0 -- scontrol reconfigure  # reload slurm.conf
kubectl -n tenant-foo rollout restart deployment/slurmctld
kubectl -n tenant-foo rollout restart daemonset/slurmd

# --- slurmdbd ---
kubectl -n tenant-foo exec slurmdbd-0 -- sacct --federation --all | head -20
kubectl -n tenant-foo exec mariadb-0 -- mysqladmin -u root -p status
kubectl -n tenant-foo exec mariadb-0 -- mysql -u root -p -e "SHOW PROCESSLIST;"

# --- Enroot / pyxis ---
ls -lSh /mnt/nvme/enroot/cache/       # cache contents on a compute node
find /mnt/nvme/enroot/cache -name "*.sqsh" -mtime +14   # stale images
enroot list                            # images available to current user

slurmctld failover — controller crashes

Symptom

Cause

Fix

slurmdbd MariaDB row locks at scale

Symptom

Cause

Fix

Stuck NodeFail state after kernel panic

Symptom

Cause

Fix

gres.conf out of sync with hardware after driver upgrade

Symptom

Cause

Fix

Multi-tenant unfair scheduling — one tenant draining others

Symptom

Cause

Fix

Reservation cleanup leaks

Symptom

Cause

Fix

pyxis container cache fills disk

Symptom

Cause

Fix

Login pod SSH storm during business hours

Symptom

Cause

Fix

Quick reference — commands by scenario

See also