Slurm job failure modes: triage and fixes

A job didn't start, or it started and died, or it finished with exit code 1 and no output. Each has a different signature in Slurm's logs and a different first command to run. This page is the lookup table.

The job state machine

                  submission
                      │
                      ▼
                 ┌─────────┐                ┌─────────┐
                 │   PD    │── start ──▶    │    R    │── exit ──▶ ┌─────────┐
                 │ Pending │                │ Running │             │   CG    │
                 └────┬────┘                └────┬────┘             │ Compl-  │
                      │                          │                  │  eting  │
                      │                          │                  └────┬────┘
                      ▼                          ▼                       │
                 ┌─────────┐               ┌─────────┐                   ▼
                 │   CA    │               │   NF    │              ┌─────────┐
                 │ Cancel  │               │NodeFail │              │   CD    │
                 └─────────┘               └─────────┘              │Compl-   │
                                                                    │   eted  │
                                           ┌─────────┐              └─────────┘
                                           │   F     │
                                           │  FAILED │
                                           └─────────┘
                                           ┌─────────┐
                                           │  TO     │
                                           │ Timeout │
                                           └─────────┘
                                           ┌─────────┐
                                           │  OOM    │
                                           └─────────┘
                                           ┌─────────┐
                                           │ PR      │
                                           │ Preempt │
                                           └─────────┘

The compact codes you see in squeue/sacct:

Job stuck in PD: reading the Reason field

scontrol show job <id> prints a Reason= field. There are dozens; the high-frequency ones:

BadConstraints pitfalls

The user requested something no node can ever satisfy:

• --gres=gpu:8 --nodes=1 on a partition where every node has 4 GPUs.
• --constraint=h100&ib400&local-nvme where no single node has all three features.
• --cpus-per-task=64 --ntasks-per-node=8 --gres=gpu:8 on a 128-core node — that's 512 cores requested per node. Slurm catches this at submission usually, but stale slurm.conf data sometimes lets it through to PD.
• --mem=2T on nodes with RealMemory=2000000 (2000 GB ≠ 2 TB; off by ~7%). Use --mem-per-gpu= for portability.

scontrol show partition shows the partition's max nodes/CPUs/mem. Compare against the request. If still stuck: slurmctld -Dvvvv (debug) on a test instance and re-submit; the log will show the rejected node-by-node enumeration.

Job stuck in CG (Completing)

The job exited but Slurm hasn't released the resources. Causes:

1. Epilog hung. The epilog script (e.g., NVIDIA persistence-mode reset, ACS re-disable, fabric counter capture) is blocked. Slurm waits up to EpilogSlurmctldTimeout then kills it.
2. Container cleanup hung. enroot couldn't unmount the rootfs (busy file in /scratch held open by a stuck process).
3. Process stuck in D state (uninterruptible). Often a stuck NFS/Weka I/O operation. The kernel won't kill it.
4. GPU stuck. A CUDA process won't release the device — nvidia-smi shows processes: gpu busy. cgroup cleanup blocks waiting on the device handle.

UnkillableStepProgram is Slurm's escape hatch:

# slurm.conf
UnkillableStepProgram=/etc/slurm/unkillable.sh
UnkillableStepTimeout=180     # after 3 minutes, run UnkillableStepProgram

unkillable.sh typically: capture diagnostics (dmesg, ps, stuck-process stacks), then mark the node DRAIN so no new jobs land. Sample:

#!/bin/bash
# /etc/slurm/unkillable.sh
JOBID="$SLURM_JOB_ID"
NODE="$(hostname -s)"
LOGDIR="/var/log/slurm/unkillable"
mkdir -p "$LOGDIR"
{
  echo "=== unkillable triggered at $(date -u) for job $JOBID on $NODE ==="
  ps auxfwww
  cat /proc/*/stack 2>/dev/null | head -200
  nvidia-smi
  dmesg | tail -100
} > "$LOGDIR/${JOBID}-${NODE}.log" 2>&1
scontrol update NodeName="$NODE" State=DRAIN Reason="unkillable job $JOBID"

After the node is drained: page on-call, reboot the node, then scontrol update Node=<n> State=RESUME.

Job in NF (NodeFail)

A node went unhealthy mid-run. Slurm marks the job NF and (depending on JobRequeue=1) requeues it. To attribute:

# Slurm's view of the job at the moment of failure
sacct -j <id> --format=JobID,JobName,State,ExitCode,DerivedExitCode,NodeList,Reason -p

# Per-node attribution: which node killed it?
scontrol show job <id> | grep -E "(NodeList|Reason)"

# Check the slurmd log on each listed node, around the failure timestamp
kubectl logs -n slurm <slurmd-pod-on-target> --since=1h | grep -i "node fail\|down\|unresponsive"

# Host-level: dmesg, GPU state, HW errors
ssh gpu-01 -- 'dmesg -T | tail -200; nvidia-smi -q | grep -E "Xid|ECC|Fallen"'

Common root causes:

• GPU Xid error → reboot may help; persistent → RMA. See NVIDIA GPU operator for the Xid catalog.
• IB link flap → check switch logs (see IB switches L2).
• Kubernetes evicted the slurmd Pod (memory pressure on the node, kubelet OOM).
• Underlying VM/host crashed (rare on bare metal, common on virtualized).

Job FAILED with exit code: walking the layers

sacct shows two exit codes:

$ sacct -j 12345 -o JobID,JobName,State,ExitCode,DerivedExitCode -p
JobID|JobName|State|ExitCode|DerivedExitCode
12345|train|FAILED|1:0|1:0
12345.batch|batch|FAILED|1:0|1:0
12345.0|torchrun|FAILED|1:0|1:0

Format is <exit>:<signal>. 1:0 = exit 1, no signal. 0:9 = killed by SIGKILL. 0:15 = SIGTERM (often the cgroup OOM-killer or scancel).

The investigation order, layer by layer:

1. User script stdout/stderr — --output= and --error= in the job script. Look for the actual Python traceback.
2. Slurm-side per-job log — kubectl logs -n slurm slurmd-pod-on-the-node --since=1h | grep job=12345. Shows prolog/epilog output, container creation, signal exchange.
3. Pyxis log — pyxis prints to slurmd's stderr. Look for image pull failures, mount errors, container start refusal.
4. Container OCI exit — when the container's CMD exits, that's the user's exit code passed up. If the user's script did exit 1, that's where it came from.
5. Host kernel log — dmesg for OOM, NCCL fabric errors, CUDA driver messages, IB link state.

Job FAILED with OOM

Cgroup memory limit hit. Slurm marks State=OUT_OF_MEMORY (code OOM) and the dmesg on the node shows:

Memory cgroup out of memory: Killed process 1234567 (python) total-vm:...

The chain:

# Slurm's verdict
sacct -j <id> -o State,ExitCode,DerivedExitCode

# Cgroup events (cgroup v2)
ssh gpu-01 -- cat /sys/fs/cgroup/slurm/uid_5042/job_12345/memory.events
# oom 5
# oom_kill 5

# dmesg attribution
ssh gpu-01 -- dmesg -T | grep -A 30 "Killed process"

Common causes:

• User asked for --mem=200G but the model needs 250G. Tell them --mem-per-gpu= is more portable.
• Memory leak in the framework (PyTorch with grad-accumulate without zero-grad, etc.). Not your problem — refer to user.
• RealMemory= in slurm.conf is set above actual physical RAM. Slurm allocates "memory it doesn't have" and the system OOMs the slurmd cgroup itself before the user script. Verify: dmidecode -t memory | grep "Size:" vs RealMemory= in slurm.conf.

Job FAILED with NCCL errors

NCCL failures look like exit 1 with traceback containing NCCL error or unhandled cuda error. Distinguish two classes:

Get NCCL debug from the user's job (or rerun with):

export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_DEBUG_FILE=/scratch/${USER}/nccl-%h-%p.log

Then grep -E "NCCL INFO|NCCL WARN|NCCL ERROR" on the per-host log files.

To attribute mid-run failures to a specific node:

• sacct -j <id> -o NodeList
• For each node, grep its slurmd log for the job's stop time
• Cross-reference with switch port-counter polls (if you have InfiniBand telemetry into Prometheus)

The NCCL doc covers the hierarchy. From SUNK's perspective: NCCL failures look like ordinary exit-1 jobs, the diagnosis is on the network side.

Prolog / epilog issues

Slurm runs scripts at well-defined points around a job:

Common prolog tasks on a GPU cluster:

• Disable PCIe ACS (re-disable, since reboots re-enable it). See GPUDirect.
• Set NVIDIA persistence mode: nvidia-smi -pm 1.
• Reset GPU clocks to default if a previous job changed them.
• Drop page cache if the previous job filled it: sync; echo 3 > /proc/sys/vm/drop_caches.
• Verify dcgmi diag -r 1 quick health check (skip if too slow).

Common epilog tasks:

• Capture per-job IB counters into the accounting log.
• Reset MIG mode if it was changed.
• Wipe /tmp and /scratch/$USER/$SLURM_JOB_ID if the user didn't.
• Detect and report Xid errors that occurred during the job.

A failing prolog manifests as Reason=Prolog in squeue, LaunchFailed in sacct, and a slurmd log line:

slurmd: error: prolog failed for job 12345 on gpu-01: exit 127

exit 127 is "command not found" — usually a missing binary in slurmd's PATH, or a bash interpreter mismatch (#!/bin/bash on a system with only /usr/bin/bash). The slurmd container in SUNK has its own PATH; don't rely on what's in your login shell.

A failing epilog rarely fails the job (the user has already exited successfully) but can leave the node DRAINed. sinfo -R will show the reason.

To debug: run the script manually with the same env Slurm gives:

sudo SLURM_JOB_ID=test SLURM_JOB_USER=alice SLURM_JOB_GID=5042 \
     SLURM_JOB_UID=5042 /etc/slurm/prolog.sh

Job-array failures: retry only failed indices

sbatch --array=0-99 creates 100 jobs with IDs 12345_0 ... 12345_99. After a partial failure:

# Which indices failed?
sacct -j 12345 --format=JobID,State -p | awk -F'|' '$2=="FAILED" {print $1}' | \
    sed 's/.*_//' | sort -n | paste -sd,

# 7,12,33,89

# Resubmit just those
sbatch --array=7,12,33,89 train-array.sh

# Or, if the original was --array=0-99%4 (max 4 concurrent),
# preserve the throttle:
sbatch --array=7,12,33,89%4 train-array.sh

For frameworks that natively understand Slurm array IDs (SLURM_ARRAY_TASK_ID), the user script can parameterize on the index. For ML training, this is usually a hyperparameter sweep — give them the failed-index list and let them rerun.

"Job not found" after submission

Submission appears to succeed but squeue -j <id> returns nothing.

Possibilities:

1. slurmdbd connectivity — sbatch returned the jobid, slurmctld accepted it, but the asynchronous accounting write failed. Check kubectl logs -n slurm slurmctld | grep -i dbd. With AccountingStorageEnforce=safe, slurmctld will buffer; without, it just keeps going and sacct is wrong.
2. Job completed almost instantly — squeue only shows pending+running. sacct -j <id> sees historical jobs.
3. Dual-cluster federation confusion — the job ran on a different cluster. Use sacct -M <cluster> to enumerate.
4. MinJobAge purged it — slurmctld purges completed jobs from in-memory queue after MinJobAge (default 300s). Use sacct, not squeue.

To debug submission failures (where sbatch itself returns non-zero):

sbatch -vvvv myjob.sh
# or
sbatch --test-only myjob.sh    # validates without submitting

-vvvv shows the RPCs to slurmctld; useful for "Unable to contact slurm controller" diagnostics.

Diagnosis command quick reference

For the long-running historical lookup:

sacct -S 2026-05-01 -E 2026-05-04 -u alice \
      --format=JobID,JobName%30,Partition,Account,AllocNodes,AllocCPUS,AllocTRES%50,State,ExitCode,Start,End,Elapsed

Adjust column widths with %N. --format=...,AllocTRES%80 is essential for GPU jobs (otherwise the GPU count is truncated).

See also

• Slurm scheduling — why a job is in PD
• SUNK troubleshooting — when the K8s side breaks the job
• enroot + pyxis — container-side failure modes
• NCCL — multi-node communication failures
• DCGM — GPU Xid attribution

External:

• Slurm sacct reference
• Slurm job state codes
• Prolog and Epilog Guide