Slurm job debugger

Guided flow: job IDs → sacct script → per-node Grafana links → cordon/taint commands.

Glues three steps you do every time a Slurm job fails: pull the sacct + scontrol details for each job, identify the suspect nodes from the Comment= fields, then jump to per-node dashboards or generate the kubectl cordon/taint commands needed to pull a node for triage.

1

Job IDs → diagnostic script

Paste one or more job IDs, copy the script, run it on a Slurm controller / login pod.

Enter at least one job ID
#!/bin/bash
JOBIDS=""

SUMMARY="JobID,JobName%20,User,State,ExitCode,Reason%60,Start,End,Elapsed,AllocTRES%60,ReqMem,MaxRSS,NodeList%500"
NODELIST="NodeList%500"

for JOBID in $JOBIDS; do
  echo "== Job $JOBID =="
  sacct -j "$JOBID" --format="$SUMMARY"
  echo
  for n in $(sacct -j "$JOBID" --noheader --format="$NODELIST" \
              | grep -v -E '^$|^N/A$|^None assigned$' | sort -u); do
    echo "== $n - Job $JOBID =="
    scontrol show node "$n" | grep -E 'NodeName|State|Reason|Comment'
    echo
  done
done
2

Paste output → identify nodes

Paste the script output. Node names are pulled from Comment= or NodeList= entries.

No nodes resolved — looking for Comment=… or NodeList=… lines.