Slurm job templates + job-debug script

Build a sacct + scontrol script for a list of job IDs, plus copy-ready sbatch templates.

The top section generates a bash script that loops over a list of job IDs, prints a verbose sacct summary for each, and dumps the Comment field on every node the job touched (useful for spotting recurring node-level issues across a batch of failures).

Below that are three sbatch templates we use as a starting point for new jobs — each is annotated with the knobs most likely to need tuning per cluster.

Job-debug script

Generated for 4 jobs
#!/bin/bash
JOBIDS="81663 81665 81666 81669"   # <-- edit this line only

SACCT_SUMMARY_FORMAT="JobID,JobName%20,User,State,ExitCode,Reason%60,Start,End,Elapsed,AllocTRES%60,ReqMem,MaxRSS,NodeList%500"
SACCT_NODELIST_FORMAT="NodeList%500"

for JOBID in $JOBIDS; do
  echo "== Job $JOBID =="
  echo

  sacct -j "$JOBID" --format="$SACCT_SUMMARY_FORMAT"
  echo

  for n in $(sacct -j "$JOBID" --noheader --format="$SACCT_NODELIST_FORMAT" \
              | grep -v -E '^$|^N/A$|^None assigned$' | sort -u); do
    echo "== $n - Job $JOBID =="
    scontrol show nodes "$n" | grep Comment || echo "No Comment found"
    echo
  done

  echo
done

sbatch templates

8-GPU single-node training
Pyxis container, single node, all 8 GPUs.
#!/bin/bash
#SBATCH --job-name=train-1n
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gpus=8
#SBATCH --cpus-per-task=96
#SBATCH --mem=0
#SBATCH --time=04:00:00
#SBATCH --output=logs/%x-%j.out

# Pyxis container
#SBATCH --container-image=registry.example.com/team/train:latest
#SBATCH --container-mounts=/mnt/shared:/mnt/shared
#SBATCH --container-workdir=/workspace

set -euo pipefail
nvidia-smi
torchrun --nproc_per_node=8 train.py
4-node × 8-GPU MPI / NCCL job
NCCL-friendly defaults. The IB rail filter is the most common knob to tune.
#!/bin/bash
#SBATCH --job-name=train-4n
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --mem=0
#SBATCH --time=12:00:00
#SBATCH --output=logs/%x-%j.out

#SBATCH --container-image=registry.example.com/team/train:latest
#SBATCH --container-mounts=/mnt/shared:/mnt/shared
#SBATCH --container-workdir=/workspace

set -euo pipefail

# NCCL — adjust HCA list to match your fabric (one per GPU rail).
export NCCL_DEBUG=INFO
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
export NCCL_SOCKET_IFNAME=^docker,lo,virbr,bond
export NCCL_IB_GID_INDEX=3

# Optional, only on RoCE fabrics where DSCP/PFC requires explicit class.
# export NCCL_IB_TC=106

srun --mpi=pmix python -m torch.distributed.run \
  --nproc_per_node=8 train.py
NCCL all-reduce sanity test
Run a quick all-reduce across the requested nodes — handy after a fabric change.
#!/bin/bash
#SBATCH --job-name=nccl-allreduce
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=00:15:00

#SBATCH --container-image=nvcr.io/nvidia/pytorch:24.01-py3
#SBATCH --container-mounts=/mnt/shared/nccl-tests:/nccl-tests

export NCCL_DEBUG=INFO
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7

srun --mpi=pmix \
  /nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1