Slurm scheduling: backfill, fair-share, QoS, partitions

The most common operator question after "why is this node down" is "why is this job not running". Both have the same answer shape: Slurm made a deterministic decision, you just don't have visibility into the inputs. This page is the inputs.

Two schedulers, one daemon

slurmctld runs two scheduler loops:

1. The main scheduler (SchedulerType=sched/backfill is the default and the only one you should run in production). Triggered by job submission, job completion, node state change, and a periodic timer. Looks at the highest-priority pending job and tries to start it. If it can't, stops.

2. The backfill scheduler. Same daemon, separate loop, runs every bf_interval seconds (default: 30s). Walks the queue in priority order; for each job, computes "when would this job start naturally?" Then asks "is there a lower-priority job that fits in the gap without delaying this one?" If yes, starts it.

Without backfill, a 64-GPU pending job blocks every smaller job behind it even if 16 GPUs sit idle and the small jobs would finish before the big one's reservation. Backfill is what makes Slurm's utilization numbers look good.

# slurm.conf — production-ish backfill tuning
SchedulerType=sched/backfill
SchedulerParameters=bf_interval=30,bf_window=4320,bf_resolution=600,bf_max_job_test=500,bf_max_job_user=20,bf_continue,default_queue_depth=200,partition_job_depth=50

Knobs that matter on a busy cluster:

Multi-factor priority

Pending jobs don't sit in submission order — they sit in priority order. With PriorityType=priority/multifactor (the default), priority is a weighted sum:

priority = PriorityWeightAge       * age_factor
         + PriorityWeightFairshare * fairshare_factor
         + PriorityWeightJobSize   * jobsize_factor
         + PriorityWeightPartition * partition_factor
         + PriorityWeightQOS       * qos_factor
         + PriorityWeightTRES      * Σ tres_factors
         + PriorityWeightAssoc     * association_factor
         - nice_value

Each factor is normalized to [0.0, 1.0]; the weight is an integer. Sensible starting weights for a multi-tenant GPU cluster:

# slurm.conf
PriorityType=priority/multifactor
PriorityDecayHalfLife=14-0       # 14 days; usage decays at this rate
PriorityCalcPeriod=5             # recalc every 5 minutes
PriorityFavorSmall=NO
PriorityFlags=FAIR_TREE,SMALL_RELATIVE_TO_TIME

PriorityWeightAge=10000          # ~10k after 1 day waiting (caps at PriorityMaxAge)
PriorityMaxAge=7-0               # age factor saturates at 7 days
PriorityWeightFairshare=100000   # fair-share dominates
PriorityWeightJobSize=5000       # mild bonus for big jobs (or 0 if you want neutral)
PriorityWeightPartition=20000    # interactive partition outranks batch
PriorityWeightQOS=50000          # `priority` QoS = jump the queue
PriorityWeightTRES=CPU=0,Mem=0,GRES/gpu=10000

Read these as: "fair-share is the dominant force, QoS can override it, age slowly catches up, big jobs and GPU-heavy jobs get a small bonus."

sprio -j <jobid> shows the per-factor breakdown for a pending job:

$ sprio -j 12345
   JOBID PARTITION   PRIORITY       SITE        AGE FAIRSHARE    JOBSIZE  PARTITION        QOS
   12345    gpu-h100   1452301          0       2500     94821       1980      20000      50000

If a user complains "my job is stuck behind theirs", sprio is the answer.

Fair-share

Fair-share is the most-misunderstood Slurm concept. The mental model:

• Every account has a RawShares value — its claim on the cluster.
• Every account/user has a RawUsage — accumulated TRES-seconds, decayed exponentially with half-life PriorityDecayHalfLife.
• A user's fair-share factor is roughly effective_shares / effective_usage, normalized.
• "Effective" means accounting for the tree: a child's share is bounded by its parent's, and usage rolls up.

With PriorityFlags=FAIR_TREE (recommended), the calculation walks the account tree and ranks users by (level1_score, level2_score, ...) so that account-level fairness is preserved before user-level. Without FAIR_TREE, a single power user can starve their entire tenant's siblings.

A typical multi-tenant tree:

root  (RawShares=1)
├── tenant-foo  (RawShares=300)
│   ├── ml-team   (RawShares=200)
│   │   ├── alice    (RawShares=1, parent=ml-team)
│   │   └── bob      (RawShares=1, parent=ml-team)
│   └── infra-team (RawShares=100)
└── tenant-bar  (RawShares=700)
    └── default   (RawShares=1)

Tenant-bar gets ~70% of the cluster long-run, tenant-foo ~30%. Within tenant-foo, ml-team gets twice infra-team's share. Within ml-team, alice and bob split equally.

sshare -a dumps the tree:

$ sshare -a -A tenant-foo
            Account       User  RawShares  NormShares    RawUsage  EffectvUsage  FairShare
-------------------- ---------- ---------- ----------- ----------- ------------ ----------
tenant-foo                             300    0.300000     1.2e+09     0.450000   0.354000
 ml-team                               200    0.667000     8.4e+08     0.700000
  ml-team             alice              1    0.500000     5.0e+08     0.595000   0.412000
  ml-team             bob                1    0.500000     3.4e+08     0.405000   0.589000
 infra-team                            100    0.333000     3.6e+08     0.300000
  infra-team          carol              1    1.000000     3.6e+08     1.000000   0.241000

Reading this: alice has used more than bob this decay-window, so bob's FairShare (0.589) > alice's (0.412). When the dominant priority weight is fair-share, bob's next job will start ahead of alice's all-else-equal.

When fair-share goes wrong

• One user dominates fair-share but their jobs keep failing. Slurm only counts successful TRES-seconds toward usage in some configurations. If PriorityFlags includes INCR_ONLY, a flapping job won't accumulate usage. This is rare; usually failed jobs do count.
• Usage looks frozen. slurmdbd is down or the decay loop is stuck. Check sacctmgr show stats.
• A user gets very low fair-share immediately after onboarding. New users start with 0 usage but inherit their account's normalized usage; if the account is heavily-used, the new user inherits that "debt" until their direct usage dominates.

QoS

QoS is the Slurm primitive for "this class of jobs is special". Common patterns:

# A "high" QoS that jumps the queue and has a 2x priority bonus
sacctmgr add qos high \
    Priority=10000 \
    Flags=DenyOnLimit \
    MaxWall=1-00:00:00 \
    GrpTRES=gres/gpu=64 \
    PreemptMode=requeue

# A "low" / preemptible QoS for opportunistic work
sacctmgr add qos low \
    Priority=0 \
    Flags=DenyOnLimit \
    MaxWall=4:00:00 \
    PreemptMode=cluster

# An "interactive" QoS with short walltime and per-user limits
sacctmgr add qos interactive \
    Priority=1000 \
    MaxWall=8:00:00 \
    MaxJobsPerUser=4 \
    MaxTRESPerUser=gres/gpu=8

# Allow alice to use the high QoS
sacctmgr modify user alice account=ml-team set qos+=high

The user submits with --qos=high (or it's the default for that association). Useful flags:

Preemption

For the preemption knobs to do anything, slurm.conf needs:

PreemptType=preempt/qos
PreemptMode=REQUEUE
PreemptExemptTime=00:30:00     # don't preempt jobs that have run < 30min

Then a high job arriving while a low job runs will: kick the low job (requeue it), reclaim its GPUs, start the high job.

PreemptExemptTime matters a lot: without it, a flood of high-priority short jobs can preempt long-running low jobs over and over, and a 6-hour training job restarts from checkpoint every 30 minutes. Tune it to roughly the cost of a checkpoint restore.

Reservations

Reservations carve out a subset of nodes for a specific user/account/QoS for a window of time. They are the "block this rack for the maintenance team Saturday morning" tool, and also the "guarantee tenant-foo always has 16 GPUs" tool.

# Maintenance reservation for ops team this Saturday
scontrol create reservation \
    starttime=2026-05-09T08:00 \
    duration=08:00:00 \
    nodes=gpu[01-04] \
    Users=ops-engineer1,ops-engineer2 \
    Flags=MAINT,IGNORE_JOBS \
    ReservationName=maint-rack-a

# Standing reservation for tenant-foo
scontrol create reservation \
    starttime=now \
    duration=infinite \
    nodes=gpu[01-08] \
    Accounts=tenant-foo \
    Flags=ANY_NODES,PURGE_COMP=01:00:00 \
    ReservationName=tenant-foo-standing

Useful flags:

AllowGroups= and Users=/Accounts= control who can submit. scontrol show reservation lists everything.

Partition design for multi-tenant GPU clusters

There are two schools:

One partition per tenant — gpu-tenant-foo, gpu-tenant-bar, gpu-tenant-baz. Hardware is split. Pros: simple, hard isolation, tenant-foo cannot DoS tenant-bar with a 10000-job array. Cons: rigid, fragmented capacity, no spillover when tenant-foo is idle.

One shared partition + QoS isolation — gpu-h100 for everyone, with QoS limits by association: tenant-foo users get QoS tenant-foo with GrpTRES=gres/gpu=128, etc. Pros: utilization, easy to flex. Cons: noisy-neighbor risk, fair-share has to be tuned tightly, and one tenant's prolog/epilog bug can affect another.

For most clusters: shared partition + QoS, with a separate interactive partition (smaller MaxWall, lower per-user job limits) so srun-from-login-pod doesn't fight with batch.

# slurm.conf — shared GPU partition
PartitionName=gpu-h100 \
    Nodes=gpu[01-32] \
    Default=YES \
    State=UP \
    MaxTime=7-00:00:00 \
    DefaultTime=01:00:00 \
    OverSubscribe=NO \
    AllowAccounts=ALL \
    QOS=normal \
    PriorityJobFactor=1000 \
    PriorityTier=1

# Interactive partition — small subset, shorter walltime
PartitionName=interactive \
    Nodes=gpu[01-04] \
    State=UP \
    MaxTime=08:00:00 \
    DefaultTime=01:00:00 \
    OverSubscribe=NO \
    AllowQOS=interactive \
    PriorityTier=10                # higher tier = always considered first

# Maintenance partition (drained nodes get moved here for testing)
PartitionName=maint \
    Nodes=gpu[01-32] \
    State=DOWN \
    AllowGroups=ops \
    Hidden=YES

PriorityTier is a hard sort key — partitions with higher tier are always scheduled before lower tiers, regardless of multi-factor priority. Use it to guarantee the interactive partition is never starved by batch.

Useful commands

scontrol diag is the under-loved one. The Bf section tells you if backfill is keeping up:

$ scontrol diag
...
Backfilling stats
        Total backfilled jobs (since last slurm start): 51234
        Total backfilled jobs (since last stats cycle): 312
        Total backfilled heterogeneous job components: 0
        Total cycles: 4521
        Last cycle when: Mon May 04 14:32:11 2026
        Last cycle: 1234567 (microsec)
        Max cycle:  9876543 (microsec)
        Last depth cycle: 487
        Last depth cycle (try sched): 422
        Last queue length: 1450

If Last cycle is consistently approaching bf_interval * 1000000, backfill is saturated — raise bf_interval or lower bf_max_job_test. If Last queue length is many times bf_max_job_test, you're missing jobs.

Troubleshooting

Job stuck in PD with reason Priority — a higher-priority job is ahead of it. squeue --start -j <id> shows when Slurm thinks it'll start. If the estimate is N/A or far in the future, see the next bullet.

Job stuck in PD with reason Resources — at this snapshot, no node-set satisfies the request. Check sinfo -N -l and scontrol show node for actually-IDLE nodes; check that the partition's nodes have the right GRES; check whether a reservation is blocking the time window.

Backfill is "stuck" / users complain --start shows N/A — bf_window is too small relative to walltime, or bf_max_job_test is too low. Verify with scontrol diag.

A user with high fair-share keeps losing to one with low fair-share — check sprio weight columns. PriorityWeightQOS or PriorityWeightPartition may be dominating fair-share. Also check that FAIR_TREE flag is set.

Reservation is created but jobs say ReqNodeNotAvail, Reserved — the user submitting is not in AllowGroups/Users/Accounts of the reservation. Or the reservation has MAINT flag and they didn't pass --reservation=.

Preemption isn't preempting — verify PreemptType, PreemptMode, the higher QoS has Priority greater than the lower's, and PreemptExemptTime hasn't elapsed.

scontrol reconfigure "fails" silently — some changes need a slurmctld restart (partition Nodes lists, NodeName changes, SchedulerType, PriorityType). The log will say "Cannot reconfigure X without restart". In SUNK: kubectl rollout restart deploy/slurmctld.

See also

• Slurm + SUNK intro — daemons, CRDs, data model
• Job failures — the other half of "why isn't this job running"
• Multi-tenant Slurm — accounts and QoS in production
• SUNK troubleshooting — when the K8s side breaks scheduling
• Login pods — where users hit the scheduler

External:

• Slurm scheduling configuration guide
• Multi-factor priority
• Fair-share with FAIR_TREE
• QoS reference
• Reservations