Multi-tenant Slurm on shared GPU clusters

Account hierarchy, associations, QoS, cgroup-based GPU isolation, network and FS separation, PAM/LDAP auth, and the operational onboarding/offboarding flows that turn one Slurm cluster into a multi-tenant platform.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

A Slurm cluster shared by multiple paying tenants is a different operational problem from a single-team cluster. You need fairness, isolation, and a clean off-ramp when a tenant leaves. The Slurm pieces have all existed for years; the trick is wiring them up consistently and not letting drift accumulate.

This page covers the layers — accounts, associations, QoS, cgroups, network, FS, auth — and the standard onboarding/offboarding scripts.

Account hierarchy: top-level → tenant → user

Slurm accounts are hierarchical. The convention that scales:

root
├── platform                       # internal: ops, infra-eng, security
│   ├── ops
│   └── infra-eng
├── tenant-foo                     # paying customer
│   ├── ml-team-foo
│   │   ├── alice                  # users go directly under team
│   │   └── bob
│   └── data-team-foo
│       ├── carol
│       └── dave
├── tenant-bar
│   ├── default                    # single-team tenant — flatter is fine
│   │   ├── eve
│   │   └── frank
└── tenant-baz
    └── ...

Two reasons for this shape:

  1. Fair-share rolls up. With PriorityFlags=FAIR_TREE, tenant-foo's share is the sum of its subtrees, ensuring tenant-foo can't be starved by tenant-bar regardless of internal team activity.
  2. Limits roll down. A GrpTRES=gres/gpu=128 on the tenant-foo account caps the entire tenant's concurrent GPU usage, even across teams.

Create with sacctmgr:

# Create top-level tenant account with limits
sacctmgr -i add account tenant-foo \
    Description="Tenant Foo" \
    Organization=tenant-foo \
    Cluster=production \
    Fairshare=300 \
    GrpTRES=gres/gpu=128 \
    GrpJobs=200

# Sub-account
sacctmgr -i add account ml-team-foo \
    parent=tenant-foo \
    Description="Tenant Foo ML Team" \
    Fairshare=200

# User in sub-account, set defaults
sacctmgr -i add user alice \
    Account=ml-team-foo \
    DefaultAccount=ml-team-foo \
    Fairshare=parent

Fairshare=parent tells Slurm to inherit the parent's share — every user gets equal share within their team unless explicitly overridden.

Associations: per-user-per-account limits

The association table — (user, account, cluster, partition) — is where per-user limits live. After creating users, layer on QoS and limits:

# Allow alice to use the high QoS in addition to default
sacctmgr -i modify user alice \
    where Account=ml-team-foo \
    set qos=normal,high

# Set this user's default QoS so they don't have to pass --qos= every time
sacctmgr -i modify user alice \
    where Account=ml-team-foo \
    set DefaultQOS=normal

# Per-association GPU-minutes budget (a "fuel tank" that decays)
sacctmgr -i modify user alice \
    where Account=ml-team-foo \
    set GrpTRESMins=gres/gpu=10000

# Cap on concurrent jobs for this user
sacctmgr -i modify user alice \
    where Account=ml-team-foo \
    set MaxJobs=20

# Inspect
sacctmgr show association where user=alice tree -p

GrpTRESMins=gres/gpu=10000 means alice gets 10000 GPU-minutes of budget. As her jobs run, usage accumulates. With PriorityDecayHalfLife=14-0, that usage decays back over 14 days, so it's a soft cap on rate-of-spend rather than a hard cap on lifetime.

MaxJobs=20 is a hard cap on running+pending jobs. Hits show as AssocMaxJobsLimit. Useful to prevent one user from filling the queue with 5000 array tasks.

Cgroup isolation

ProctrackType=cgroup and TaskPlugin=cgroup are non-negotiable for multi-tenancy. Without them, Slurm can't actually constrain CPU/memory/GPU.

slurm.conf:

ProctrackType=proctrack/cgroup
TaskPlugin=task/affinity,task/cgroup
JobAcctGatherType=jobacct_gather/cgroup
PrologFlags=Contain                      # cgroup created at prolog, not first task

cgroup.conf:

# /etc/slurm/cgroup.conf
CgroupPlugin=autodetect

# Memory: hard limit at AllocatedMemory * AllowedRAMSpace/100
ConstrainRAMSpace=yes
AllowedRAMSpace=100
# Don't constrain swap (containers shouldn't be swapping anyway)
ConstrainSwapSpace=no
AllowedSwapSpace=0

# CPU cores: pin to allocated set
ConstrainCores=yes

# Device cgroup: hide GPUs not allocated by GRES
ConstrainDevices=yes

# K-mem (kernel memory) — leave default
ConstrainKmemSpace=no

The critical line: ConstrainDevices=yes. Without it, a user with --gres=gpu:1 can nvidia-smi (or worse, run on) all 8 GPUs on the node — they just got "billed" for 1, but they have access to 8. With ConstrainDevices=yes, the device cgroup hides the un-allocated GPUs at the kernel level.

Verify on a node with a running job:

# Cgroup v2
cat /sys/fs/cgroup/slurm/uid_5042/job_12345/devices.allow 2>/dev/null
# (cgroup v2 uses BPF instead — different inspection path)

# What does the user actually see?
sudo -u alice srun --gres=gpu:1 --pty nvidia-smi
# Should show 1 GPU, not 8

For full isolation also turn on JobAcctGatherType=jobacct_gather/cgroup so resource accounting comes from the cgroup itself rather than walking /proc.

GPU isolation specifics

Slurm + NVIDIA needs three things to align:

  1. gres.conf File= — pinpoints which /dev/nvidiaN corresponds to which GRES slot.
  2. ConstrainDevices=yes in cgroup.conf — kernel-level enforcement.
  3. NVIDIA Container Toolkit / CDI — when the container runs, the toolkit only injects the GPUs from NVIDIA_VISIBLE_DEVICES, which Slurm sets based on the GRES allocation.

If any of these is missing, you get either "user sees more GPUs than allocated" (security/billing problem) or "user sees fewer GPUs than allocated" (jobs fail).

MIG adds another dimension: each MIG instance is a separate device file, so gres.conf lists them individually:

# gres.conf for an A100 with 7 1g.10gb instances
Name=gpu Type=a100_1g_10gb File=/dev/nvidia-caps/nvidia-cap1
Name=gpu Type=a100_1g_10gb File=/dev/nvidia-caps/nvidia-cap2
# ...etc

See MIG for the layout.

Network isolation

Slurm doesn't natively manage network policy — it's all on K8s + IB. The patterns:

Per-tenant K8s NetworkPolicy — restricts pod-to-pod traffic. Tenant foo's slurmd Pods can talk to tenant-foo's login Pods and slurmctld, and that's it. Stops a misbehaving job from scanning other tenants' services.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-foo-isolation
  namespace: tenant-foo
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: tenant-foo
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: slurm-control
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: tenant-foo
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: slurm-control
    - to:                              # allow DNS
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports: [{ port: 53, protocol: UDP }]

InfiniBand partition keys (PKeys) — for tenants who need genuine fabric isolation. The IB subnet manager assigns each tenant its own PKey; their NICs are members of that PKey only. Cross-tenant IB traffic is rejected by the SM.

Setup is on the SM side (subnet manager) and the slurmd side. The slurmd Pod gets RDMA_DEV and RDMA_PKEY envvars from the network operator, and pyxis passes them into the container. Real isolation, but adds operational weight (PKey changes require SM reconfig).

Most multi-tenant setups skip PKey isolation and rely on namespace + NetworkPolicy + the trust model.

Shared filesystem

Two paths matter:

  • /home/<user> — small, durable, owned by the user, on a high-availability filesystem. Quotas per user.
  • /scratch/<user>/ (or /tenant/<tenant>/scratch) — large, ephemeral, no backup. Per-user or per-tenant quotas.

Weka and Lustre are the common backends. Quotas:

# Weka quota example
weka fs quota set --path /home/alice --hard 100GB --soft 90GB

# Per-tenant scratch quota
weka fs quota set --path /tenant/tenant-foo/scratch --hard 50TB --soft 45TB

The fragility: if /scratch mount fails on a slurmd node, jobs that write there fail with ENOSPC or EROFS-like errors. Weka stale mounts (a mount from a previous tenant org left behind, blocking the new one) need explicit auditing — see the Weka troubleshooting page.

pam_mkhomedir.so on the login pod creates $HOME on first login if missing:

session    optional     pam_mkhomedir.so skel=/etc/skel umask=0077

But pam_mkhomedir runs as root in the login pod's UID namespace; the resulting dir's owner is the user via PAM's NSS-resolved UID/GID. If LDAP is down at that moment, the directory gets owned by nobody:nogroup and the user can't write to it. Ensure LDAP is up before enabling pam_mkhomedir.

Authentication: PAM + LDAP + SSH keys

The auth flow on a login pod:

   user SSH ──▶ sshd reads authorized_keys.d/<user>
                       │ (key matched)
                       ▼
                 PAM account stage
                       │
              ┌────────┴────────┐
              ▼                 ▼
         pam_unix.so       pam_ldap.so   ──▶ Authentik LDAP outpost
       (root, slurm,      (tenants)            │
        sshd local)                           ▼
              │                          tenant-foo group?
              ▼                          gidNumber matches?
         PAM session                          │
              │                               │
              ▼                               ▼
     pam_mkhomedir.so + pam_limits.so   user resolved
              │                               │
              ▼                               ▼
        login shell ──▶ Slurm clients (sbatch, srun)

The key conventions:

  • Authentik computes gidNumber deterministically — typically gidNumber = pk + 4000 so two users in different tenants never collide. The same scheme is needed in nslcd.conf for both the login pod and the slurmd Pod, so id alice returns the same numbers everywhere. File ownership stamps with the UID/GID number, not the name; if numbers differ between pods, files appear owned by nobody.

  • SSH keys are stored in Authentik on the user object, synced into a per-tenant Kubernetes Secret (tenant-foo-authorized-keys) with one file per user, mounted at /etc/ssh/authorized_keys.d/<user> on the login pod. See login pods for the SSH config.

  • Never bake credentials into the slurmd image. Auth is via munge (system-level) and LDAP (user-level); both via mounted Secrets that can rotate.

Onboarding flow: new user

# 0. Pre-flight: verify the tenant exists in Slurm
sacctmgr show account tenant-foo                       # exists? has limits?

# 1. Create user in Authentik (UI or API)
#    - sets username, email, full name, login (uid)
#    - assigns to group: tenant-foo-users
#    - Authentik computes uidNumber = pk + 4000, gidNumber = pk + 4000
#    - user pastes their SSH public key into the user object

# 2. Wait for sync to render Kubernetes Secret
kubectl -n tenant-foo get secret tenant-foo-authorized-keys -o yaml | \
    grep -A2 alice                  # confirm key is in the Secret

# 3. Create user in Slurm DB
kubectl -n slurm exec deploy/slurmctld -- sacctmgr -i add user alice \
    Account=ml-team-foo \
    DefaultAccount=ml-team-foo \
    Fairshare=parent \
    DefaultQOS=normal \
    qos=normal \
    MaxJobs=20

# 4. Quota on shared FS
weka fs quota set --path /home/alice --hard 100GB --soft 90GB

# 5. (Optional) create $HOME ahead of first login so pam_mkhomedir doesn't race
kubectl -n tenant-foo exec login-0 -- bash -c '
    mkdir -p /home/alice
    chown alice:alice /home/alice
    chmod 700 /home/alice
    cp -r /etc/skel/. /home/alice/
    chown -R alice:alice /home/alice
'

# 6. Smoke-test as them
kubectl -n tenant-foo exec login-0 -- sudo -u alice -i bash -c '
    id
    sbatch --test-only --account=ml-team-foo --gres=gpu:1 --time=10:00 --wrap "echo ok"
    sacctmgr show association user=alice
'

# 7. Email/Slack the user with the login pod address and key fingerprint to verify

Wrap this in a script keyed off Authentik webhooks for "user added to tenant-foo-users". The whole flow should be idempotent — re-running after partial failure should converge.

Offboarding flow: user departure

The wrong way: sacctmgr delete user alice. This immediately removes them, including from the historical sacct records via cascading delete. You lose the ability to attribute their past usage.

The right way:

# 1. Block new submissions (stays as a "ghost" in the DB so sacct works)
kubectl -n slurm exec deploy/slurmctld -- sacctmgr -i modify user alice \
    where Account=ml-team-foo \
    set MaxJobs=0 GrpJobs=0

# 2. Cancel pending jobs (don't kill running ones unless asked)
kubectl -n slurm exec deploy/slurmctld -- scancel --user=alice --state=PENDING

# 3. Wait for or drain running jobs
kubectl -n slurm exec deploy/slurmctld -- squeue -u alice
# (advise the user / coordinate timing)

# 4. Disable the user in Authentik (keeps the account but blocks login)
#    - removes the SSH key from authorized_keys via sync
#    - getent passwd alice still resolves but PAM denies

# 5. After ~30 days: archive their home dir, then archive their sacct records
weka fs snapshot create --path /home/alice --name "offboard-alice-$(date +%Y%m%d)"
kubectl -n slurm exec deploy/slurmctld -- sacctmgr archive dump \
    Directory=/var/spool/slurm/archive Step=full

# 6. Only after archive: delete the slurm user record
kubectl -n slurm exec deploy/slurmctld -- sacctmgr -i delete user alice

The sacctmgr archive dump writes accounting records to a flat-file archive before deletion, so cost-attribution audits remain possible.

Tenant offboarding (whole tenant leaves)

Same pattern, scaled up:

# Block whole tenant
sacctmgr modify account tenant-foo set GrpJobs=0

# Cancel pending
scancel --account=tenant-foo --state=PENDING

# Wait for running (or `scancel --account=tenant-foo` if hard cutoff)

# Reclaim K8s resources
kubectl delete slurmcluster -n tenant-foo cluster-tenant-foo
# (operator should clean up logically; verify NodeSets, Login pods, Secrets are gone)

# Archive home, scratch, accounting
weka fs snapshot create --path /tenant/tenant-foo --name "offboard-final-$(date +%Y%m%d)"
sacctmgr archive dump Directory=/var/spool/slurm/archive

# Remove from K8s tenant labels
kubectl label nodes -l reserved.tenant=tenant-foo reserved.tenant-

# Remove tenant Reservation (see /kubernetes/reservations)
kubectl delete reservation tenant-foo-reservation

# Finally, after retention period
sacctmgr delete account tenant-foo

Order matters: block first, then cancel, then drain, then snapshot, then delete. Snapshot before delete; deletion is forever.

Common failure modes

User can SSH but sbatch returns "Invalid account" — they exist in LDAP but not in slurmdbd. Run step 3 of onboarding (sacctmgr add user).

User has wrong fair-share — usually inherited usage from a recently-active sibling. Wait one decay-half-life or override with Fairshare=N on the association.

File written by user appears as nobody:nogroup — UID/GID inconsistency between login pod and slurmd. Check getent passwd alice and id alice on both. Usually nslcd config drift or a stale cache (nscd -i passwd).

A tenant's QoS limits aren't enforced — AccountingStorageEnforce is missing the right flags. Should be AccountingStorageEnforce=associations,limits,qos,safe for multi-tenant. Without qos, QoS limits are advisory only.

A tenant escapes their cgroup — ConstrainDevices=yes is missing or ProctrackType isn't cgroup. Also check that PrologFlags=Contain is set so the cgroup is created at prolog (not at first task launch — that races).

A user spawns a srun --pty shell that survives the parent job — pam_slurm_adopt not enabled on compute nodes (see login pods) or the user used kubectl exec instead of ssh. Restrict kubectl access at the namespace RBAC level.

Onboarding script claims success but user can't log in — race between Authentik writing the Secret and kubelet projecting it onto the login pod. Wait 60s and retry, or kubectl rollout restart the login Deployment to force a refresh.

See also

External: