Tenant onboarding — end-to-end runbook

Full sequenced procedure for adding a new tenant to a multi-tenant GPU cluster: capacity reservation, Authentik group + LDAP, K8s namespace + Reservation, Slurm accounts and QoS, login pod, Weka org and PVCs, smoke test, handover.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

A new tenant landing on a shared GPU cluster touches every layer of the stack: capacity, identity, Kubernetes, Slurm, storage, networking, and the operator-to-customer handoff. Skipping a step does not produce an immediate failure — it produces a silent inconsistency that surfaces three days later when a researcher's first multi-node job hangs for reasons nobody can yet explain.

This runbook is the long-form sequence. Each step has a why, the exact commands, and the expected output. Run top to bottom. Do not parallelize across layers — the layers depend on each other and partial state is harder to debug than no state.

Validated against: RKE2 v1.30.x / SUNK / Authentik 2024.10 / Weka 4.3.x. Operator-side tooling is intentionally generic; substitute your own CRD names where they differ.

What this runbook covers

A new tenant arrives with a contract specifying:

  • A GPU count (e.g. 64 H100), a duration, and a partition type (reserved / on-demand / spot).
  • A team size (e.g. 12 named users).
  • Storage volumes (e.g. 5 TB /home, 50 TB /scratch).
  • Optional dedicated network isolation (PKey or strict NetworkPolicy).

By the end of this runbook, the tenant has: a working SSH login, a Slurm account with the agreed budget, a K8s namespace bound to a Reservation of GPU nodes, mounted home and scratch directories with quotas enforced, and a passing single-node and 2-node smoke test.

Prerequisites checklist

Before you begin, confirm:

  • Capacity request is reviewed and signed. The contract specifies node count, GPU SKU, partition type, term, and SLA.
  • Authentik is reachable and the LDAP outpost is healthy (kubectl -n auth get pods | grep ldap-outpost).
  • Slurm controller and slurmdbd are reachable from the cluster (kubectl -n slurm get pods).
  • Weka cluster has free capacity above the requested storage + 20% buffer.
  • The reservation operator and Slurm operator have RBAC to label nodes and create namespaces.
  • Tenant identifier is agreed: a short slug (lowercase, hyphenated). Used in every namespace, account, label, and DNS name. You cannot rename later without a migration.

If any of these is unclear, stop and clarify before touching anything.

Step 1 — Capacity check and reservation creation

Why this is first: the rest of the runbook configures the world to assume the nodes exist for this tenant. If the math does not add up, you want the failure here, not after you have created a Slurm account.

Confirm free, healthy capacity that matches the contracted SKU:

# Healthy GPU nodes not yet in any Reservation
kubectl get nodes -l node.kubernetes.io/instance-type=h100-8x \
    -o json | jq -r '
  .items[]
  | select(.metadata.labels."reserved.tenant" == null)
  | select(.status.conditions[] | select(.type=="Ready").status == "True")
  | .metadata.name'

# Cross-check against unhealthy or cordoned
kubectl get nodes -l node.kubernetes.io/instance-type=h100-8x \
    -o wide | grep -E 'NotReady|SchedulingDisabled'

Pick the specific node names that will form the Reservation. Record them in your change ticket — this list is the artifact you reconcile against later.

Create the Reservation:

apiVersion: scheduling.example.io/v1alpha1
kind: Reservation
metadata:
  name: tenant-foo-reservation
spec:
  tenant: tenant-foo
  nodeSelector:
    matchLabels:
      hardware-pool: gpu-h100-rack-a
  reservedNodes:
    - gpu-01
    - gpu-02
    - gpu-03
    - gpu-04
    - gpu-05
    - gpu-06
    - gpu-07
    - gpu-08
  applyTaint: true
  taintEffect: NoSchedule
  evictOnRelease: false

Apply and wait for the operator to converge:

kubectl apply -f reservation-tenant-foo.yaml

# Operator labels and taints the nodes; wait until all 8 show the label
kubectl get nodes -l reserved.tenant=tenant-foo
# NAME     STATUS   ROLES    AGE     VERSION
# gpu-01   Ready    <none>   30d     v1.30.4+rke2r1
# ...

If the count does not match reservedNodes, check operator logs:

kubectl -n reservation-operator logs deploy/reservation-operator --tail=100

Common cause: a node is already in another Reservation. The operator should refuse to label it; the Reservation status will show the conflict.

Step 2 — Authentik group, users, SSH keys, LDAP outpost

Why this is second: every downstream layer (Slurm, login pod PAM, Weka quota by user) needs the user's uidNumber / gidNumber to exist and be consistent. Identity must be in place before you create accounts that reference it.

2a. Create the tenant group in Authentik

Via UI or API. The group name should match the tenant slug exactly:

Group name:  tenant-foo
Parent:      tenants
Attributes:  { "tenant_slug": "tenant-foo" }

The group's pk (primary key) determines gidNumber via the property mapping gidNumber = group.pk + 4000. Record the resulting gidNumber — it is the primary GID for every user in this tenant. See Authentik for the property-mapping detail.

2b. Create users

For each named tenant user, create the user in Authentik and assign to tenant-foo. Required fields:

  • username (lowercase, hyphenated, no whitespace)
  • email
  • name (full name)
  • attribute ssh_keys populated with the user's SSH public key(s), one per line

API form (idempotent, scriptable):

curl -sS -X POST "https://auth.example.internal/api/v3/core/users/" \
  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "username": "alice",
    "name": "Alice Researcher",
    "email": "alice@tenant-foo.example",
    "groups": ["<tenant-foo-group-uuid>"],
    "attributes": {
      "ssh_keys": "ssh-ed25519 AAAAC3Nz... alice@laptop"
    }
  }'

Verify each user resolves through the LDAP outpost:

kubectl -n auth exec deploy/authentik-ldap-outpost-default -- \
  ldapsearch -x -H ldap://localhost:3389 \
  -D 'cn=ldap-bind,ou=users,dc=auth,dc=example,dc=internal' \
  -w "$LDAP_BIND_PASSWORD" \
  -b 'dc=auth,dc=example,dc=internal' \
  "(uid=alice)" uid uidNumber gidNumber memberOf

# uid: alice
# uidNumber: 2147
# gidNumber: 4099
# memberOf: cn=tenant-foo,ou=groups,dc=auth,dc=example,dc=internal

uidNumber and gidNumber must be non-zero and consistent across the entire tenant's users (same gidNumber, distinct uidNumbers). If gidNumber is 0 or missing, the property mapping is broken — fix that before continuing or every file the tenant writes will be owned by nobody:nogroup.

2c. LDAP outpost binding

If this is the cluster's first tenant, deploy the LDAP outpost. Otherwise the existing outpost already serves all tenants — no change required.

Confirm the login pods (we will deploy in step 5) will be able to reach the outpost:

kubectl -n auth get svc | grep ldap
# authentik-ldap-outpost   ClusterIP   10.43.x.y   <none>   3389/TCP,636/TCP

If the login pod runs in a tenant namespace with strict NetworkPolicy, ensure egress to auth/authentik-ldap-outpost:636 is allowed.

Step 3 — K8s namespace, Reservation binding, RBAC

3a. Create the namespace

kubectl create namespace tenant-foo

kubectl label namespace tenant-foo \
  tenant=tenant-foo \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

The pod-security labels are non-negotiable for multi-tenancy. A tenant that can launch privileged pods can escape into the host network namespace and read other tenants' Weka credentials.

3b. ResourceQuota and LimitRange

Bound the namespace at the API level even though the Reservation already bounds the physical capacity:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-foo-quota
  namespace: tenant-foo
spec:
  hard:
    requests.nvidia.com/gpu: "64"
    requests.cpu: "1024"
    requests.memory: "8Ti"
    limits.memory: "10Ti"
    persistentvolumeclaims: "50"
    requests.storage: "100Ti"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-foo-defaults
  namespace: tenant-foo
spec:
  limits:
    - type: Container
      default:
        cpu: "4"
        memory: "16Gi"
      defaultRequest:
        cpu: "1"
        memory: "4Gi"

3c. ReservationBinding

This is what actually makes tenant pods land on tenant nodes:

apiVersion: scheduling.example.io/v1alpha1
kind: ReservationBinding
metadata:
  name: tenant-foo-binding
  namespace: tenant-foo
spec:
  reservationName: tenant-foo-reservation
  podSelector:
    matchLabels:
      app.kubernetes.io/managed-by: tenant-foo
  injectAffinity: true
  injectToleration: true

See Reservations for the full pattern. Note the podSelector: every tenant workload must carry app.kubernetes.io/managed-by: tenant-foo. The login pod chart and the Slurm-via-SUNK chart both set this label by default; custom workloads need to set it explicitly.

3d. NetworkPolicy

Deny-by-default plus tenant-internal:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-foo-deny-all
  namespace: tenant-foo
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant-foo-allow
  namespace: tenant-foo
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: tenant-foo } }
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: slurm-control } }
  egress:
    - to:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: tenant-foo } }
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: slurm-control } }
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: auth } }
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
      ports:
        - { port: 53, protocol: UDP }
        - { port: 636, protocol: TCP }
        - { port: 443, protocol: TCP }

3e. RBAC for tenant operators (optional)

If the tenant has a designated cluster-admin contact who needs kubectl access scoped to their namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-foo-admins
  namespace: tenant-foo
subjects:
  - kind: Group
    name: tenant-foo-admins
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io

The tenant-foo-admins group is an Authentik group (separate from tenant-foo); membership flows through the OIDC groups claim into the K8s API server.

Step 4 — Slurm: account, association, QoS, partition, quotas

Why this layer comes after K8s: SUNK runs Slurm-on-Kubernetes; the Slurm control daemons live in slurm-control and need the tenant namespace + reservation in place before scheduling tenant jobs.

4a. Account hierarchy

# Top-level tenant account, with global GPU and concurrent-job caps
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i add account tenant-foo \
    Description="Tenant Foo" \
    Organization=tenant-foo \
    Cluster=production \
    Fairshare=300 \
    GrpTRES=gres/gpu=64 \
    GrpJobs=200

# Sub-account for the team (skip for single-team tenants)
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i add account ml-team-foo \
    parent=tenant-foo \
    Description="Tenant Foo ML Team" \
    Fairshare=200

GrpTRES=gres/gpu=64 is the Slurm-side mirror of the contract. Even with a 64-node Reservation, the user's account caps at 64 concurrent GPUs across the tenant.

4b. Per-user associations

For each user from Step 2:

kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i add user alice \
    Account=ml-team-foo \
    DefaultAccount=ml-team-foo \
    Fairshare=parent \
    DefaultQOS=normal \
    qos=normal,interactive \
    MaxJobs=20 \
    GrpTRESMins=gres/gpu=240000

GrpTRESMins=gres/gpu=240000 is a 240,000 GPU-minute budget per user (decays per PriorityDecayHalfLife). Adjust to the contracted total divided by user count.

4c. QoS

If the tenant gets preemption priority for production training (versus interactive debugging), define separate QoS:

# Default QoS for normal jobs
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i add qos normal-foo \
    Priority=100 \
    GrpTRES=gres/gpu=64 \
    MaxWall=24:00:00 \
    Flags=DenyOnLimit

# Interactive QoS — tighter wall but never preempted
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i add qos interactive-foo \
    Priority=200 \
    GrpTRES=gres/gpu=8 \
    MaxWall=04:00:00 \
    PreemptMode=OFF

Bind to the tenant account:

kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i modify account tenant-foo \
    set qos=normal-foo,interactive-foo \
        DefaultQOS=normal-foo

See Slurm multi-tenant for the QoS layout patterns.

4d. Partition

Partition pins jobs onto the right nodes. The partition's node list must match the Reservation:

# /etc/slurm/slurm.conf — added via the Slurm-operator's ConfigMap
PartitionName=tenant-foo \
    Nodes=gpu-[01-08] \
    AllowAccounts=tenant-foo \
    Default=NO \
    State=UP \
    MaxTime=24:00:00 \
    DefaultTime=01:00:00 \
    PreemptMode=OFF \
    OverSubscribe=EXCLUSIVE

OverSubscribe=EXCLUSIVE means a job allocates whole nodes — the right default for distributed training where co-tenancy at the GPU level is undesirable. For inference or fine-tuning where partial-node usage is fine, switch to OverSubscribe=NO.

Reload the controller:

kubectl -n slurm-control exec deploy/slurmctld -- scontrol reconfigure
kubectl -n slurm-control exec deploy/slurmctld -- sinfo -p tenant-foo

4e. Filesystem quotas

# Per-user /home quota
weka fs quota set --path /home/alice --hard 100GB --soft 90GB

# Per-tenant /scratch quota
weka fs quota set --path /scratch/tenant-foo --hard 50TB --soft 45TB

Quotas are enforced by Weka, not by Slurm. A user hitting /home quota gets ENOSPC from inside the job. Communicate the soft / hard threshold to the tenant.

Step 5 — Login pod deployment and SSH config

5a. Deploy the login pod chart

The login pod is the user-facing gate. It runs sshd, nslcd (LDAP client), the Slurm client tools, and pyxis/enroot. See login pods for the chart internals.

apiVersion: helm.example.io/v1
kind: HelmRelease
metadata:
  name: login-tenant-foo
  namespace: tenant-foo
spec:
  chart:
    name: sunk-login
    version: 1.4.x
  values:
    tenant: tenant-foo
    replicas: 2
    ldap:
      uri: "ldaps://authentik-ldap-outpost.auth.svc:636"
      baseDN: "dc=auth,dc=example,dc=internal"
      bindDN: "cn=ldap-bind,ou=users,dc=auth,dc=example,dc=internal"
      bindPasswordSecret: "ldap-bind-creds"
    slurm:
      controllerHost: "slurmctld.slurm-control.svc"
    sshKeysSecret: "tenant-foo-authorized-keys"
    homeMount:
      type: weka
      path: "/home"
      filesystem: "cluster-foo-home/tenant-foo"
    scratchMount:
      type: weka
      path: "/scratch"
      filesystem: "cluster-foo-scratch/tenant-foo"
    service:
      type: LoadBalancer
      annotations:
        external-dns.alpha.kubernetes.io/hostname: "login.tenant-foo.example.internal"

Apply and wait for the pod to become ready:

kubectl -n tenant-foo get pods -l app=login
kubectl -n tenant-foo logs login-tenant-foo-0 -c sshd --tail=20
# expected: "Server listening on 0.0.0.0 port 22."

5b. Verify identity flow on the login pod

# LDAP resolves the user
kubectl -n tenant-foo exec login-tenant-foo-0 -- getent passwd alice
# alice:x:2147:4099:Alice Researcher:/home/alice:/bin/bash

# Group resolves
kubectl -n tenant-foo exec login-tenant-foo-0 -- getent group tenant-foo
# tenant-foo:*:4099:alice,bob,carol

# SSH key landed in authorized_keys.d
kubectl -n tenant-foo exec login-tenant-foo-0 -- ls /etc/ssh/authorized_keys.d/
# alice  bob  carol

# Slurm client reaches the controller
kubectl -n tenant-foo exec login-tenant-foo-0 -- sinfo -p tenant-foo

5c. SSH config for the user

This is what you send the tenant. Sanitized template:

# ~/.ssh/config snippet for tenant-foo
Host login-foo
    HostName       login.tenant-foo.example.internal
    Port           22
    User           alice
    IdentityFile   ~/.ssh/id_ed25519
    ServerAliveInterval 30
    ServerAliveCountMax 4

The user connects with ssh login-foo, lands in /home/alice, and runs sinfo, sbatch, srun like a normal Slurm cluster.

Step 6 — Weka org assignment and PVC provisioning

6a. Org option (strict isolation)

For tenants that contractually require data-plane isolation:

weka org create tenant-foo
weka org tenant-foo user create admin-foo --role admin

weka fs create tenant-foo-home --org tenant-foo --capacity 5TB
weka fs create tenant-foo-scratch --org tenant-foo --capacity 50TB

# One-shot service-account token for the K8s CSI driver
weka org tenant-foo token create --description "tenant-foo CSI"

Record the token in a Kubernetes Secret in tenant-foo:

apiVersion: v1
kind: Secret
metadata:
  name: weka-tenant-foo-creds
  namespace: tenant-foo
type: Opaque
stringData:
  endpoint: "https://weka-api.example.internal:14000"
  org: "tenant-foo"
  token: "<one-shot-token>"

See Weka operations for org management detail.

6b. Single-org option (RBAC isolation)

For trusting-cooperating tenants on the same Weka org, skip org creation. Just provision per-tenant directories with quotas (already done in Step 4e) and let RBAC + POSIX permissions enforce.

6c. PVC provisioning via CSI

For tenant workloads (training jobs, dataset prep) that prefer a PVC interface over bind mount:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: weka-tenant-foo
provisioner: csi.weka.io
parameters:
  filesystemName: "tenant-foo-scratch"
  capacity: "10Ti"
  csi.storage.k8s.io/provisioner-secret-name: "weka-tenant-foo-creds"
  csi.storage.k8s.io/provisioner-secret-namespace: "tenant-foo"
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-scratch
  namespace: tenant-foo
spec:
  accessModes: [ReadWriteMany]
  storageClassName: weka-tenant-foo
  resources:
    requests:
      storage: 10Ti

reclaimPolicy: Retain is a safety default — a kubectl delete pvc does not destroy the data, just unbinds. Reclaim it explicitly during offboarding (see Tenant offboarding).

Step 7 — Smoke tests

Now, before handover, run the tests that prove the stack works for this tenant. If any of these fails, the tenant will hit it as their first job.

7a. Single-node sanity

From the login pod, as the tenant's user:

ssh alice@login.tenant-foo.example.internal

# Slurm sees a partition with our nodes
sinfo -p tenant-foo
# PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
# tenant-foo  up     1-00:00:00     8   idle gpu-[01-08]

# A trivial job that confirms GPU + cgroup
sbatch --partition=tenant-foo --gres=gpu:1 --time=00:05:00 --wrap '
  set -euo pipefail
  hostname
  nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
  # Expect to see exactly 1 GPU because of cgroup ConstrainDevices
  test "$(nvidia-smi -L | wc -l)" -eq 1
'
# tail the log
sleep 30
ls slurm-*.out
cat slurm-*.out
# gpu-01
# H100 80GB HBM3, 81559 MiB

If the test sees 8 GPUs instead of 1, ConstrainDevices=yes is missing in cgroup.conf — fix at the SUNK chart level, not at the per-tenant level. See multi-tenant Slurm.

7b. 2-node NCCL all-reduce

This catches: PCIe ACS still on, NCCL not picking up the IB plane, partition wrong, fabric MTU mismatch.

sbatch --partition=tenant-foo \
       --nodes=2 --ntasks-per-node=8 --gres=gpu:8 \
       --time=00:15:00 \
       --wrap '
  set -euo pipefail
  module load nccl-tests
  srun --mpi=pmix all_reduce_perf -b 8 -e 4G -f 2 -g 1
'

# After completion:
grep -E 'busbw' slurm-*.out
# expect: at 4 GB messages, busbw close to the per-tier expectation
# (e.g. ~190+ GB/s on 8x H100 + 8x 400G IB nodes)

If busbw is dramatically below tier expectation, run network validation before declaring the tenant operational.

7c. Storage write/read

sbatch --partition=tenant-foo --time=00:05:00 --wrap '
  set -euo pipefail
  cd /scratch/tenant-foo
  # Random 10 GB write
  dd if=/dev/urandom of=smoke.bin bs=1M count=10240 status=progress
  # Read it back
  dd if=smoke.bin of=/dev/null bs=1M
  # Quota visible
  df -h /scratch/tenant-foo
  rm smoke.bin
'

If df -h /scratch/tenant-foo shows the host's root filesystem instead of the Weka mount, /scratch is not actually mounted in the slurmd Pod — the SUNK NodeSet template is missing the volume mount.

Step 8 — Handover documentation

What the tenant receives:

  1. Connection sheet (sanitized template):

    Tenant: tenant-foo
Login host: login.tenant-foo.example.internal (port 22)
Users: alice, bob, carol
SSH config snippet: <see above>

Slurm partition: tenant-foo
Slurm account:    ml-team-foo
Default QoS:      normal-foo
Interactive QoS:  interactive-foo  (--qos=interactive-foo)

GPU pool:    8x H100-80GB nodes (64 GPUs total)
Wall limit:  24h normal, 4h interactive

Filesystems:
  /home/<user>      100 GB / user, backed up nightly
  /scratch/tenant-foo   50 TB shared, no backup

Container runtime: pyxis + enroot
  Example: srun --container-image=docker://pytorch/pytorch:2.4-cuda ...

  2. Pointers to the public-facing portions of the docs site:

  3. Escalation path. Email / Slack / on-call rota for issues. State explicitly the SLA on first response (e.g. 1 hour business hours, 4 hours otherwise).

  4. The capacity contract. A copy of the GrpTRES, GrpJobs, partition node list, /scratch quota — so the customer knows what they bought.

Validation: what "done" looks like

Run these as the operator, post-handover, to confirm the configuration is consistent end-to-end:

# 1. Reservation matches partition matches contract
kubectl get nodes -l reserved.tenant=tenant-foo -o name | sort
kubectl -n slurm-control exec deploy/slurmctld -- \
    scontrol show partition tenant-foo | grep -E 'Nodes|Default|MaxTime'
# Node lists match.

# 2. User identity consistent everywhere
for layer in tenant-foo/login-tenant-foo-0 slurm-control/slurmctld-0; do
    kubectl exec -n ${layer%/*} ${layer#*/} -- getent passwd alice
done
# Same uid/gid in both outputs.

# 3. Smoke tests passed
ssh alice@login.tenant-foo.example.internal -- "ls slurm-*.out"

# 4. Quota enforcement live
weka fs quota --path /home/alice
weka fs quota --path /scratch/tenant-foo

# 5. Authentik-side sanity
curl -sS -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
    "https://auth.example.internal/api/v3/core/groups/?name=tenant-foo" \
    | jq '.results[0].users_obj[].username'

Rollback

A failed onboarding leaves debris. Reverse in opposite order — Slurm associations first (cheap), then K8s objects, then Authentik users last (most disruptive to undo).

# Slurm: drop user associations and account
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i delete user alice
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i delete account ml-team-foo
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i delete account tenant-foo

# K8s: tear down the namespace and the binding
kubectl delete reservationbinding -n tenant-foo tenant-foo-binding
kubectl delete namespace tenant-foo

# Reservation: release nodes
kubectl delete reservation tenant-foo-reservation

# Weka: drop the org / quotas / filesystems if dedicated
weka fs delete tenant-foo-home --skip-confirmation
weka fs delete tenant-foo-scratch --skip-confirmation
weka org delete tenant-foo

# Authentik: deactivate users (keep records), then remove from group
curl -X PATCH ... is_active=false

If you only got partway through and the partial state is blocking a retry, the safe sequence above unwinds cleanly. Don't try to manually reconcile — delete and re-run.

What could go wrong — sidebar

These are the silent inconsistencies that cause first-week incidents. Check for them deliberately during onboarding.

gidNumber mismatch between login pod and slurmd Pod. Files written by the user during a Slurm job appear as nobody:nogroup when listed from the login pod. Cause: nslcd config drift, cached entries on one side, or different LDAP property mappings active. Fix: ensure both pods read from the same LDAP outpost with the same property mapping. Restart nslcd on both sides:

kubectl -n tenant-foo exec login-tenant-foo-0 -- pkill -HUP nslcd
kubectl -n slurm-control exec slurmd-tenant-foo-0 -- pkill -HUP nslcd

Pod is on the wrong partition / wrong nodes. Tenant submits a job; it lands on a different tenant's nodes (or platform nodes). Cause: ReservationBinding.podSelector does not match the Slurm-operator-generated NodeSet pod labels. Fix: verify the SUNK chart sets app.kubernetes.io/managed-by: tenant-foo on every pod template. If it sets a tenant-specific label key the binding does not select, scheduling falls back to whatever toleration matches.

ACS not disabled before the first NCCL job. First multi-node training run gets a third of the expected busbw. Cause: PCIe ACS is on, halving GPUDirect P2P. This is node hardening, not a tenant step — but if the node was provisioned freshly for this tenant and the hardening playbook missed a step, the tenant discovers it. Fix: run the ACS disable runbook on each tenant node before validation. Re-run the smoke test.

LDAP outpost rate-limited Authentik. First wave of users hits the login pod simultaneously; some getent calls succeed, some return nothing. Cause: nslcd default cache TTL is 600s; first lookup goes through, the request rate spikes, and the outpost or Authentik backend throttles. Fix: increase nslcd cache TTL, pre-warm by running getent passwd <user> for every user during onboarding (Step 5b's loop already does this).

/scratch looks mounted on the login pod but isn't on slurmd. Tenant runs an interactive srun and writes to /scratch; later their batch jobs see an empty directory. Cause: SUNK NodeSet template missing the Weka volumeMount that the login pod's chart includes. Fix: align the volume mounts in both charts. Smoke-test 7c catches this.

Partition MaxTime shorter than the tenant expects. First long training run gets killed at hour 24. Cause: contract specified "no wall limit" but partition default is MaxTime=1-00:00:00. Fix: set MaxTime=UNLIMITED (or longer) in the partition definition. Communicate clearly during handover what the wall limit is.

Reservation evicts pods due to evictOnRelease=true. Operator removes a node from the Reservation (e.g. for an RMA); evictOnRelease=true immediately kills running tenant jobs on that node. Fix: set evictOnRelease=false so the operator drains gracefully. Communicate before any reservation change.

LDAP bind credentials in the wrong namespace. Login pod logs ldap_simple_bind: Invalid credentials. Cause: the bindPasswordSecret referenced by the chart is in tenant-foo but the actual Secret with the LDAP bind password lives in auth. Fix: copy or reference the Secret correctly; never hard-code the bind password in chart values.

Authentik user has SSH key but authorized_keys is empty on login pod. Cause: the sync operator (CronJob or controller) that translates Authentik's ssh_keys attribute into the Kubernetes Secret hasn't run yet, or its RBAC fell behind. Fix: trigger the sync, verify the Secret content, restart the login pod to refresh the projected volume.

See also

External:

  • Slurm sacctmgr reference: schedmd.com/sacctmgr.html
  • Authentik LDAP outpost: goauthentik.io/docs/outposts/integrations/ldap
  • Kubernetes Pod Security Standards: kubernetes.io/docs/concepts/security/pod-security-standards/