Tenant offboarding — end-to-end runbook

Sequenced procedure for removing a tenant from a multi-tenant GPU cluster: stop new jobs, drain running, archive accounting, snapshot then delete /home and /scratch, revoke SSH and Authentik, release reservations, return capacity to the pool. Covers compliance and data-retention considerations.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

A tenant leaves. The contract ends, they migrated to another platform, the project ran its course, or — less pleasantly — billing ran out. Whatever the reason, removal has to be done in a specific order: block first, drain next, archive third, then delete. Reverse the order and you destroy data you needed for billing reconciliation, or you race jobs against a deletion and corrupt their checkpoints.

This runbook is the mirror of tenant onboarding. Same layers, opposite direction. It also has a compliance dimension that onboarding does not: data retention, audit-log preservation, and the question of when the tenant's data is truly gone.

Validated against: RKE2 v1.30.x / SUNK / Authentik 2024.10 / Weka 4.3.x.

What this runbook covers

The tenant has a contracted termination date. By that date you must:

Stop new jobs from being submitted.
Drain or cancel running jobs on the operator's terms (not the tenant's, by this point).
Archive Slurm accounting records and any logs required by retention policy.
Snapshot user home directories and scratch.
Delete the working filesystem mounts.
Revoke SSH and Authentik access.
Tear down K8s namespace and Reservation.
Return capacity to the pool.
After the retention period, hard-delete the snapshots and Authentik user records.

Compliance considerations (data retention, GDPR right-to-erasure, contractual audit-log requirements) are interleaved throughout. The default position is archive before delete — you can always delete an archive, you cannot recover deleted live data.

Prerequisites

The contractual termination date and any agreed grace period.
A clear policy on data retention. Default below: 30 days for snapshots, 7 years for accounting records (financial audit horizon).
A handover or migration outcome on the customer side, if the customer is moving data elsewhere. Confirm before destructive steps.
Operator access to: the slurmctld pod, the Authentik admin API, the Weka cluster, kubectl on the cluster.
Change ticket open with: tenant slug, termination date, retention policy in effect, signoff names.

Step 0 — Confirm and announce

Before any action, confirm:

# Tenant exists and is accounted for
kubectl get reservation tenant-foo-reservation -o yaml | grep -E 'reservedNodes|tenant'
kubectl get namespace tenant-foo
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr show account tenant-foo

# Active state — running jobs, mounted volumes
kubectl -n slurm-control exec deploy/slurmctld -- squeue --account=tenant-foo
weka fs quota --path /home -p | grep tenant-foo
weka fs quota --path /scratch/tenant-foo

Send the customer a final notice with the retention timeline and what remains accessible (read-only login? scratch read-only for a week?). Capture confirmation.

If you do not have explicit confirmation from a customer counterparty, do not proceed past step 4 without an internal escalation signoff. Destructive offboarding without a paper trail is a future audit problem.

Step 1 — Stop new jobs

The first action is to take the tenant's submission rate to zero without disturbing in-flight work. Set Slurm-side hard caps to zero on the parent account:

kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i modify account tenant-foo \
    set GrpJobs=0 \
        GrpSubmitJobs=0

# Roll the same to sub-accounts so submitters get a clear error
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr -i modify account ml-team-foo \
    set GrpJobs=0 \
        GrpSubmitJobs=0

GrpSubmitJobs=0 rejects new submissions at submit time with AssocGrpSubmitJobsLimit. Already-pending and already-running jobs are unaffected.

Confirm:

# Try to submit as a tenant user — must fail
kubectl -n tenant-foo exec login-tenant-foo-0 -- sudo -u alice -i bash -c '
    sbatch --partition=tenant-foo --gres=gpu:1 --time=00:01:00 --wrap "echo test"
'
# expected: sbatch: error: Batch job submission failed: Job violates accounting/QOS policy

Lock the partition itself as a belt-and-braces:

kubectl -n slurm-control exec deploy/slurmctld -- scontrol update PartitionName=tenant-foo State=DOWN

State=DOWN partition reflects in sinfo and rejects job placement, even if a user finds an account that previously had quota.

Step 2 — Drain running jobs

Two paths depending on the customer agreement.

2a. Graceful drain (let jobs finish)

Allowable when there's runway between "stop submissions" and "must be off cluster". Wait for squeue to empty:

# Watch in-flight jobs
watch 'kubectl -n slurm-control exec deploy/slurmctld -- squeue --account=tenant-foo'

# Or programmatic poll
while [ "$(kubectl -n slurm-control exec deploy/slurmctld -- \
    squeue --account=tenant-foo --noheader | wc -l)" -gt 0 ]; do
    sleep 60
done

This is the safe path. The customer's checkpoints land in /scratch cleanly, no aborts, no half-written files.

2b. Hard cutoff

If contractually required (e.g. they refused to leave, billing dispute, fixed termination minute):

# Cancel pending first — non-disruptive
kubectl -n slurm-control exec deploy/slurmctld -- scancel --account=tenant-foo --state=PENDING

# Cancel running with grace period
kubectl -n slurm-control exec deploy/slurmctld -- scancel --account=tenant-foo --signal=TERM
sleep 60
kubectl -n slurm-control exec deploy/slurmctld -- scancel --account=tenant-foo --signal=KILL

# Confirm
kubectl -n slurm-control exec deploy/slurmctld -- squeue --account=tenant-foo

A scancel --signal=TERM lets the job's epilog run (often the framework's checkpoint flush). Wait at least 60s before KILL. If the job was checkpointing to /scratch, you have just preserved the customer's last hour of training — this matters.

2c. K8s-side workloads

If the tenant ran K8s-native workloads (not just Slurm):

# Scale tenant deployments to zero
kubectl -n tenant-foo scale deploy --all --replicas=0

# Delete tenant-managed resources (let Slurm's NodeSets handle Slurm pods)
kubectl -n tenant-foo delete pods -l app.kubernetes.io/managed-by=tenant-foo --grace-period=60

# Don't delete PVCs yet — that's step 4

Step 3 — Archive Slurm accounting and logs

This is the audit-trail step. The accounting database holds everything the tenant ever ran — job IDs, GPU-minutes, account, exit codes, queue/run/end times. Customers occasionally come back six months later asking "why did we get billed X" — you need to be able to answer.

3a. Dump the slurm DB to flat files

# Archive *all* completed records older than the cutoff to flat-file
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr archive dump \
    Directory=/var/spool/slurm/archive \
    Step=full \
    PurgeJobAfter=0 \
    PurgeStepAfter=0 \
    PurgeSuspendAfter=0
# (PurgeAfter=0 means "do not delete from live DB after archiving" — safer)

The archive lives on a PV. Copy it off-cluster to long-term object storage:

# Slurm controller has /var/spool/slurm mounted; copy to a controller-side staging dir
kubectl -n slurm-control cp slurmctld-0:/var/spool/slurm/archive \
    /tmp/slurm-archive-tenant-foo

# Compress + upload to retention bucket (your own infra)
tar czf slurm-archive-tenant-foo-$(date +%F).tgz /tmp/slurm-archive-tenant-foo
aws s3 cp slurm-archive-tenant-foo-$(date +%F).tgz \
    s3://retention-bucket/tenant-foo/slurm/ \
    --metadata "tenant=tenant-foo,retention-until=$(date -d '+7 years' +%F)"

3b. Export per-tenant `sacct` view as CSV (for billing)

A flat CSV is what your billing pipeline ingests. Generate one final pull for the full tenant lifetime:

kubectl -n slurm-control exec deploy/slurmctld -- sacct \
    --account=tenant-foo \
    --starttime=$(date -d '5 years ago' +%Y-%m-%d) \
    --format=JobID,User,Account,Partition,QOS,ReqGRES,AllocGRES,Submit,Start,End,Elapsed,State,ExitCode \
    --parsable2 \
    > sacct-tenant-foo-final.csv

# Sanity: row count > 0
wc -l sacct-tenant-foo-final.csv

Upload to the retention bucket alongside the DB archive. This is the file you grep when the tenant's CFO asks for a breakdown.

3c. Archive slurmctld and slurmd logs

Slurm's own logs (/var/log/slurm/slurmctld.log, slurmd.log on each node) are separate from accounting and have shorter retention by default. For a tenant offboarding, snapshot:

# Controller logs
kubectl -n slurm-control cp slurmctld-0:/var/log/slurm/slurmctld.log \
    /tmp/slurmctld-at-offboard-tenant-foo.log

# Per-node slurmd logs (across the tenant's reservation)
for node in gpu-01 gpu-02 gpu-03 gpu-04 gpu-05 gpu-06 gpu-07 gpu-08; do
    pod=$(kubectl -n slurm-control get pods -l app=slurmd -o name \
        | xargs -I{} kubectl -n slurm-control get {} -o jsonpath="{.spec.nodeName}={.metadata.name}\n" \
        | grep "^$node=" | cut -d= -f2)
    kubectl -n slurm-control cp slurm-control/${pod}:/var/log/slurm/slurmd.log \
        /tmp/slurmd-${node}-at-offboard-tenant-foo.log
done

# Archive
tar czf slurm-logs-tenant-foo-$(date +%F).tgz /tmp/slurm*-tenant-foo*.log
aws s3 cp slurm-logs-tenant-foo-$(date +%F).tgz s3://retention-bucket/tenant-foo/slurm-logs/

Logs typically retain 90 days post-offboarding. Accounting retains for the financial-audit horizon (often 7 years). Confirm your jurisdiction's policy.

Step 4 — Snapshot then delete /home and /scratch

This is the destructive step. The order is:

Snapshot.
Verify snapshot landed.
Then unmount and delete.

Skip step 1 or 2 and the working data is gone the moment you delete.

4a. Snapshot home

# Per-user home dirs
for user in alice bob carol; do
    weka fs snapshot create cluster-foo-home/tenant-foo/$user \
        offboard-${user}-$(date +%F) \
        --description "tenant-foo offboard for $user"
done

# Or, if /home is a single tenant-foo subdirectory
weka fs snapshot create cluster-foo-home/tenant-foo \
    offboard-tenant-foo-home-$(date +%F) \
    --description "tenant-foo offboard, all users"

# Verify the snapshot is listed and has nonzero usage
weka fs snapshot --filesystem cluster-foo-home/tenant-foo

4b. Snapshot scratch

weka fs snapshot create cluster-foo-scratch/tenant-foo \
    offboard-tenant-foo-scratch-$(date +%F) \
    --description "tenant-foo offboard scratch"

weka fs snapshot --filesystem cluster-foo-scratch/tenant-foo

Snapshots are copy-on-write — instant create, takes the same physical capacity until the source diverges. After deletion of the live data the snapshot becomes the only copy and it now consumes its full size. Plan for this in the capacity check before deletion.

4c. Tier or copy snapshots off-cluster (optional)

If retention policy requires the snapshot to survive even a Weka-cluster-loss event, lifecycle the snapshot to S3 cold tier:

weka fs tier policy create cluster-foo-home/tenant-foo \
    --rule "snapshot.name == 'offboard-tenant-foo-home-$(date +%F)'" \
    --action tier \
    --target retention-cold-tier

Or weka fs snapshot upload to a separately-managed object bucket if your retention regime is strict (regulated industries).

4d. Confirm tenant is not actively writing

Before unmounting:

# No tenant pods running
kubectl -n tenant-foo get pods --field-selector status.phase=Running

# No open files on the login pod
kubectl -n tenant-foo exec login-tenant-foo-0 -- lsof /home/tenant-foo 2>/dev/null
kubectl -n tenant-foo exec login-tenant-foo-0 -- lsof /scratch/tenant-foo 2>/dev/null

# No squeue entries
kubectl -n slurm-control exec deploy/slurmctld -- squeue --account=tenant-foo

All three must be empty. If any returns content, go back to step 2 and resolve.

4e. Delete PVCs and live filesystems

# K8s-side PVCs (will trigger CSI to release the underlying volume,
# but reclaimPolicy=Retain means the data stays in Weka — that's fine,
# the snapshot already captures it)
kubectl -n tenant-foo delete pvc --all

# Live filesystem deletion. After this, the data is *only* in the snapshot.
weka fs delete cluster-foo-home/tenant-foo --skip-confirmation
weka fs delete cluster-foo-scratch/tenant-foo --skip-confirmation

# If the tenant had a dedicated org, drop org-bound filesystems too
weka org tenant-foo fs list
weka fs delete tenant-foo-home --skip-confirmation
weka fs delete tenant-foo-scratch --skip-confirmation

The snapshots persist independently of the source filesystem. They sit on the cluster until step 9.

Step 5 — Revoke Authentik access and SSH

Now that data is archived and the tenant has no compute, kill the auth path. Doing this earlier risks a tenant losing access mid-run; doing it later leaves a window where they could re-enter.

5a. Disable Authentik users

For each tenant user:

# Disable, don't delete (preserves audit trail of who-ran-what)
curl -sS -X PATCH "https://auth.example.internal/api/v3/core/users/<user-id>/" \
  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"is_active": false}'

Disabled users still resolve via LDAP outpost (so historical file ownership records stay sane) but PAM rejects login. The user record stays as a tombstone.

5b. Remove from tenant group

curl -sS -X PATCH "https://auth.example.internal/api/v3/core/groups/<tenant-foo-group-id>/" \
  -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
  -d '{"users": []}'

This drops them from the LDAP outpost's memberOf queries. Subsequent getent passwd may still resolve them depending on cache TTL, but they don't appear in the tenant's group enumeration.

5c. Force-rotate the SSH-keys Secret

Even with users disabled, take the keys away from the login pod's authorized_keys.d to remove the attack surface:

# The key-sync job runs on a cron — wait for next run, or trigger:
kubectl -n tenant-foo create job --from=cronjob/sshkey-sync sshkey-sync-offboard-$(date +%s)

# Or just delete the Secret and let the controller no-op (login pod will see empty dir)
kubectl -n tenant-foo delete secret tenant-foo-authorized-keys

# Restart the login pod to drop the projected volume content
kubectl -n tenant-foo rollout restart deploy/login-tenant-foo

5d. Confirm SSH is dead

From an external host (with the user's old SSH key on disk):

ssh -o ConnectTimeout=5 alice@login.tenant-foo.example.internal
# expected: "Permission denied (publickey)" or connection refused

If you get a shell, something didn't take. Check: nslcd cache (run nscd -i passwd or restart nslcd in the login pod), authorized_keys.d content (ls /etc/ssh/authorized_keys.d/), Authentik user is_active state.

Step 6 — Tear down K8s tenant resources

Now safe to delete the namespace and its workloads.

6a. Slurm deletion

# Mark account+associations as deleted in Slurm. Sub-account first.
kubectl -n slurm-control exec deploy/slurmctld -- \
    sacctmgr -i delete user where Account=ml-team-foo
kubectl -n slurm-control exec deploy/slurmctld -- \
    sacctmgr -i delete account ml-team-foo
kubectl -n slurm-control exec deploy/slurmctld -- \
    sacctmgr -i delete account tenant-foo
kubectl -n slurm-control exec deploy/slurmctld -- \
    sacctmgr -i delete qos normal-foo interactive-foo

sacctmgr delete is irreversible for the live DB but the archive (step 3) preserved the records.

6b. Remove the partition

Edit the slurm.conf ConfigMap (or whatever GitOps source manages it) and delete the PartitionName=tenant-foo line. Reconfigure:

kubectl -n slurm-control exec deploy/slurmctld -- scontrol reconfigure
kubectl -n slurm-control exec deploy/slurmctld -- sinfo | grep tenant-foo
# expected: empty

6c. Delete the namespace

# This cascades: deletes login pod, Slurm NodeSet pods, PVCs, Secrets,
# NetworkPolicies, the ReservationBinding, ResourceQuota, etc.
kubectl delete namespace tenant-foo

# Confirm
kubectl get ns tenant-foo 2>&1 | grep NotFound

If the namespace gets stuck in Terminating, common cause is a finalizer on a CRD whose controller has already been deleted. Inspect:

kubectl get namespace tenant-foo -o yaml | grep -A3 finalizers
kubectl api-resources --namespaced -o name | xargs -n1 -I{} \
    kubectl get -n tenant-foo {} 2>/dev/null | grep -v "^No resources"

Remove the offending finalizer manually only after confirming the controller is gone (otherwise the controller will fight you).

Step 7 — Release the Reservation, return capacity

# Set evictOnRelease=false so any leftover pods drain rather than abort
kubectl patch reservation tenant-foo-reservation \
    --type=merge \
    -p '{"spec":{"evictOnRelease":false}}'

# Delete the reservation
kubectl delete reservation tenant-foo-reservation

# Verify nodes lost the label and taint
kubectl get nodes -l reserved.tenant=tenant-foo
# expected: No resources found
kubectl describe node gpu-01 | grep -E 'Taints|Labels' | grep tenant
# expected: nothing tenant-related

The nodes are now free. Run a quick health check before declaring them ready for the next tenant — see health check runbook:

for node in gpu-01 gpu-02 gpu-03 gpu-04 gpu-05 gpu-06 gpu-07 gpu-08; do
    kubectl debug node/$node -it --image=busybox -- chroot /host bash -c '
        nvidia-smi --query-gpu=name,ecc.errors.uncorrected.aggregate.total --format=csv,noheader
        ibstat | grep -E "State|Rate"
    '
done

A node that was healthy when the tenant got it but is unhealthy at offboarding (XID errors in dmesg, ECC counts ticking up) goes into RMA, not the next tenant's pool.

Step 8 — Final confirmation

End-of-pipeline verification:

# 1. No tenant artifacts in K8s
kubectl get all -A | grep tenant-foo
# expected: empty

# 2. No tenant artifacts in Slurm
kubectl -n slurm-control exec deploy/slurmctld -- sacctmgr show account tenant-foo
# expected: empty

# 3. No tenant artifacts in Weka (live; snapshots remain)
weka fs | grep tenant-foo
# expected: empty
weka org list | grep tenant-foo
# expected: empty
weka fs snapshot list | grep tenant-foo
# expected: snapshots ARE listed (deliberate, retention)

# 4. No active Authentik users
curl -sS -H "Authorization: Bearer $AUTHENTIK_API_TOKEN" \
    "https://auth.example.internal/api/v3/core/groups/?name=tenant-foo" \
    | jq '.results[0].users_obj | length'
# expected: 0

# 5. Reservation gone
kubectl get reservations | grep tenant-foo
# expected: empty

# 6. Capacity counted as free in the planner
# (See capacity-planning runbook for the operator math)

Step 9 — After the retention period

The retention period (typically 30 days for snapshots, 7 years for accounting) is the window when a customer can credibly come back and say "we need that data". After it elapses:

# Hard-delete the snapshots
weka fs snapshot delete cluster-foo-home/tenant-foo offboard-tenant-foo-home-$(date +%F)
weka fs snapshot delete cluster-foo-scratch/tenant-foo offboard-tenant-foo-scratch-$(date +%F)

# After the much longer accounting retention (7y), purge the archive
aws s3 rm s3://retention-bucket/tenant-foo/ --recursive

# Final Authentik user removal (after legal hold expires)
for user_id in <list>; do
    curl -sS -X DELETE "https://auth.example.internal/api/v3/core/users/${user_id}/" \
      -H "Authorization: Bearer $AUTHENTIK_API_TOKEN"
done

This step is run from a scheduled job tracked against a retention calendar. Don't do it by hand at offboarding time — you'll forget which tenant's retention expired when.

Compliance considerations

If the tenant or one of their named users invokes the right-to-erasure during the retention window, the position is:

Delete the snapshot for that user's home dir immediately.
Redact PII (email, full name, IP-of-last-login) from the user's Authentik record but retain the username + UID for accounting integrity. The sacct records are linked by UID; if you delete the UID-to-name mapping you can no longer attribute past compute-hours, which is itself a legal/financial obligation.
Update the retention bucket to mark this tenant as "GDPR-redacted" so future audits know.

Document the redaction in a separate ledger. The tension between "right to erasure" and "tax/financial-audit retention" is real — your legal team should give you the position; this runbook reflects a typical resolution but is not legal advice.

Audit-log integrity

The slurmctld and slurmd logs (step 3c) are system logs, not customer data. They typically do not need to be redacted on a GDPR request — they're operationally necessary records of system behaviour. Confirm with legal.

Multi-jurisdiction

If the tenant operated under multiple legal jurisdictions (e.g. EU and US data), retention policy may differ per record. Default: keep the longest required retention, redact under the strictest erasure rules.

Subpoena hold

If the tenant or an associated party is subject to a litigation hold, all retention timers pause. The operator does not delete; legal communicates the hold release before step 9 runs. Make sure your retention automation reads from a "hold flag" so a tenant under hold is silently skipped during scheduled cleanups.

Validation: what "offboarded" looks like

Operator-side checklist, signed off in the change ticket:

Capacity returned: nodes appear in kubectl get nodes without the tenant label, and the planner shows them as available.
No tenant K8s objects, namespaces, partitions, or accounts remain in the live state.
Snapshots exist and are tagged with the offboard date.
Slurm archive uploaded to retention bucket; sacct CSV uploaded.
Authentik users disabled; group emptied; SSH path verified rejected.
Customer notified of completion with snapshot retention end-date.
Retention timer entered in the operator's calendar.

Rollback / "wait, they're staying"

Sometimes a tenant decides at the last minute to extend. The further down the runbook you got, the harder this is.

At step 1-2: Trivial. sacctmgr modify ... GrpJobs=N back to original; scontrol update PartitionName=tenant-foo State=UP. Customer keeps running.
At step 3: No actual destructive action yet — archive is a copy. Trivially reversible.
At step 4 (snapshots taken, not deleted): Reversible. Live FS is intact.
At step 4 (FS deleted): Restore from snapshot via weka fs snapshot clone. The clone is a new filesystem; remount. Communicate the data state — anything written between the snapshot and the deletion is lost.
At step 5: Re-enable Authentik users; restore Secret with SSH keys; restart login pod.
At step 6 (account deleted): Re-create accounts via the onboarding runbook. sacct historical records linked to the now-deleted account become orphans; if you need them attributed, manual DB surgery on slurmdbd is required (paid Slurm support territory).
At step 7 (Reservation gone): Re-create the Reservation. If the operator already gave the nodes to another tenant in the meantime, you cannot get them back — they belong to whoever holds the Reservation now.

The honest summary: rollback is painful past step 6. Confirm aggressively before crossing that line.

Common failure modes

kubectl delete namespace hangs in Terminating. Finalizer on a CRD whose controller has been removed, or a stuck PVC. Inspect kubectl get -n tenant-foo all; resolve the specific resource before retrying. Force-removal of finalizers (kubectl patch -p '{"metadata":{"finalizers":[]}}' --type=merge) is the last resort.

weka fs delete returns "filesystem is in use". A client still has it mounted somewhere. Common cause: a stale wekafs mount on a node that left the cluster but kept the kernel module loaded. See Weka stale mount. Find the holder, unmount, retry.

Slurm archive succeeds but the CSV is empty. Time-window for sacct may be too short, or the user did not have any jobs in the period. Re-run with --starttime extending further back. Always check wc -l on the CSV before declaring archive complete.

Authentik user deletion via API returns 200 but user still resolves. LDAP outpost has a momentary disconnect from the API; or nslcd cache on the login pod hasn't expired yet. Restart nslcd in the login pod, or wait 600s (default cache TTL).

Reservation deletion succeeds but nodes still have the taint. Operator's reconciler crashed or was paused. Check kubectl -n reservation-operator logs deploy/reservation-operator; restart; the next reconcile will clean up.

Snapshot disk usage surges after live FS deletion. Expected — see step 4. Plan for a temporary capacity bump in the cluster equal to the tenant's data footprint until the snapshot retention expires.

scancel hits a job in CG (completing) state and never finishes. Slurm's epilog is hung, usually waiting on a wedged process. SSH to the slurmd's host and inspect:

kubectl -n slurm-control exec slurmd-tenant-foo-0 -- ps -ef | grep -E 'epilog|slurmd'

If the epilog is genuinely stuck, scontrol update NodeName=gpu-XX State=DOWN Reason=stuck-epilog and remove the CG state via DB surgery. Document this is a slurmd-level problem to fix during the next maintenance window.

Login pod cordons before the last user has logged out. A user's interactive session is sitting on /home. They get disconnected mid-edit. Usually fine because their work is in /home (snapshotted), but communicate the moment the lockout happens.