Reservations and ReservationBindings — multi-tenant GPU allocation

How a Reservation locks a set of GPU nodes for a specific tenant, how ReservationBinding injects nodeAffinity, and how a tenant pod ends up scheduled to its reserved capacity only.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

In a multi-tenant GPU cluster, every node belongs to one tenant. Pods from tenant A must not get scheduled to tenant B's nodes, and a tenant's reserved capacity must not be auto-released back to a general pool. Plain Kubernetes does not have a first-class concept for this; the closest primitive is taints + tolerations, which is too coarse and too easy to get wrong.

CoreWeave's Reservation and ReservationBinding CRDs (and similar designs in other multi-tenant K8s platforms) wrap that pattern into something declarative and operator-managed.

This page is a generic explanation of how the pattern works — the names map directly to CoreWeave's CRDs but the model applies to any operator-driven multi-tenant scheduler.

The pattern

                  ┌──────────────────┐
                  │   Reservation    │  "lock these nodes for tenant-foo"
                  │   tenant-foo     │
                  └────────┬─────────┘
                           │
           ┌───────────────┴────────────────┐
           ▼                                ▼
  ┌──────────────────┐             ┌──────────────────┐
  │ ReservationBind. │             │ ReservationBind. │
  │  ns: tenant-foo  │             │  ns: tenant-foo  │
  │  selects pods... │             │  selects pods... │
  └────────┬─────────┘             └─────────┬────────┘
           │                                 │
           ▼                                 ▼
   pod gets nodeAffinity injected   pod gets nodeAffinity injected
   to reserved.tenant=foo           to reserved.tenant=foo

Two CRDs:

Reservation — cluster-scoped. Names the tenant, the node selector that defines "the reserved set", and any taints to apply.
ReservationBinding — namespace-scoped. Lives in the tenant's namespace. Selects the tenant's pods and tells the admission webhook to inject nodeAffinity so they only land on reserved nodes.

The operator does three things:

Watches Reservation objects, applies labels (reserved.tenant=foo) and taints (reserved=foo:NoSchedule) to matching nodes.
Watches ReservationBinding objects, registers an admission webhook to mutate matching pods at creation time.
Mutates pod spec: adds nodeAffinity requiring reserved.tenant=foo and a matching toleration for reserved=foo:NoSchedule.

The combined effect: tenant pods land only on tenant nodes; non-tenant pods don't land on tenant nodes (taint blocks them).

Reservation example

apiVersion: scheduling.coreweave.cloud/v1alpha1
kind: Reservation
metadata:
  name: tenant-foo-reservation
spec:
  tenant: tenant-foo
  nodeSelector:
    matchLabels:
      hardware-pool: gpu-h100-rack-a
  reservedNodes:
    - gpu-01
    - gpu-02
    - gpu-03
    - gpu-04
  applyTaint: true
  taintEffect: NoSchedule

Operator's reconcile, conceptually:

For each node in reservedNodes:
  ensure label: reserved.tenant=tenant-foo
  ensure taint: reserved=tenant-foo:NoSchedule (if applyTaint)

For each node not in reservedNodes that has the label:
  remove the label and taint (released back to pool)

Nodes that are removed from reservedNodes get cordoned first if they have running tenant pods, then cleaned up — the operator should not yank capacity from under a running job.

ReservationBinding example

In the tenant's namespace:

apiVersion: scheduling.coreweave.cloud/v1alpha1
kind: ReservationBinding
metadata:
  name: tenant-foo-binding
  namespace: tenant-foo
spec:
  reservationName: tenant-foo-reservation
  podSelector:
    matchLabels:
      app.kubernetes.io/managed-by: tenant-foo
  injectAffinity: true
  injectToleration: true

podSelector is the gate. Pods in the namespace that match the selector get the mutation; pods that don't (e.g., a debug pod the platform team launched) are unaffected and go elsewhere.

What gets injected at admission:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: reserved.tenant
                operator: In
                values: ["tenant-foo"]
  tolerations:
    - key: reserved
      operator: Equal
      value: tenant-foo
      effect: NoSchedule

requiredDuringSchedulingIgnoredDuringExecution is the right semantics for HPC: scheduling decisions are hard, but we don't want pods evicted mid-run if the operator briefly disagrees.

Pod-side: what the tenant sees

The tenant submits an ordinary pod:

apiVersion: v1
kind: Pod
metadata:
  name: training-step
  namespace: tenant-foo
  labels:
    app.kubernetes.io/managed-by: tenant-foo
spec:
  containers:
    - name: trainer
      image: my-tenant.io/trainer:v1
      resources:
        limits:
          nvidia.com/gpu: 8

After admission:

spec:
  affinity:
    nodeAffinity: ...    # injected
  tolerations: ...        # injected
  containers:
    - name: trainer
      ...

The tenant doesn't have to know about the labels or taints. They just write normal pod YAML.

Releasing capacity

To return nodes to the general pool:

kubectl edit reservation tenant-foo-reservation
# remove gpu-04 from reservedNodes

The operator:

Cordons gpu-04 (no new pods land there).
Waits for tenant-foo pods to drain naturally — or, if evictOnRelease: true, evicts them.
Removes the reserved.tenant=tenant-foo label and the taint.
The node is now schedulable for any pod that tolerates the general-pool taint.

A tenant losing capacity mid-job is operationally bad. The default should be drain-and-wait, not evict. Notify the tenant before changing reservations.

Day-2 operations

Adding a node to an existing tenant

kubectl edit reservation tenant-foo-reservation
# add gpu-05 to reservedNodes

Within seconds the operator labels and taints gpu-05; tenant-foo pods can immediately schedule there.

Inspecting what's running on whose nodes

# All nodes labeled for a tenant
kubectl get nodes -l reserved.tenant=tenant-foo

# All pods on that tenant's nodes
kubectl get pods -A --field-selector spec.nodeName=gpu-01

# Reservations cluster-wide
kubectl get reservations
# NAME                       TENANT       RESERVED   AGE
# tenant-foo-reservation     tenant-foo   4          12d
# tenant-bar-reservation     tenant-bar   8          2d

# Bindings in a tenant namespace
kubectl get reservationbindings -n tenant-foo

Quotas inside a reservation

Reservation only controls which nodes the tenant can use. To bound how much of those nodes' resources, use a normal ResourceQuota in the tenant namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-foo-quota
  namespace: tenant-foo
spec:
  hard:
    requests.nvidia.com/gpu: "32"
    limits.memory: "2Ti"

Common failure modes

Tenant pod stuck Pending with "0/N nodes available" — usually one of:

Reservation exists but the pod's labels don't match ReservationBinding.podSelector. Check the binding's selector vs the pod's labels.
The reserved nodes are all cordoned (drained for ops). kubectl get nodes -l reserved.tenant=tenant-foo and look for SchedulingDisabled.
Resource shortage (no GPU available on tenant's nodes). Standard "0/4 nodes are available: 4 Insufficient nvidia.com/gpu".

kubectl describe pod shows the exact predicate failures.

Pods landing on the wrong tenant's nodes — ReservationBinding is missing or scoped wrong. The binding should match every workload-bearing pod in the namespace; if a pod is missing labels, admission won't mutate it and it'll schedule wherever a toleration allows.

Pods landing on the right nodes but taking forever to schedule — admission webhook is slow or down. kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep reservation and check the operator's logs.

Operator unable to label nodes after a tenant change — RBAC. The operator's ServiceAccount needs nodes update/patch permissions. kubectl auth can-i patch nodes --as=system:serviceaccount:reservation-operator:reservation-operator.

A node is in two reservations — bug in the operator if it allows that. The reconcile should reject overlapping reservedNodes lists.

When not to use Reservations

Single-tenant clusters. Reservations add complexity for no benefit.
Spot/burst workloads where any-tenant-can-use-any-node is the model. Use plain nodeSelector and taints instead.
Compute auctions / preemption-heavy environments. The drain-and-wait release model fights with preemption.