Capacity planning — operator-side GPU math, forecasting, expansion

GPU-hours math, projected vs actual utilization, expansion lead times, spot vs reserved tradeoffs, idle hunting, reservation overcommit, forecasting workload growth, deciding between adding tenants and expanding infrastructure.

Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

Capacity planning on a multi-tenant GPU cluster is not finance, even though it gets handed to people who think it is. It is operations: knowing what you have, what is actually being used, what new demand is coming, and what physical lead time stands between you and "more". The math is simple. Getting the inputs right is the hard part.

This page is the operator's view: the numbers you compute, where you get them from, and how you make decisions when the answers are uncomfortable.

The unit: GPU-hour

Capacity is measured in GPU-hours. One H100 running for one hour is 1 GPU-hour. A 64-GPU node running for a 24-hour day delivers 64 × 24 = 1536 GPU-hours. A 1024-GPU cluster running flat for a 30-day month produces 1024 × 24 × 30 = 737,280 GPU-hours.

This is the unit you bill in, the unit you forecast in, and the unit your monitoring should track. Not "node-hours" (different SKUs deliver different work), not "jobs" (jobs vary 100,000x in size), not "TFLOPS" (theoretical, not what you delivered).

The two derived metrics:

  • Allocated GPU-hours — what tenants reserved or were assigned. This is what they paid for.
  • Used GPU-hours — what they actually computed on. Sometimes called "wall-clock GPU-hours". Always less than allocated.

The ratio Used / Allocated is utilization. The ratio Used / Available (where Available = total fleet capacity) is occupancy. They diverge when allocated > available (overcommit) or when nodes are down for maintenance.

Track both, separately, monthly.

Projected vs actual: the forecasting baseline

You build a forecast at the start of a quarter. You compare to actuals at the end. If projected and actual diverge consistently in the same direction, your model is wrong and you need to adjust.

Pull the actuals from sacct:

# GPU-hours used per tenant in the last 30 days
kubectl -n slurm-control exec deploy/slurmctld -- sacct \
    --starttime=$(date -d '30 days ago' +%Y-%m-%d) \
    --endtime=$(date +%Y-%m-%d) \
    --state=COMPLETED,FAILED,TIMEOUT,CANCELLED \
    --format=Account,Elapsed,AllocTRES%50 \
    --noheader \
    --parsable2 \
    --allocations \
    | awk -F'|' '
        {
          n_gpu = 0
          if (match($3, /gres\/gpu=([0-9]+)/, m)) n_gpu = m[1]
          # Elapsed in seconds via "[DD-]HH:MM:SS"
          # ...parse and accumulate per account...
        }'

In practice you don't roll your own — you pipe sacct into Prometheus via an exporter, or scrape into a data warehouse. The shape of the report:

ACCOUNT          ALLOCATED_GPU_HOURS   USED_GPU_HOURS   UTILIZATION
tenant-foo       46080.0               31204.7          67.7%
tenant-bar       30720.0               28890.1          94.0%
tenant-baz       15360.0                4200.5          27.4%
platform          7680.0                6120.3          79.7%
TOTAL            99840.0               70415.6          70.5%

Three things this tells you:

  1. Cluster-wide actual utilization is 70.5%. You sold ~99k GPU-hours and delivered ~70k. The 30k delta is real money the customer paid for and didn't use.
  2. tenant-bar is at 94% utilization. They are at-or-near saturation. They will ask for more capacity in the next contract round.
  3. tenant-baz is at 27%. They are over-bought. Either the contract was too generous, or they have been disrupted. Reach out, find out which.

That last row is the input to the next-quarter forecast.

Why utilization is never 100%

Even a tenant doing everything right will not hit 100%. The realistic ceiling for production training is 75-85% over a month. Reasons:

  • Job submission gaps — researchers don't have a job ready every minute. Iteration time exists.
  • Pipeline parallel waits — the slowest stage of a multi-stage workload caps the throughput.
  • Reboot / maintenance windows — driver upgrades, kernel patches, NIC firmware. Plan for ~2-4% of capacity lost per month to this.
  • Dead-time on preemption — for spot, the time between preemption and the job's restart-from-checkpoint is unbillable but allocated.
  • Partition fragmentation — a tenant with 64 GPUs but jobs that need 128 GPUs gets 0% utilization until they refactor.

So when you see a 70% number, ask: where did the other 30% go? If the answer is "evenly across the above", the cluster is healthy. If the answer is "20% on one user's wedged interactive sessions", you have an idle problem (see below).

Idle hunting

Idle GPUs that hold an allocation are pure waste. The customer pays, you provide the GPU, the GPU does nothing. The classic shapes:

  • A user runs srun --pty bash to debug, walks away, the shell sits open for 11 hours.
  • A training job stalls on a data-loading bug — GPU memory is allocated but compute is 0%.
  • A K8s pod requests 8 GPUs, runs an import torch; while True: sleep(3600) placeholder.

DCGM is what you query for the actual compute utilization:

# DCGM exporter Prom metrics
DCGM_FI_DEV_GPU_UTIL{Hostname="gpu-01", gpu="0"}    # 0-100 instantaneous
DCGM_FI_DEV_FB_USED_BYTES                            # framebuffer bytes used
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE                      # tensor cores active fraction

The query for "GPUs allocated but idle for 30 minutes":

# GPU is in a Slurm job
slurm_job_state{state="RUNNING"} > 0
unless on(hostname, gpu_id)
# but its 30-min average compute is < 20%
(avg_over_time(DCGM_FI_DEV_GPU_UTIL[30m]) > 20)

A reasonable threshold for "idle waste" is < 20% sustained for 30+ min. Tighter thresholds catch more waste but also more legitimate stalls (data loading, checkpointing). Looser thresholds miss nothing significant.

What to do with the result:

  • Soft enforcement — daily report to tenants showing per-user idle GPU-hours. Just visibility often fixes it.
  • Hard enforcement — Slurm JobSubmitPlugin that auto-cancels interactive sessions after --time= or after N minutes of < 20% util. Aggressive, but effective on shared partitions.
  • Tier-based — interactive QoS allows idle (it's debugging); production QoS gets canceled (it's a bug). See Slurm scheduling for how to bind QoS to enforcement policies.

The math: if your cluster is 1024 GPUs at 70% utilization, and idle hunting recovers 5 percentage points (70% → 75%), that's 51,200 additional billable GPU-hours per month. At any nonzero $/hour, the engineering effort to set up the dashboard pays back in days.

Reservation overcommit

A simpler lever: sell more capacity than you have, when usage statistics show that not everyone uses their full allocation simultaneously.

If three tenants each have 32-GPU reservations but each averages 60% utilization, the simultaneous peak is rarely 96 GPUs. You can size the underlying fleet at 80 GPUs, sell three 32-GPU reservations, and use preemptive QoS to handle the rare hour when all three peak at once.

Implementation:

# Two QoS classes:
# - "reserved": runs on the tenant's nominal nodes, never preempted
# - "spillover": runs anywhere with capacity, gets preempted when reserved bumps it

sacctmgr add qos reserved \
    Priority=200 \
    PreemptMode=OFF \
    Flags=DenyOnLimit

sacctmgr add qos spillover \
    Priority=50 \
    PreemptMode=REQUEUE \
    Preempt=reserved \
    Flags=DenyOnLimit

# Tenant gets both — reserved is default, spillover is opt-in for jobs that can absorb preemption
sacctmgr modify account tenant-foo set qos=reserved,spillover DefaultQOS=reserved

Set PreemptMode=REQUEUE so a spillover job is requeued (not killed) when a reserved job needs the GPU. The spillover job restarts from its last checkpoint when capacity is free again — see scheduling for the preemption mechanics.

The key safety constraint: never set the overcommit ratio so high that the expected peak exceeds the actual fleet. Compute statistically:

peak_usage_over_30d = quantile(0.99, sum(allocated_gpus[t]) - sum(idle_gpus[t]))

Use the 99th percentile peak, not the max. Sizing for max means you're not really overcommitting. Sizing for 95th percentile is aggressive — there will be hours of contention. 99th percentile is a reasonable balance for paying customers who can absorb the rare 1% delay.

Spot vs reserved tradeoffs

Two procurement models, two tenant tradeoffs.

Reserved capacity — the tenant has a contract for N GPUs for M months. They pay for the full term whether they use it or not. They get guaranteed scheduling, no preemption.

Spot / on-demand — the tenant pays only for compute used. They get preempted when reserved tenants need the capacity.

From the operator's perspective, the math:

  • Reserved tenants smooth your revenue. You commit physical capacity to them; they commit cash.
  • Spot tenants fill the gap when your average utilization is below 100%. They convert the otherwise-wasted 30% (from the earlier example) into revenue.
  • The right ratio of reserved-to-spot capacity depends on your fleet size and the volatility of your reserved tenants' actual demand.

Typical mix: 70% capacity sold as reserved, 30% sold as spot. Spot prices are lower (often 30-50% of reserved per-hour), but spot revenue compensates because it's revenue you wouldn't otherwise have.

When not to do spot:

  • Workloads with hard real-time constraints (inference SLA). Preemption breaks them.
  • Workloads that can't checkpoint (no framework support, or the model's too large to checkpoint cheaply). A 10-hour preemption hit on a workload that re-trains from scratch is a customer relationship problem.
  • Brand-new tenants without trust. Don't put them on spot until you've seen their workload behave.

When spot is great:

  • Long-running training where checkpointing every 10 minutes is fine.
  • Inference on stateless workloads (e.g. batch scoring) where preemption is just a delay, not a failure.
  • Hyperparameter sweeps where individual jobs are short and re-runnable.

Expansion lead times

The hardest part of capacity planning. You can't conjure GPUs.

Expansion typeTypical lead timeNotes
Spin up cloud spot capacity (existing acct)hoursHighest unit cost. Bursty workloads only.
Spin up cloud on-demand reserveddaysCheaper than spot per-month-equivalent if used > 60%.
Add nodes to existing colo4-8 weeksVendor delivery + colo install + cabling.
New colo expansion (rack space available)8-16 weeksHardware delivery + cabling + IB fabric.
New colo expansion (no rack space)16-32 weeksDatacenter contract + power + cooling work.
New region / new datacenter6-12 monthsReal-estate, regulatory, fabric, full operations bring-up.

The 8-16 week number for "new bare metal in an existing facility" is the one operators most often underestimate. It includes:

  • Order placement and supplier lead time (often 4-8 weeks for current-gen GPU systems)
  • Shipping and customs (1-2 weeks if international)
  • Datacenter receiving, racking, cabling (1-2 weeks)
  • Network bring-up, IB fabric extension, switch port allocation (1-2 weeks)
  • Operating system, drivers, GPU operator deployment (a few days)
  • Validation pass — see network validation (1-3 days for a rack)
  • Integration into existing reservation pool (hours, the easy part)

So when a tenant says "I need double capacity by next quarter," you have to know which segment of the timeline you can compress (spot, cloud burst) and which you cannot (bare-metal expansion). Communicate clearly.

When to add a new tenant vs expand existing

A tenant says "we want more". You have spare capacity. Two paths:

  1. Expand the existing tenant. Bump their Reservation, increase their Slurm GrpTRES, raise their quota.
  2. Onboard them as a "second tier" with separate accounting and possibly a separate partition.

Choose (1) when:

  • The increase is < 50% of their current size.
  • The work is the same type (same workloads, same QoS pattern, same SLA).
  • They will absorb the new capacity within 30 days (you've checked utilization trends).

Choose (2) when:

  • The increase doubles their footprint or more.
  • The new work has different requirements (e.g. they had training, now they want inference with stricter latency).
  • The accounting is meaningfully separate (different cost centers, different chargeback codes).
  • The new work would distort their fair-share if mixed with the old.

A common operator mistake: expanding a tenant who actually needed a separate sub-tenant, leading to fair-share fights inside the tenant's own account hierarchy ("ML team vs data team are fighting over GPU-time"). It's not your fight, but you can avoid causing it.

Forecasting growth

You forecast by extrapolating the last 6-12 months of per-tenant usage and adding announced new tenants.

projected_gpu_hours[next_quarter] =
    sum_over_existing_tenants(
        actual_gpu_hours[last_quarter] * (1 + tenant_growth_rate)
    )
    + sum_over_announced_tenants(contract_size)
    + contingency_buffer (typically 10-15%)

The contingency buffer is for the things you don't see coming: an existing tenant lands a major new project, a new tenant accelerates onboarding. 10% if your tenants are stable and predictable, 15% if you have a high signing rate.

Compare projected to fleet capacity:

fleet_capacity[next_quarter] =
    current_capacity
    + already_in_flight_expansion
    - planned_decommissions
    * (1 - typical_downtime_fraction)   # ~0.95 to 0.97

If projected > fleet_capacity, you have a gap. Look at the lead-time table above and figure out which expansion path closes it.

If projected < fleet_capacity, you have surplus. Either sell more spot (preferred), or start trimming planned expansion before commitments lock in.

The key discipline is doing this every quarter. A 4-week deviation from projection is normal noise; a 2-quarter trend in the same direction is a model error you have to understand.

Daily / weekly / monthly cadence

FrequencyWhat you check
DailyCluster utilization, idle GPU report, alert noise
WeeklyPer-tenant utilization, capacity available for spot, expansion-pipeline status
MonthlyForecast vs actual, contract renewals coming up, decommissions coming up
QuarterlyRe-baseline forecast, expansion decisions, tenant renewals, retirement of old hardware

The daily report should be 1 page. The monthly should be 3-5 pages. The quarterly review feeds the next-quarter procurement decision.

Concrete dashboards

Recommended Grafana panels for a capacity-planning dashboard:

1. Cluster occupancy over time (last 30d, hourly avg)        # how full is the cluster?
2. Per-tenant GPU-hours allocated vs used (this month)        # who's wasting?
3. Idle GPU-hours, by tenant, by week                          # where's the idle?
4. Spot vs reserved revenue mix (this month)                  # how's the spot business?
5. Forecast vs actual delta (per quarter, last 4 quarters)    # is my model wrong?
6. Expansion pipeline (open POs, ETA, attached tenant)        # what's coming?
7. Per-tenant fair-share usage vs limit                        # fairness check

Wire from Prometheus (DCGM exporter, Slurm exporter) and the slurmdbd directly. Export monthly snapshots to a data warehouse so you can run year-over-year comparisons without depending on Prometheus retention.

Decommissions

When a node is N years old (typically 4-5 for GPU systems), it gets decommissioned. Plan this months in advance — you can't just yank a node, especially if it's part of a tenant's reservation.

# 1. Identify candidates
kubectl get nodes -L hardware-purchase-date

# 2. Notify reservation holders 60+ days before
# 3. Migrate workloads to replacement nodes (typically a parallel new-rack expansion)
# 4. Cordon and drain (see decommission runbook in your ops docs)
# 5. Physical removal during a maintenance window

The trap: decommissioning a node from a tenant's reservation when you have no replacement counts as a capacity reduction for them. The contract may forbid it. Communicate before committing to the decommission date.

Worked example: a quarterly forecast run

To make the math concrete, here's what a quarterly forecast looks like for a 768-GPU H100 cluster.

State at the start of the quarter

Cluster:           768 H100 GPUs total
                   8 GPUs/node × 96 nodes
                   3 racks, IB fat-tree, 400G per port
Operating since:   2024-Q3 (this is start of 2026-Q2)
Tenants:           4 reserved + 2 spot

Reservations active:
  tenant-foo:      256 GPUs reserved, 32 nodes, 12-month term ending 2026-Q4
  tenant-bar:      128 GPUs reserved, 16 nodes, 6-month term ending 2026-Q3
  tenant-baz:       64 GPUs reserved,  8 nodes, 3-month term ending 2026-Q2  <-- ending this quarter
  tenant-qux:      192 GPUs reserved, 24 nodes, 18-month term ending 2027-Q1
  spot-pool:       128 GPUs           16 nodes, no fixed term

Last-quarter actuals

Per-tenant utilization from sacct:

ACCOUNT       ALLOC_GPU_HRS    USED_GPU_HRS    UTIL%
tenant-foo    552,960          421,608         76.2%
tenant-bar    276,480          254,361         92.0%
tenant-baz    138,240           54,892         39.7%
tenant-qux    414,720          317,894         76.6%
spot-pool     276,480          268,144         97.0%
TOTAL       1,658,880        1,316,899         79.4%

Observations:

  • tenant-foo and tenant-qux are in the healthy 75-80% range. Both will likely renew at current size.
  • tenant-bar at 92% is saturated. They'll ask for more on renewal.
  • tenant-baz at 39.7% is dramatically under-bought. Either the workload didn't materialize as expected, or contracts allowed too much headroom. Either way, on offboarding next month, capacity returns to the pool.
  • spot-pool at 97% means there is unmet spot demand — every spot GPU-hour is being absorbed. Indicates spot supply is constrained, not just balanced.

Forecast for next quarter

Demand additions:
  + tenant-foo renewal:       256 GPUs (flat, healthy util)
  + tenant-bar renewal:       192 GPUs (+50%, they asked for 64 more)
  + tenant-qux mid-term:      192 GPUs (flat, no change)
  + tenant-quux NEW onboarding (signed 2026-04-15):   128 GPUs from 2026-Q2

Demand reductions:
  - tenant-baz off-board:    -64 GPUs returns at end of 2026-Q2

Demand summary at end of next quarter:
  reserved demand:  256 + 192 + 192 + 128 = 768 GPUs
  spot demand:      saturated at 128, demand likely > supply

Fleet capacity at end of next quarter:
  current capacity:                768 GPUs
  - decommissions planned:         0
  + expansion in flight:           0 (none ordered)
  expected available:              768 GPUs

Reserved fits exactly. Spot has nowhere to go.

Decision points

The numbers raise four questions:

  1. Is the spot supply constraint a real problem? If spot revenue is meaningful and spot tenants are asking for more, expand. If spot is gravy and reserved is the business, shrug.
  2. Should we overcommit reserved? Average reserved utilization is ~80%. If you sold 900 GPUs of reserved against 768 of fleet (with preemptive QoS), the math works out. But: bear-quarter risk if reserved demand spikes and you have nowhere to spill.
  3. Is the tenant-bar increase a signal of broader demand? If the renewal conversation revealed they're scaling a project, expect similar requests from others. Order capacity now (16-week lead time) for delivery in 2026-Q3.
  4. What about tenant-baz's offboard? 64 GPUs returning to the pool can absorb tenant-bar's +64. Check the timing — does tenant-baz return capacity before tenant-bar needs the increase? If not, you have a 2-week gap of overcommit.

What you actually do

Based on the analysis:

Action items for 2026-Q2:
  [ ] Sign tenant-bar +64 contract; aim for delivery 2026-Q3 start
  [ ] Confirm tenant-baz offboarding hard date (week 2 of Q2)
  [ ] Onboard tenant-quux (128 GPUs against current spot pool — shrink spot to 64)
  [ ] If spot demand visible at QBR — order 1 rack expansion (8 nodes / 64 GPUs)
       lead time 12-16w; delivery target 2026-Q3 mid
  [ ] Re-baseline forecast at end of Q2 (this exercise repeats)

Track each item in your project tracker. Each has a date, an owner, and a "next milestone". Don't let "order more capacity" sit as an action without a placed PO.

How tenants actually consume — patterns to watch

Some patterns repeat across customers. Recognize them in the utilization data and forecast accordingly.

The "spike at deadline" tenant. Used 30% for 10 weeks, then 100% for 2. Total quarterly utilization 50% but they cannot live with capacity reductions. Don't undersize them based on the average; use the 95th percentile.

The "ramp" tenant. Started at 20% utilization, growing 5pp/month. By month 6 they're at 50% and projected to hit 75% by month 9. The forecast for next quarter is the trajectory, not the average.

The "step function" tenant. Stable for 5 months, then a project starts and they want 2x. Watch for advance notice; the contract may have a flex clause for it.

The "we're leaving" tenant. Utilization gradually declines as they move workloads elsewhere. By the time they tell you, it's been visible in metrics for a quarter. Use the early signal to plan for the offboard timing.

The "burning down the budget" tenant. Approaching the end of a fiscal year, they suddenly run jobs at 100% to "use what they paid for". Expect it; don't be surprised; don't accidentally interpret the spike as new sustained demand.

The "always on" production-inference tenant. Diurnal pattern: 90% utilization peak hours, 30% off-peak. Total daily utilization 60% but the peak is the constraint. Capacity sizing is against the peak, not the average.

Each pattern has a different forecasting math. The cluster operator's job is to see the patterns and apply the right model.

Capacity at the rack-fragment level

Aggregate "free GPUs" hides operational pain. A 100-GPU cluster with 96 GPUs allocated and 4 free, where the 4 free are scattered as 1 each across 4 different nodes, cannot run an 8-GPU job. Standard kubectl get won't show this.

Track the largest contiguous block:

# Per-node free GPUs, sorted descending — operator-side view
kubectl get nodes -l reserved.tenant -o json | jq -r '
  .items[]
  | {
      node: .metadata.name,
      tenant: .metadata.labels."reserved.tenant",
      capacity: (.status.capacity."nvidia.com/gpu" // "0" | tonumber),
      allocatable: (.status.allocatable."nvidia.com/gpu" // "0" | tonumber)
    }
  | "\(.tenant)\t\(.node)\tcap=\(.capacity)\talloc=\(.allocatable)"
' | sort

# Then per tenant: max contiguous = max single-node free GPUs
# (assumes intra-tenant intra-node packing; for inter-node tightness,
#  sum top-N nodes' free GPUs to compute "biggest jobs that fit")

A weekly fragmentation report should highlight: "tenant-foo has 24 free GPUs across 8 nodes; max contiguous 4. Their largest queued job needs 16. They will wait for natural job completion or external action."

When fragmentation is the bottleneck, options:

  • Defragmentation drain — pause new submissions to a partition, let it drain, restart. Painful and rarely worth it for a short fragmentation event.
  • Relax topology constraints — if the tenant's job is scheduling-pinned to "must be 8 GPUs in 1 node" but actually works fine on "8 GPUs across 2 nodes", relax. Often this is just --ntasks-per-node=4 instead of =8 in the sbatch.
  • Smarter binpacking in the scheduler — Slurm's SelectType=cons_tres with topology weighting helps; see scheduling.
  • Per-tenant defrag windows — schedule "no large jobs" hours where pending small jobs drain, freeing contiguous space.

Fragmentation is the single most common reason a cluster looks "not full" to the dashboard but is "full" to the user.

Tracking expansion-pipeline state

A multi-month procurement effort has too many moving parts to live in someone's head. Track in a single source:

EXPANSION-2026-Q3-RACK-D
  Hardware:        8x 8-GPU H100 nodes
  Vendor:          OEM-foo
  PO date:         2026-04-10
  Vendor ack ETA:  2026-04-14 (received)
  Vendor mfg ETA:  2026-06-01 (received)
  Ship target:     2026-06-15
  Receiving:       colo-bkk
  Rack space:      rack D, slots 1-8 (confirmed available)
  Power:           +56 kW, confirmed in colo agreement
  Network:         16x 400G ports on switch foo-leaf-3 (allocated)
  Validation:      runbook /docs/operations/network-validation-by-speed
  Tenant assigned: tenant-bar (renewal +64)
  Status:          IN_TRANSIT
  Risk:            ETA slip 2 weeks observed in supplier portal — track

Update weekly. Every "Status: IN_TRANSIT" item that doesn't get a status update for 2 weeks is a red flag — your supplier is going dark on you, escalate.

The tenant assignment field connects the procurement to the contract. If tenant-bar is the assigned tenant and tenant-bar churns before the rack arrives, the rack still arrives — make sure you have a Plan B in the spot pool.

Common operator mistakes

Common operator mistakes

Forecasting on best-case usage. You forecast tenants will grow at the high end of their range. Reality: half of them grow at the low end. Plan against the median, not the high.

Not watching the idle. Cluster looks "70% utilized" — but what's actually compute vs allocated-but-idle? The allocated-but-idle fraction is invisible in occupancy metrics; you need DCGM-level visibility.

Treating reservation as a hard limit. A tenant with a 64-GPU reservation will occasionally want 65 for a brief job. If your overcommit policy doesn't allow them to spill over (with appropriate QoS), they hit a hard ceiling and the relationship suffers. Build in soft headroom.

Ignoring fragmentation. A 64-GPU pool with 60 GPUs in use and 4 free can't run a 16-GPU job. Aggregated occupancy can be 95% while real fragmentation makes new jobs queue. Track the largest contiguous block metric, not just total free.

Forgetting the time dimension on lead time. "We have 3 months." If you start the procurement conversation today and the lead time is 16 weeks, you don't have 3 months. You have negative four weeks. Start the procurement conversation when forecast > current_capacity, not when need > current_capacity.

Assuming spot revenue is bonus. It is, until you've sold the same capacity twice — once as reserved with overcommit, once as spot. Then peak-hour preemption hits both customers and you owe credits to both. Track committed spot capacity against the overcommit headroom, not against fleet total.

Validation: are you tracking the right numbers?

Once a quarter, check: can you, in 5 minutes, produce these answers?

  1. Total cluster GPU-hour capacity for last month.
  2. Allocated GPU-hours, by tenant, for last month.
  3. Used GPU-hours, by tenant, for last month.
  4. Largest single block of free contiguous GPUs, today.
  5. Idle GPU-hours (allocated but < 20% util) for last month.
  6. Spot GPU-hour revenue for last month.
  7. Forecast vs actual for last quarter.
  8. Open expansion pipeline (units, expected delivery dates).

If any of those takes more than 5 minutes, the dashboard is wrong. Fix the dashboard.

See also

External:

  • Google SRE Workbook Ch. 30 — Capacity planning principles
  • "The Datacenter as a Computer" (Hoelzle, Barroso) — chapter on utilization economics