Cost optimization — operator levers that lower the dollar-per-useful-GPU-second

The cost of running a GPU cluster is dominated by two things: the GPUs themselves (capex amortized + power), and the time those GPUs spend not doing useful work. The first is mostly fixed by the time the cluster exists. The second is where operators have leverage — and where most platforms leave money on the table.

This page is a list of knobs you can actually turn. Each entry is the lever, the metric you watch, and the typical impact. Where a knob trades off against something else (latency, robustness), it says so. The point is not to pull every lever to its extreme — it is to know which levers exist when the finance team asks "where can we cut".

Operator's framing throughout: dollar-per-useful-GPU-second is the metric. "Useful" matters — a GPU at 95% utilization running a stuck training loop is 100% waste. A GPU at 60% utilization producing a model is 60% useful.

Lever 1 — Rightsizing per workload

H100 vs H200 vs B200 are not interchangeable. The right SKU depends on what the tenant is doing. Operators serving multiple workload types should partition their fleet, not run every workload on the most expensive option.

Training (large foundation models, distributed):

• Want highest interconnect bandwidth and HBM. H100 (80GB) was the sweet spot until H200; H200 (141GB) reduces tensor-parallel sharding pressure for 70B+ models.
• Power draw is high (700W on H100/H200 SXM5). Cooling cost factor is real.
• Less concerned about absolute price-per-GPU; the gradient sync bandwidth and HBM size dominate the math.

Inference (latency-sensitive, batch < 32):

• HBM bandwidth and clock speed matter more than raw FLOPS. H200 wins on memory-bound transformer decode.
• For smaller models (< 13B), L40S or even older A100 are price-effective.
• Multi-instance GPU (MIG) on inference splits one H100 into 7 logical GPUs at fractional cost — see lever 6.

Inference (throughput, batch > 64):

• Compute-bound. Latest gen always wins, but the price gap may not justify.
• Quantization (lever 5) reshapes the math significantly.

Fine-tuning / LoRA:

• Smaller models, sequence-parallel rather than full tensor-parallel. Modest interconnect demands.
• A100, L40S sufficient for many workloads. Don't burn H200s on a 7B LoRA.

Operator action: segment partitions by SKU, communicate to tenants which workload goes where, charge appropriately. A tenant trying to run inference on H200 nodes when L40S is plenty is overpaying — and taking your scarcest hardware away from training tenants who need it.

Lever 2 — Idle detection: DCGM util < 20% for 30 min on paid GPUs

The single highest-impact knob in most clusters. Idle GPU-hours sit at 5-25% of total in unmonitored fleets — a multi-million-dollar leak.

The detection rule:

GPU is allocated to a job
AND avg(DCGM_FI_DEV_GPU_UTIL) over last 30 min < 20%
AND avg(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE) over last 30 min < 5%
THEN flag as idle

The two metrics together because GPU_UTIL includes any kernel including nvidia-smi polling — the more reliable signal of "actually computing" is tensor pipe activity.

What to do once flagged:

Reporting cadence:

Daily: per-user idle GPU-hours (last 24h)
Weekly: per-tenant idle ratio (idle / allocated)
Monthly: cluster-wide idle reclamation potential

Visibility alone usually halves idle. People know "if it's idle for an hour, ops is going to ask why". When that doesn't work, hard enforcement (auto-cancel) closes the rest.

The Prometheus rule:

- alert: GPUIdleAllocation
  expr: |
    (
      avg_over_time(DCGM_FI_DEV_GPU_UTIL[30m]) < 20
      and
      avg_over_time(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE[30m]) < 0.05
    )
    and on(hostname, gpu_id)
    slurm_gpu_allocated == 1
  for: 30m
  labels:
    severity: info
  annotations:
    summary: "GPU {{ $labels.gpu_id }} on {{ $labels.hostname }} idle while allocated"

Tune thresholds carefully. Too tight (5%) and you false-positive on legitimate data-loading stalls. Too loose (40%) and you miss the long tail of "training pipeline mostly running but one stage stuck".

See DCGM for metric definitions.

Lever 3 — Checkpoint cadence

Checkpoint frequency trades off lost work against IO overhead.

• Long cadence (every 4 hours): minimal IO overhead (1-2% of training time), but a node failure or preemption costs up to 4 hours of compute.
• Short cadence (every 10 minutes): higher IO overhead (5-15% on a slow filesystem, < 2% on Weka), but max loss is 10 minutes.

The math:

expected_lost_compute_per_failure = checkpoint_interval / 2
expected_failures_per_month = MTBF_node^-1 * tenant_node_count
expected_lost_compute_per_month = above * preemption_rate (if spot)

cost_of_lost_compute = expected_lost_compute_per_month * tenant_hourly_cost
cost_of_overhead = checkpoint_overhead_pct * total_compute_per_month * cost_per_hour

minimize: cost_of_lost_compute + cost_of_overhead

For a typical reserved tenant on a low-failure-rate cluster, checkpoint every 30 minutes to 1 hour. For spot with frequent preemption, every 5-10 minutes. For a single-node finetuning job on stable hardware, even every 2 hours is fine.

The operator knob: ensure the underlying filesystem (Weka, Lustre) can absorb checkpoint write bursts without backpressure. A tenant trying to checkpoint a 175B model every 10 minutes onto a slow filesystem creates 30+ minutes of write-saturated IO, slowing every other tenant. Quota-of-IOPS / quota-of-bandwidth on shared filesystems is your knob — see Weka performance.

Tell tenants this. They almost never tune checkpoint frequency themselves; the framework default (e.g. PyTorch Lightning, every epoch) is rarely optimal.

Lever 4 — Spot when feasible, fixed when not

Already covered conceptually in capacity planning. Cost-side specifics:

Operator action 1: route preemptible workloads to spot capacity.

• Hyperparameter sweeps, batch inference, embedding regeneration, eval jobs — preemption-tolerant.
• Move them off reserved partitions onto spot. Spot price is 30-50% of reserved.
• Tenants don't always know they should — the platform should have a default spot QoS for any non-real-time workload.

Operator action 2: don't oversell spot.

• Spot is real revenue, not free. If you commit 30% of fleet capacity to spot tenants on the assumption they'll get preempted regularly, but reserved tenants only consume 70% of their reservation, spot tenants are running on reserved capacity. That's fine until reserved demand spikes — then spot gets preempted en masse.
• Track committed spot capacity against the expected reserved peak headroom, not against fleet total.

Operator action 3: communicate spot SLAs honestly.

• "You may be preempted with 2 minutes notice." Not "rare preemption". Not "best effort".
• Customers running spot need to architect for it (checkpoint, retry queue, idempotent batch jobs).

When NOT spot:

• Real-time inference (the SLA usually forbids any preemption).
• Stateful long-running jobs without checkpointing.
• Compliance-required predictable scheduling (e.g. nightly batch with a hard SLA).

Lever 5 — Inference autoscaling

Inference cost optimization is dominated by not paying for idle inference servers.

Two flavors:

Scale-to-zero — when no requests for N minutes, scale replicas to zero. Cold start when the next request arrives.

• Pros: pure pay-per-use.
• Cons: cold start can be 30s to several minutes for a large model. Unacceptable for latency-SLA workloads.
• Right for: internal tools, dev/staging, low-traffic services.

Always-on minimum — replicas >= 1 (or >= N for HA), scale up on load.

• Pros: no cold start.
• Cons: pay for at least 1 replica 24/7.
• Right for: customer-facing SLA workloads.

The operator knob: tune the scale-down delay. A short delay (60s) saves money on bursty workloads but causes flapping. A long delay (15 minutes) wastes capacity but stabilizes.

Implementation via Kubernetes HPA + VPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-tenant-foo
  namespace: tenant-foo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-tenant-foo
  minReplicas: 1
  maxReplicas: 32
  metrics:
    - type: Pods
      pods:
        metric:
          name: gpu_request_queue_depth
        target:
          type: AverageValue
          averageValue: "4"   # scale up when > 4 in-flight per replica
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600   # 10-minute cooldown
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0     # scale up fast
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30

Asymmetric scale-up vs scale-down is a key cost lever: scale up fast (otherwise you reject requests), scale down slow (otherwise you flap and pay the cold-start tax).

For workloads with predictable diurnal patterns (low traffic at night), supplement HPA with a CronJob that pre-scales before the morning peak — saves the warm-up burst while still capturing the off-peak savings.

Lever 6 — Quantization (FP16, INT8, FP8)

Operator's perspective on a model-side optimization: more requests per GPU.

• FP32 — research-grade precision; almost never deployed for inference. Half the throughput vs FP16 for no quality benefit.
• FP16 / BF16 — production default for most modern inference. Fits ~2x more model in memory.
• INT8 — quantized inference. ~2-4x throughput vs FP16, modest quality regression on most LLMs (1-2% on benchmarks).
• FP8 — Hopper-and-newer. ~1.5-2x throughput vs FP16, smaller quality regression than INT8.
• INT4 — extreme; 4x throughput vs FP16, larger quality cost. Common for on-device LLMs, less common in the cloud.

The operator's leverage isn't to choose the quantization level — that's the tenant's call against their quality bar — but to:

• Make sure the platform supports the quantized formats. NVIDIA Triton + TensorRT-LLM does the heavy lifting for FP8 and INT8 on H100/H200.
• Document the throughput delta. A tenant running FP16 on H200 doing 200 RPS could be doing 400 RPS at FP8. They are paying double for compute they could be free of.
• Bind quantized inference to MIG (lever 7) for very high concurrency.

Operator action: maintain an internal benchmark sheet showing throughput at each precision for the standard model classes (Llama 70B, Qwen 7B, Mistral, etc.). When a tenant onboards an inference workload, the conversation includes "what precision; here's the throughput per GPU". Many will switch when shown the math.

Lever 7 — MIG on inference for high concurrency

Multi-Instance GPU (MIG) splits one A100/H100 into up to 7 fully isolated logical GPUs. Each instance has dedicated SM, HBM, and L2 cache slices. Throughput per physical GPU goes up; latency per request stays roughly constant.

Applies when:

• You have many small models being served (each model uses < 1 H100 worth of memory).
• You want hard latency isolation between concurrent tenants on the same physical GPU (without MIG, two tenants sharing a GPU step on each other's L2 cache).
• Inference, generally — training distributes across whole GPUs.

Doesn't apply when:

• Single large model (one Llama 70B). MIG slices are too small to hold it.
• Training (NVLink doesn't extend to MIG instances; you can't tensor-parallel across MIG slices).
• Workloads needing GPUDirect RDMA inside the slice — has limitations.

Operator setup, slim:

# Enable MIG mode on a node, partition into 7 1g.10gb instances
sudo nvidia-smi -i 0 -mig 1
sudo nvidia-smi mig -cgi 19 -C   # 7x 1g.10gb (profile 19)

# Verify
nvidia-smi -L
# GPU 0: NVIDIA H100 80GB ...
#   MIG 1g.10gb     Device 0:
#   MIG 1g.10gb     Device 1: ...

Slurm-side, each MIG instance is a separate gres.conf line — see Slurm multi-tenant: MIG.

K8s-side, the GPU operator with migStrategy: mixed exposes MIG slices as nvidia.com/mig-1g.10gb resources separate from nvidia.com/gpu.

The cost math: a node with 8 H100s in 7-way MIG mode exposes 56 logical accelerators. If each handles a 200 RPS inference workload that would otherwise need a full H100 at 1000 RPS, the per-request cost is 1/7 of the full-GPU price. Real workloads vary; the saving is generally 3-5x on small-model inference, not the full 7x.

The downside: complexity. MIG nodes are not interchangeable with non-MIG nodes (the same GPU can't be both). Plan partitions accordingly: a "MIG inference" partition and a "full-GPU training" partition.

Lever 8 — Datacenter cooling cost factor

Cost per GPU isn't just the GPU. Power and cooling at the datacenter level add 30-100% on top of GPU power, depending on PUE (Power Usage Effectiveness):

• PUE 1.1 — best-in-class hyperscale, often liquid-cooled rear-door or direct-to-chip.
• PUE 1.3 — typical good-quality colo.
• PUE 1.6 — older facilities or hot-climate sites with poor air handling.
• PUE 2.0+ — broken; you're paying $2 of power for every $1 of useful work.

PUE multiplies the GPU's wall-power draw. An H100 at 700W in a PUE-1.6 site costs you 1120W at the meter. Across 1024 GPUs that's 430 kW of overhead just on cooling.

What you can do as an operator:

• Track PUE. Most colos report it monthly. If yours doesn't, ask. If they report only an annual average, ask for monthly — the variation is real.
• Negotiate liquid cooling. Direct-to-chip cooling on H100/H200 reduces the cooling overhead substantially. The capex is real (rear-door heat exchangers, plumbing) but the opex savings amortize over ~2 years on a busy fleet.
• Density tradeoff. Higher GPU density per rack means lower per-unit network and cabling cost but harder cooling. Balance varies per site.
• Workload-shaping for off-peak. Some power tariffs charge less off-peak. Defer non-urgent batch (training overnight, eval jobs at 3am) to off-peak windows — savings can be 10-30% on power if your contract supports time-of-use.
• Right-size HBA/NIC power. Idle high-bandwidth NICs at 400G can draw 20W per port. Modern NICs sleep aggressively but firmware varies; verify with mlxconfig and / or measure at the rack PDU.

You usually cannot move datacenters easily, so the leverage here is most accessible at procurement time: choose facilities with PUE in your comfort zone, and choose liquid-cooled racks for any new H200/B200 capacity.

Lever 9 — Reduce framework / driver waste

Lower-tier knobs but they add up.

Driver / firmware bloat. A node that takes 10 minutes to boot due to a slow PCIe enumeration burns 10 GPU-minutes × 8 GPUs = 80 GPU-minutes per reboot. Aggregated across a fleet over kernel upgrade cycles, real money. See driver / firmware mismatch.

Container image pull time. A 30 GB framework image pulled cold on every job start wastes minutes. Use:

• Image pre-pulling via DaemonSet on tenant nodes.
• Layer-cached registry mirrors close to the cluster.
• Squashed enroot images cached on local SSD — see enroot/pyxis.

Slow data loading. A training job spending 40% of wall time waiting for data is at 60% effective utilization. Check whether the bottleneck is the framework (DataLoader workers, prefetch), the filesystem (Weka should not be slow), or the CPU side (decoding overhead). DCGM-low-util-but-job-running is the symptom.

Logging volume. Verbose framework logging (every step, every loss) at 10 MB/s/GPU across 1024 GPUs is 10 GB/s of log writes. Most goes to a filesystem, gets indexed, costs storage and time. Sample / batch / send to OTel + a backend with retention rather than dumping to /scratch.

Lever 10 — Power and thermal capping

A subtle knob that operators often skip: GPU power capping for known-throughput-stable workloads.

H100 SXM5 ships with a 700W TDP. Many workloads — inference, smaller training — don't actually consume 700W steady; they spike briefly then drop. You can cap at, say, 500W and lose under 5% of throughput while saving 30% of power and reducing cooling load (which itself is a power saving via PUE).

# Check current power state
nvidia-smi --query-gpu=power.draw,power.limit,power.max_limit --format=csv

# Cap at 500W (persists until reboot or further change)
sudo nvidia-smi -i 0,1,2,3,4,5,6,7 -pl 500

# Persist via systemd unit or boot script if desired

Best applied:

• Per-partition: a "low-power inference" partition where caps are set; tenants opt in with their workload knowing it.
• Per-tenant: when the tenant is willing to trade peak performance for lower cost.
• Off-peak: dynamically lowering cap during off-peak hours when latency SLA is relaxed.

Worst applied:

• Universally on training nodes — you might trim 5-10% off training throughput, costing more in elapsed time than you save in power.
• On NCCL all-reduce-bound workloads — tail-latency sensitivity to power cap is high.

Validate by running the tenant's actual workload at the cap and at the default; measure end-to-end throughput and decide.

Lever 11 — Storage cost tiering

Datasets and old checkpoints are expensive to leave on hot SSD-tier storage.

The main filesystem (Weka, Lustre) charges per TB-stored regardless of access frequency. A 500 TB dataset accessed once a quarter costs the same as 500 TB accessed continuously. Tier the cold side:

# Lifecycle: tier files not touched in 30 days to S3 cold tier
weka fs tier policy create cluster-foo/datasets \
    --rule "atime > 30d" \
    --action tier

See Weka operations for the full mechanics. Cost ratio is typically 5-10x cheaper for cold-tier S3 vs hot SSD; tiering 50% of capacity halves the storage bill.

Tradeoff: first-access latency on tiered files is on the order of seconds (object pull) vs sub-millisecond on hot. Don't tier active dataset paths used by training; do tier inactive checkpoints, archived runs, exported model weights.

Operator-facing report: "X TB of files older than 30 days, not yet tiered". Surface it; communicate; auto-tier where the tenant has opted in.

Lever 12 — Right-time scheduling for batch

Some workloads aren't time-sensitive — embedding regeneration, dataset preprocessing, eval suites against new model versions. Run them at night when the rest of the cluster is quieter and (sometimes) cheaper.

If your colo bills time-of-use power, run batch off-peak. If your spot price varies by time of day (cloud-burst case), schedule the same way.

Implementation in Slurm:

# Submit with a reservation window: only run between 22:00 and 06:00
sbatch --begin=now+4hours --time=8:00:00 ...
# Or use scontrol to create a recurring reservation, jobs only land in window

Or with K8s CronJobs:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-embedding-refresh
spec:
  schedule: "0 23 * * *"   # 11pm daily
  jobTemplate:
    spec: ...

Tenant-side savings can be 10-30% for off-peak-tariff facilities. Operator-side leverage: don't pay daytime power for a job that has all night to finish.

Worked example: monthly cost-opt report

A real-shape monthly report that the platform team sends:

GPU CLUSTER COST OPTIMIZATION — APRIL 2026
==========================================

UTILIZATION
  Cluster occupancy:        70.2%
  Idle GPU-hours (allocated < 20% util):  18,420  (13% of allocated)
  Top 3 idle offenders:
    tenant-foo / alice (interactive sessions): 4,210 GPU-hr
    tenant-bar / training-staging (broken):    3,100 GPU-hr
    tenant-baz / unattended sweep:             2,840 GPU-hr

QUANTIZATION OPPORTUNITIES
  Inference deployments still on FP16:    7
    Estimated capacity savings if FP8:    ~3.5x (one-time switch)
    Communicated to tenants:               yes (April 12)

INFERENCE AUTOSCALING
  Always-on >1 replica deployments:       12
    Of those: <30% queue depth p95:       4   (action: review)
    Of those: <30% queue depth p95 + non-SLA-critical: 2  (action: enable scale-to-zero)

SPOT
  Reserved utilization:                   71%
  Spot capacity sold:                     128 GPUs (16% of fleet)
  Spot revenue:                            $X
  Spot preemption events:                  47 (within SLA threshold)

POWER / COOLING
  PUE this month:                          1.34
  PUE last month:                          1.31  (note: hotter month, expected)
  Power-cap experiment on inference partition:
    Average draw at 500W cap:              482W
    Throughput delta vs 700W:              -2.1%
    Power saving:                          ~31%
    -> Recommend rolling to all H100 inference partitions

ACTION ITEMS
  [ ] Notify users with > 1000 idle GPU-hr in March
  [ ] Migrate inference-tenant-baz to FP8 (tenant agreed)
  [ ] Enable scale-to-zero on dev-staging deployments
  [ ] Production rollout of 500W power cap on inference partition
  [ ] Tier cold dataset paths to S3 (manual review, then policy)

This is the page that goes to the head of platform monthly. It should be 1-2 pages, action-oriented, traceable to dollars where possible.

Common operator anti-patterns (the long list)

In addition to the high-level ones called out earlier:

Idle dashboards that nobody looks at. A weekly report nobody reads doesn't reduce idle. Either route the alert to the offending team via Slack/email, or schedule a recurring 15-minute review with the platform lead.

Hard auto-cancel without warning. Surprising tenants by killing their jobs at 30 minutes idle erodes trust. Send a warning at T-15 minutes; cancel at T-30 if no acknowledgement; tenant can scontrol release if the stall was legitimate.

Quantization shame. Don't make tenants feel bad for being on FP16 — they often inherited the deployment. Frame the conversation as "here's the speedup; here's the validation we ran; want to switch?".

Inference autoscaler tuned for the wrong dimension. Scaling on CPU when the bottleneck is GPU memory or queue depth. Verify the scale-on metric matches the bottleneck.

Power-capping production training. A 5% throughput hit on a training job that runs for 6 weeks costs 2 days of training time. Don't cap training nodes without measuring the workload-specific impact first.

MIG on a shared GPU without isolation. Without proper namespace + RBAC, two tenants on different MIG slices of the same physical GPU can DoS each other (PCIe BW contention, kernel scheduling). Always wrap with appropriate isolation.

Storage tiering for active data. A dataset accessed daily but with atime aged due to filesystem mount option (noatime) gets tiered, then re-fetched on first access. Disable noatime on tiered paths; use relatime instead.

Letting checkpoint write rates spike unbounded. A tenant doing 100-GB checkpoints every 5 minutes saturates Weka and slows everyone else. Quota of bytes-per-minute per tenant on the filesystem (or at the application layer) prevents the externalization.

What to instrument

For all the above to translate into action, you need data. The metrics:

A weekly cost-optimization report should: list top 10 idle GPU-hour offenders, break out spot vs reserved revenue, surface workloads still on FP32 / FP16 that could move to FP8, and flag any inference deployments running > 3 always-on replicas at < 30% queue depth.

Common operator anti-patterns

Optimizing for utilization at the expense of usefulness. A 100%-busy GPU running a stuck training loop is worse than a 70% GPU producing a model. Watch for "high GPU_UTIL but 0 tensor activity" — signals a polling loop, not work.

Not communicating quantization options. Tenants stay on FP16 because nobody told them FP8 is supported. The platform can publish a benchmark sheet without forcing the change.

Default-on always-on inference. Easier to deploy, but expensive at low traffic. Default to scale-to-N with sensible N rather than always-on with manual scale-down.

Aggressive scale-down causing flapping. Saves money on paper, costs more in cold-starts and rejected requests. Asymmetric scaling (fast up, slow down) is the right shape.

Ignoring datacenter-side cost. GPU is 60-70% of total cost; cooling and power overhead are 20-30%. Optimizing only the GPU side leaves a sixth of the bill untouched.

Charging at the lever, not at the cost. If you charge a tenant the same per-GPU-hour for FP8 inference at 4x throughput, you're misaligned with their incentive. Either bill on actual compute consumed (preferred for cloud) or sell tier-pricing that reflects efficient deployment.

See also

• Capacity planning — utilization metrics
• Slurm scheduling — preemption, QoS for spot
• Slurm multi-tenant — MIG and gres.conf
• Weka performance tuning — checkpoint write throughput
• DCGM — utilization metrics
• MIG — partitioning
• enroot/pyxis — image caching
• Vendor management — facility PUE and procurement-side cost levers

External:

• NVIDIA TensorRT-LLM (FP8 inference): docs.nvidia.com/tensorrt-llm
• DCGM exporter: github.com/NVIDIA/dcgm-exporter
• The Uptime Institute on PUE: uptimeinstitute.com