Inference vs training: the infrastructure split

A common mistake when building "an AI cluster" is to design it for one workload and then try to run the other on it. The two are different infrastructures wearing the same GPU. Training is a parallel HPC job: thousands of GPUs synchronizing gradients, every step bottlenecked by the slowest peer, run for hours to days, checkpointed periodically. Inference is a service: hundreds of independent requests per second, latency budgets in tens of milliseconds, scale up and down with traffic, run forever, no checkpoints. They want different GPUs, different fabrics, different schedulers, different cost models. This page is the side-by-side.

The fundamental asymmetry

The decisions cascade from those rows. Designing one cluster as if it were the other yields predictable failures — training on an inference cluster crawls because the fabric is undersized; inference on a training cluster wastes 60% of HBM and burns the wrong NIC pattern.

Training: what dominates

Gradient synchronization is the load-bearing wall

Every training step ends with an all-reduce across all data-parallel ranks. For a 70B model in BF16, gradients are ~140 GB per rank. With 1024 GPUs sync'ing every step, that all-reduce is on the critical path. NCCL's ring/tree algorithms use bisection bandwidth of the fabric. Halve the IB bandwidth, halve your training throughput, double your wall-clock and your bill.

Practical implications:

• 400 Gb/s IB or RoCE per GPU minimum for H100/H200-class clusters (one CX-7 per GPU is typical on DGX H100; one OSFP per pair on B200).
• Non-blocking fat-tree or rail-optimized topology. Oversubscription kills training throughput.
• NVLink and NVSwitch fully utilized. Intra-node tensor-parallel needs every GB/s of NVLink — 900 GB/s on H100 NVSwitch, 1.8 TB/s on B200.
• NCCL tuning matters. SHARP, PXN, env vars (NCCL_IB_HCA, NCCL_IB_GID_INDEX, NCCL_TOPO_FILE) can swing throughput by 30%.

See: NCCL multi-node tuning, NCCL tests, Multi-node validation.

Compute-bound but only barely

A modern 70B-class transformer training step is:

forward + backward FLOPs ≈ 6 × parameters × tokens

For a 70B model on 4M tokens/step that's ≈ 1.7 ZFLOPs per step. On a 1024-GPU H100 cluster (≈ 3.4 EFLOPS BF16 peak), that's ~0.5 s of compute, plus all-reduce, plus checkpointing and data movement. Real-world utilization of peak FLOPS ("MFU") sits at 35-50% even on well-tuned clusters; the rest is bubbles, communication, recomputation, etc.

The GPU choice for training is usually whichever has the most TFLOPS per dollar of total cost (GPU + chassis + fabric + power), with HBM big enough to hold a tensor-parallel rank's weights. On H100 you fit a 175B model with TP=4, PP=2 inside a node. On B200/B300 the larger HBM lets you simplify model parallelism — fewer ranks, fewer collectives, less plumbing.

Predictable batch sizes

Training batch size is set in the launch script. The whole job sees the same global batch every step. That's why you can carve up the cluster cleanly: tensor-parallel and pipeline-parallel ranks are static, bound to specific GPUs at launch. A pod on GPU 7 of node 12 stays there for the whole run. No autoscaling.

Checkpoints are how you survive failures

Long training runs assume hardware fails. The mitigation is checkpoint and resume:

• Every N steps (often 30-60 minutes of wall-clock), all ranks save state to a parallel filesystem.
• A failed node → cancel job → swap node → restart from last checkpoint.
• Faster filesystems = more frequent checkpoints = less work lost.

Checkpoint sizes: 70B BF16 model ≈ 140 GB just for weights, +optimizer state (~2× weights for AdamW) = ~420 GB total. On 1024 ranks the parallel filesystem write is the bottleneck. Weka, VAST, Lustre, BeeGFS are the common choices; aggregate write bandwidth in the TB/s range is the ask for the largest jobs.

Real sbatch for training

#!/bin/bash
#SBATCH --job-name=llama70b-pretrain
#SBATCH --nodes=128
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --time=72:00:00
#SBATCH --partition=h100
#SBATCH --exclusive
#SBATCH --output=/scratch/logs/%x_%j.out

# Static fabric: rails pinned, IB tuned for collectives
export NCCL_IB_HCA=mlx5_0,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_9,mlx5_10,mlx5_11
export NCCL_IB_GID_INDEX=3
export NCCL_TOPO_FILE=/etc/nccl/topology.xml
export NCCL_DEBUG=WARN
export NCCL_ASYNC_ERROR_HANDLING=1

# Checkpoint to parallel FS every 1000 steps
export CKPT_DIR=/lustre/training/llama70b/run-$SLURM_JOB_ID
export CKPT_INTERVAL=1000

# Pyxis + enroot for the container
srun --container-image=/registry/pytorch-24.08-py3.sqsh \
     --container-mounts=/lustre:/lustre,/scratch:/scratch \
     --container-workdir=/workspace \
     bash -c '
        torchrun \
          --nproc_per_node=$SLURM_GPUS_ON_NODE \
          --nnodes=$SLURM_JOB_NUM_NODES \
          --rdzv_backend=c10d \
          --rdzv_endpoint=$(scontrol show hostnames $SLURM_NODELIST | head -1):29500 \
          /workspace/pretrain.py \
            --model llama-70b \
            --tensor-parallel 8 \
            --pipeline-parallel 4 \
            --global-batch-size 2048 \
            --ckpt-dir $CKPT_DIR \
            --ckpt-interval $CKPT_INTERVAL
     '

Notes on what's load-bearing here:

• --exclusive — no other job shares these nodes. Required for predictable NCCL collectives.
• NCCL_IB_HCA — explicitly lists the HCAs in rail order. Skip this and NCCL guesses, often wrong.
• --rdzv_backend=c10d — torch.distributed rendezvous; the head rank IP is the first in the slurm node list.
• Container image is enroot-converted (.sqsh) for fast Pyxis launch on every node.

See: Slurm scheduling, Slurm job failures, Enroot/Pyxis.

Inference: what dominates

Memory-bound, not compute-bound

For decode (the per-token generation step in an LLM), the GPU does ~2 FLOPs per parameter per token. The arithmetic intensity is terrible — decode is bottlenecked by reading the model weights from HBM, not by compute. On H100, peak compute is ~1979 TFLOPS BF16; sustained decode throughput is closer to whatever you can stream out of HBM bandwidth (3.35 TB/s on H100).

This rearranges the entire GPU choice:

The headline: for inference, more HBM and more HBM bandwidth beats more TFLOPS. H200 and B200/B300 exist because of inference economics. Customers buy them to fit longer contexts, larger batches, or more concurrent requests on a single GPU.

KV cache is the new bottleneck

Each in-flight request holds a KV cache — the keys and values for every token in its context, every attention layer. Per token, per layer:

KV bytes = 2 (K, V) × num_heads × head_dim × dtype_bytes

For Llama-3-70B in FP16: 2 × 8 (GQA) × 128 × 2 = 4 KB/layer/token, × 80 layers = 320 KB per token. At 4096-token context that's 1.3 GB per request. 30 concurrent requests = 40 GB of KV cache before you even count the model weights (140 GB FP16, 70 GB FP8).

This is why 80 GB H100 is a tight fit for a 70B model in production:

• 70 GB weights (FP8) leaves 10 GB for KV → ~7 concurrent 4k requests. Bad.
• 141 GB H200: 70 GB weights + 70 GB KV → ~50 concurrent requests. Much better.
• 192 GB B200: 70 GB weights + 120 GB KV → ~90 concurrent requests, or longer contexts.

Or you distribute KV across nodes (NIXL, see below) and trade a network round-trip for KV cache headroom.

Latency-sensitive — variable batch sizes

Inference servers do continuous batching (also called "in-flight batching" or "iteration-level scheduling"): each iteration the scheduler decides which requests get to run a step. New requests join the batch mid-run; finished requests drop out. Batch size fluctuates per iteration.

Implications:

• The GPU's optimal batch is application-dependent. Too small → under-utilized. Too large → first-token latency blows up.
• p99 latency depends on prefill chunk size and scheduler policy more than on raw GPU performance.
• Speculative decoding (small draft model proposes tokens, big model verifies in parallel) raises decode throughput by 2-3× — at the cost of operational complexity and HBM headroom for the draft model.

Runs 24/7. No checkpoints.

Inference is stateless from the operator's perspective. There's no checkpoint to take. There's nothing to resume. A pod dies → traffic reroutes → a new pod warms up (model load from disk: 30-90 s for 70B) → traffic balances back. The whole machinery is autoscaling, health-checking, rolling-updates.

What the fabric needs

• No tight all-reduce loop. Inference doesn't run NCCL all-reduce per request; tensor-parallel intra-node uses NVLink, and that's it most of the time.
• Standard datacenter Ethernet is often enough for the inference plane (request in, tokens out). 25/100 Gb LAN.
• RDMA between inference workers matters when you do prefill/decode disaggregation or distributed KV cache. NIXL over UCX, see NIXL. 200/400 Gb RoCE if you go this way.
• Cross-AZ inference fleets are now common; that means the fabric stops being one IB cluster and starts being a real network.

Real Helm chart for inference

# values.yaml — vLLM model server with HPA on tokens/sec
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama70b
  namespace: inference
spec:
  replicas: 4
  selector:
    matchLabels:
      app: vllm-llama70b
  template:
    metadata:
      labels:
        app: vllm-llama70b
    spec:
      nodeSelector:
        accelerator: h200
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.x.x
        args:
        - --model
        - meta-llama/Llama-3.3-70B-Instruct
        - --tensor-parallel-size
        - "2"
        - --max-model-len
        - "16384"
        - --gpu-memory-utilization
        - "0.92"
        - --max-num-batched-tokens
        - "8192"
        - --max-num-seqs
        - "256"
        - --enable-prefix-caching
        - --port
        - "8000"
        ports:
        - containerPort: 8000
          name: http
        resources:
          limits:
            nvidia.com/gpu: 2
            memory: 120Gi
            cpu: "16"
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60   # model load takes 30-90s for 70B
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
        volumeMounts:
        - name: model-cache
          mountPath: /root/.cache/huggingface
        - name: shm
          mountPath: /dev/shm
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: hf-cache
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: 16Gi
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-llama70b-hpa
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama70b
  minReplicas: 2
  maxReplicas: 16
  metrics:
  - type: Pods
    pods:
      metric:
        name: vllm:num_requests_running   # vLLM Prometheus metric
      target:
        type: AverageValue
        averageValue: "200"
  - type: Pods
    pods:
      metric:
        name: DCGM_FI_DEV_GPU_UTIL
      target:
        type: AverageValue
        averageValue: "75"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300       # don't thrash on dips

Notes on what's load-bearing here:

• --tensor-parallel-size 2 on H200 — fits the 70B in HBM with KV headroom.
• --gpu-memory-utilization 0.92 — vLLM grabs that fraction of HBM for the KV cache pool. The rest is for activations and overhead.
• --enable-prefix-caching — shared prompt prefixes are cached across requests; large win for chat workloads.
• HPA on vllm:num_requests_running (a vLLM-emitted Prometheus metric) is more meaningful than GPU util — vLLM's continuous batcher will run hot at low queue depth.
• initialDelaySeconds: 60 on the readiness probe — anything less and pods get killed during model load.
• emptyDir { medium: Memory } for /dev/shm — vLLM uses shared memory for IPC between workers; the default 64 MB is not enough.
• Anti-affinity (omitted for brevity) should spread replicas across nodes so a node failure halves capacity at most.

Cost models differ

A useful split: training costs in GPU-hours; inference costs in tokens-per-dollar.

Training cost

Training $ = GPU-hours × $/GPU-hour
GPU-hours = (model FLOPs to train) / (GPU FLOPs × MFU × num_GPUs) × num_GPUs

A 70B model from scratch on 1024 H100s at 50% MFU on 15T tokens is on the order of 1.5M GPU-hours. That's a single bill, paid once, depreciated over the model's useful life.

Inference cost

Inference $/M-tokens = ($/GPU-hour × num_GPUs) / (tokens/sec × 3600) × 1e6

For a 70B served on 2× H200 at 800 decode tok/s: with $4/GPU-h list price, that's ~$2.78 per million decode tokens of GPU cost — paid every day, every request, forever.

The optimization levers differ:

• Training: bigger fabric, more MFU, less idle time. Capex/long-term economics.
• Inference: bigger HBM (more concurrency), better batcher, quantization (FP8/INT4), speculative decoding, KV cache reuse. Opex/per-token economics.

This is why the same GPU vendor sells different SKUs for these markets. H100 was a balanced workhorse. H200 / B200 / B300 with bigger HBM tilt the SKU toward inference. Future SKUs will likely diverge further.

Storage: completely different problems

The data path on each side has almost nothing in common.

Training storage

The training cluster wants two things from storage:

1. Sustained read throughput for the training data. A 1024-GPU H100 job processing 4M tokens/step at 0.5 s/step is reading multi-GB/s from disk per rank in some workloads. Over the cluster: multi-TB/s aggregate.
2. Burst write throughput for checkpoints. Every N steps, every rank writes a slice of model+optimizer state. For a 70B model with optimizer state, that's 400+ GB written across all ranks in seconds.

The answer is a parallel filesystem — Weka, VAST, Lustre, BeeGFS, GPFS. All of them shard the namespace across hundreds of NVMe drives and serve clients in parallel via RDMA-capable protocols. The metadata side matters too: huge dataset trees with millions of small files punish naive POSIX servers.

Two operational realities:

• Cache hierarchy. Hot dataset shards in node-local NVMe (some clusters use the storage's own client-side cache; some use a separate fs2-style staging). Cold data on object store (S3, Ceph, vendor object).
• Format. Modern training rarely uses raw files; it uses WebDataset / TFRecord / Parquet shards, often pre-shuffled, sometimes pre-tokenized. The format affects throughput more than the FS choice does.

Inference storage

The inference cluster needs storage for model weights, plus optionally:

• A shared cache of HuggingFace / NGC models so cold-start pods don't all hit the WAN at once.
• A KV cache offload tier for long-context workloads (host RAM / NVMe / remote, see NIXL).
• Logs and metrics, but those go to standard observability storage.

Weights are read once at pod startup and held in HBM. The throughput requirement is "load 140 GB once in under 60 s" — needs ~2.5 GB/s read bandwidth, easily satisfied by node-local NVMe with the model pre-staged, or a shared object store with parallel multipart download.

Common patterns:

• Pre-stage on every node. A sidecar / DaemonSet syncs the model catalogue to local NVMe. Pod start is instant.
• Object store + warm-up. Pod pulls from S3 on first start; subsequent starts on the same node hit local cache.
• PVC backed by ReadOnlyMany. NFS / shared filesystem with the model files. Simple; bandwidth ceiling is the FS.

The contrast: training storage is a continuously-streaming production-load workhorse; inference storage is "stage these blobs efficiently and get out of the way."

Scheduling: opposite philosophies

Training scheduling is gang scheduling: 1024 GPUs all-or-nothing. Slurm with srun --exclusive is the default. Kubernetes options include Volcano, Kueue, and NVIDIA Run:AI for batch workloads. The job either gets all of its requested resources at once or it queues — there is no "partial start, scale up later."

# Slurm queue with reservation
$ scontrol create reservation \
    starttime=2026-05-10T08:00:00 \
    duration=72:00:00 \
    user=research-team-a \
    nodes=128 \
    flags=ignore_jobs \
    reservationname=llama-pretrain-may

Inference scheduling is independent pods: each replica is a unit. Kubernetes Deployment + HPA, or Knative-style serverless. Any pod can come up or down without affecting peers. Headroom is engineered as buffer replicas, not gang allocation.

# Kubernetes inference rollout
$ kubectl rollout restart deployment/vllm-llama70b -n inference
$ kubectl rollout status deployment/vllm-llama70b -n inference --watch

The schedulers don't share a worldview. Don't try to make one scheduler do both. Co-existence patterns (Sunk runs Slurm on top of Kubernetes; Run:AI exposes a higher-level API over both) are valid, but each layer respects the other's primitives.

See: Reservations and queues, Slurm scheduling, Sunk troubleshooting.

Failure modes: side-by-side

The common theme: training failures are big and rare; inference failures are small and continuous. Operations tooling needs to match — training wants alerting on job-state and node-health; inference wants alerting on SLO breaches and per-tenant error rates.

See: Incident response, K8s GPU pod failures, Health check runbook.

Bring-up checklists

A short version, because the long version is in the validation pages.

Training cluster bring-up

1. NCCL all-reduce sweep at 2, 4, 8, 16, ... node sizes. Catches fabric and topology bugs.
2. GPUDirect RDMA confirmed end-to-end. ib_write_bw between nodes should saturate the wire.
3. Bisection bandwidth measured across the spine. Below spec → renegotiate with the fabric vendor or accept the cap.
4. Parallel FS read and write at full target bandwidth from all nodes simultaneously.
5. Driver / fabric-manager / OFED versions consistent across the fleet.
6. NCCL topology file generated and pinned (NCCL_TOPO_FILE).
7. A real training run for at least 4 hours, hitting checkpoints, MFU within target.

See: Multi-node validation, perftest validation, NCCL tests.

Inference cluster bring-up

1. Pod cold-start time measured for each model size. < 90 s is the typical target for 70B.
2. HPA scale events triggered with synthetic load. Confirm scale-up under 2 minutes, scale-down stable.
3. A representative load test: prompts and concurrency that match production. Measure p50/p99 TTFT and decode latency, tokens/sec, KV cache fullness.
4. Replica failure drill: kill a pod under load; ensure traffic re-balances within seconds, no client-visible errors beyond retries.
5. Rolling deploy without SLO breach. Surge replicas; bake time at each step.
6. Metrics pipeline: model-server metrics → Prometheus → Grafana, alerts wired to on-call.
7. API gateway: auth working; rate limits enforced; per-tenant attribution accurate.

Where the two clusters overlap (and where to fight against them merging)

Pressure to merge: "I have spare GPUs at night; can I run inference on training nodes?" Sometimes yes, but watch out:

• Fabric topology. Training nodes are optimized for collective bandwidth; the inference plane usually doesn't need it. Putting inference traffic across the IB cluster is fine; just don't expect IB to autoscale with traffic.
• MIG. Training rarely uses MIG (no NVLink across slices). Inference loves MIG. If you partition a training node for inference, you've lost it for training without a mig-disable cycle.
• Driver state. MPS, fabric-manager configuration, MIG mode — switching modes is a node-drain operation. Don't do it in the middle of a training run.
• Scheduler. Slurm and Kubernetes coexisting on the same nodes is a known pattern (Sunk, Run:AI, Volcano), but it's an architecture decision, not an afterthought.

The pattern that works: separate clusters with a shared object store and shared identity, oversubscription within each, occasional manual rebalancing. Trying to make a single scheduler do both well usually means it does both poorly.

Operational gotchas

• Inference clusters with training-grade IB. Wasted capex; standard datacenter Ethernet would have been fine. Conversely, training on a non-RDMA cluster does not work — you'd find out at first all-reduce.
• HBM-tight inference. The KV cache fills up; the server starts evicting; tail latency explodes. Either bigger HBM or KV cache offload (NIXL).
• Training jobs with too-small checkpoint cadence. A failure 10 hours from last checkpoint is a 10-hour waste. Cadence should be tuned to MTBF of the cluster.
• Inference autoscaling on the wrong metric. GPU util is misleading on continuous-batch servers — they run hot at low load. Use queue-depth, batch-size, tokens-in-flight metrics instead.
• Inference pods without anti-affinity. Two replicas on the same node: one node failure = full outage. Spread across racks if possible.
• Mixed-generation inference fleets without DRA. With H100/H200/B200 in one cluster, the device plugin can't say "any GPU with ≥ 100 GB" — DRA can; see GPU sharing.

See also

• GPU sharing: MIG vs MPS vs time-slicing vs vGPU — the sharing decisions look very different on each side.
• Inference stack components — Triton, vLLM, TGI, SGLang, TensorRT-LLM and when to pick which.
• NIXL — KV cache transport for prefill/decode disaggregation and KV offload.
• NCCL multi-node tuning — load-bearing for training.
• GPU generations — HBM/bandwidth/NVLink tradeoffs across H100 / H200 / B200 / B300.
• Reservations and queues — how training claims the cluster vs how inference shares it.
• Slurm + Sunk troubleshooting — when one cluster runs both.
• Multi-node validation — bring-up tests that prove the cluster is training-grade.