Production inference stack: components and choices

A production LLM service is more than a model on a GPU. It is a stack: a model server doing continuous batching, a runtime executing kernels, a Kubernetes deployment scaled by a real metric, an API gateway in front for auth and rate limits, a KV cache pool sized to fit the workload, sometimes a draft model for speculative decoding, sometimes a fleet of LoRA adapters multiplexed onto a base model, and a metrics pipeline that tells you when any of it is sick. This page is the operator's tour: which components matter, what each one does, how to choose, and what breaks under load.

The shape of the stack

                ┌─────────────┐    ┌──────────────┐
   client  ───► │ API gateway │ ─► │ load balancer│ ─► [model server pods]
                │ auth, rate, │    │ tokens-aware │       │
                │ routing     │    │ session-aware│       ▼
                └─────────────┘    └──────────────┘     ┌──────────────────┐
                                                       │ model server      │
                                                       │  - scheduler      │
                                                       │  - continuous     │
                                                       │    batcher        │
                                                       │  - KV cache pool  │
                                                       │  - LoRA registry  │
                                                       └──────────────────┘
                                                                │
                                                                ▼
                                                       ┌──────────────────┐
                                                       │ runtime           │
                                                       │  - vLLM / TRT-LLM │
                                                       │  - kernels (FlashAttn │
                                                       │    PagedAttn)     │
                                                       └──────────────────┘
                                                                │
                                                                ▼
                                                       ┌──────────────────┐
                                                       │ GPU + HBM         │
                                                       └──────────────────┘

Optional layers on top: a router (NVIDIA Dynamo, Envoy with custom logic) that does prefix-aware routing; a prefill/decode-disaggregated topology where prefill happens on one pool and decode on another, with NIXL shuttling KV in between.

Model servers: the field

Five frameworks dominate. They overlap; they're not interchangeable.

A few things operators learn the hard way:

• vLLM and TensorRT-LLM are not mutually exclusive — Triton can host TensorRT-LLM via the tensorrt_llm backend and vLLM via the vllm backend, and even put them in an ensemble. "vLLM vs Triton" is sometimes a category error.
• Engine-compile vs eager-load matters operationally. TensorRT-LLM compiles per-(model, GPU, max-seq-len, batch-size, dtype) tuple — minutes to hours of build time, then you redeploy when any of those change. vLLM loads the safetensors / HF weights and runs. Choose based on whether your model and limits change frequently.
• Multi-model serving is a Triton job. vLLM serves one base model per process (with multiple LoRAs, see below). If you have ten different models behind one endpoint, you're running ten vLLM processes or one Triton with ten models loaded.

vLLM: the production default

vLLM has become the default for LLM serving for three reasons:

1. PagedAttention — the KV cache is paged like virtual memory, eliminating fragmentation. You can pack ~2-4× more concurrent requests into the same HBM than with naive contiguous-allocation servers.
2. Continuous batching — every iteration the scheduler picks which requests advance one step. New requests merge into the running batch immediately; finished requests drop out. No waiting for the slowest request in a static batch.
3. OpenAI-compatible API — most clients already speak this. Drop-in.

A real serve command:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --dtype auto \
  --port 8000 \
  --api-key $VLLM_API_KEY

Knobs that move the most:

• --gpu-memory-utilization — fraction of HBM vLLM grabs for the KV cache pool. 0.90-0.95 is typical; lower if you see OOMs from activations.
• --max-num-batched-tokens — the cap on tokens-in-flight per iteration. Raising it improves throughput at the cost of first-token latency.
• --max-num-seqs — cap on concurrent requests. Drives KV cache sizing.
• --enable-prefix-caching — share prompt prefixes across requests. Free wins for chat workloads with system prompts.
• --max-model-len — the longest sequence length you'll allow. Lower → bigger KV pool for the same HBM, more concurrent requests.

TensorRT-LLM: when you need the last 30%

When you've squeezed everything out of vLLM and need more, TensorRT-LLM is the next step. The deal:

• You compile an engine per model variant: (model, dtype, max-batch-size, max-seq-len, GPU model). Build time: minutes for small models, hours for 70B+.
• Engines run on the TensorRT-LLM C++ runtime, usually wrapped in Triton's tensorrt_llm backend.
• Throughput on H100/H200/B200 with FP8 is consistently the highest measured in public benchmarks, especially for long-context and large-batch workloads.

The operational tax:

• Build cache somewhere — engines are large (GB-scale) and per-GPU. Object store + cache key on (model, dtype, GPU, build-flags).
• Redeploy on change — bumping max-seq-len from 8K to 32K means a new build.
• Less flexibility for fast-moving model spaces — if you swap models every week, the build cycle hurts.

# Build an engine (TensorRT-LLM 0.x.x)
trtllm-build \
  --checkpoint_dir /workspace/checkpoints/llama70b-fp8 \
  --output_dir /engines/llama70b-h200-fp8 \
  --gemm_plugin fp8 \
  --gpt_attention_plugin auto \
  --max_batch_size 256 \
  --max_input_len 8192 \
  --max_seq_len 16384 \
  --tp_size 2

# Serve via Triton with tensorrt_llm backend
tritonserver \
  --model-repository=/models \
  --grpc-port=8001 \
  --http-port=8000 \
  --metrics-port=8002 \
  --log-verbose=0

Triton: when the zoo is mixed

Triton's strength is everything that isn't a single LLM:

• Multiple models behind one endpoint, dynamically loaded/unloaded via a model-control API.
• Ensemble pipelines — preprocess in Python backend → embed in ONNX → rerank in PyTorch → final answer in TensorRT-LLM.
• Multi-framework: ONNX, PyTorch, TensorFlow, TensorRT, vLLM, Python, custom C++.
• Per-model config.pbtxt with dynamic batching, instance groups, scheduling policies.

Sample config for an LLM via Triton's vllm backend:

# /models/llama-70b/config.pbtxt
backend: "vllm"
max_batch_size: 0      # vLLM does its own batching

model_transaction_policy {
  decoupled: true      # streaming responses
}

instance_group [
  { count: 1, kind: KIND_MODEL }
]

parameters: {
  key: "model"           value: { string_value: "meta-llama/Llama-3.3-70B-Instruct" }
}
parameters: {
  key: "tensor_parallel_size" value: { string_value: "2" }
}
parameters: {
  key: "gpu_memory_utilization" value: { string_value: "0.92" }
}

Triton has its own dynamic batching for non-LLM models (vision, embedding, classification):

# /models/embedding/config.pbtxt — dynamic batching for an embedding model
backend: "onnxruntime"
max_batch_size: 64

dynamic_batching {
  preferred_batch_size: [ 16, 32, 64 ]
  max_queue_delay_microseconds: 2000   # 2 ms wait window
}

instance_group [
  { count: 2, kind: KIND_GPU }
]

max_queue_delay_microseconds is the latency-vs-throughput knob for static-batched models. 1-5 ms is the typical operator's range.

TGI and SGLang: where they fit

• TGI (HuggingFace) — easy onboarding, smooth integration with HF Hub model identifiers, decent performance. Less aggressive than vLLM on KV management; some shops still prefer it for simplicity.
• SGLang — RadixAttention shares KV across many concurrent requests with overlapping prefixes; very strong for agent workloads (one base prompt, many tool-use trajectories) and structured output (JSON-schema-constrained generation).

Both are valid choices. Neither is the default for "I need the most tokens/sec on this H100."

Batching: the most important policy

Two regimes dominate:

Static (request-level) batching

The server collects requests for a fixed window (e.g. 5 ms) or until a batch is full, then runs the whole batch through the model. Used by Triton's dynamic_batching for non-LLM models.

Pros: simple, deterministic, great for fixed-shape workloads (image classification, text embedding). Cons: catastrophic for LLMs — short sequences wait for long sequences in the same batch, and batch padding wastes compute on sequences shorter than the longest.

Continuous (iteration-level) batching

The server runs one decode step at a time across all in-flight requests. After each step, finished requests are removed and new requests are admitted. The batch is recomputed every iteration.

Pros: no padding waste; new requests start immediately; latency-throughput Pareto frontier is much better. Cons: implementation is harder; KV cache management must be page-based (PagedAttention or equivalent).

Every modern LLM server does continuous batching: vLLM, TensorRT-LLM, TGI, SGLang. If you're evaluating a server and it doesn't, that's a 2× throughput penalty.

Chunked prefill and prefill/decode interleaving

A subtler batching choice: when a new request arrives, you have to prefill its prompt (process all input tokens through the model). Prefill is compute-heavy; if you naively interrupt decode to do it, decode latency spikes.

• Chunked prefill — split a long prompt into chunks (e.g. 512 tokens), interleave chunks with decode iterations. Smooths latency at the cost of slightly lower throughput.
• Prefill/decode disaggregation (PD) — different GPU pools run prefill and decode. KV cache transfers between them via NIXL. Best p99 latency; most operational complexity.

vLLM and TensorRT-LLM both support chunked prefill. PD-disaggregation is a deployment-architecture choice; see the disaggregation section below.

KV cache management

The KV cache is the bottleneck on inference HBM. Three optimizations matter:

PagedAttention

Treats the KV cache as fixed-size pages (e.g. 16 tokens each) in a pool. Each request holds a list of page indices. Eliminates fragmentation — without it, naive allocation leaves holes that prevent admitting new requests.

vLLM invented this; TensorRT-LLM and SGLang have their own equivalents.

Prefix caching

If two requests share a prompt prefix (e.g. system prompt + few-shot examples), the KV cache for that prefix can be shared instead of recomputed.

vllm serve <model> --enable-prefix-caching

Free win for chat: dozens of users hitting the same system prompt → only the first pays the prefill cost.

KV cache offload

When HBM is full, the server can either:

1. Evict a request's KV cache (must re-prefill if the request comes back).
2. Offload to host RAM or NVMe and pull back on demand.

Offload is shipped in vLLM (CPU offload) and TensorRT-LLM (CPU + NVMe). The transport for cross-node offload is NIXL. Operationally:

• HBM → host RAM: PCIe Gen5 x16 = ~50 GB/s. A 1 GB KV chunk moves in 20 ms.
• HBM → NVMe via GDS: ~10 GB/s per drive. Slower; only for cold cache.
• HBM → remote node via RoCE 400G: ~25 GB/s realistic. PD-disaggregation territory.

The decision is per-workload: long-context summarization with low concurrency benefits enormously from offload; high-concurrency short-prompt traffic does not.

Speculative decoding

The trick: a small draft model generates K tokens speculatively; the big model verifies them in parallel in a single forward pass. If all K accept, you got K tokens for the cost of one. Typical numbers: 2-3× decode throughput on common workloads.

Operational impact:

• Two models in HBM — the draft (often a 1-7B model) eats HBM. Your 70B + 8B draft on one H200 leaves less KV headroom.
• Verification logic adds CPU work in the scheduler. p50 stays similar; p99 sometimes improves (less queueing per request).
• Acceptance rate is workload-dependent. Boilerplate text → 80%+ accept → big speedup. Highly creative generation → 30% accept → smaller speedup, sometimes net loss.
• Frameworks vary in support. vLLM has speculative decoding; TensorRT-LLM has it via Medusa or EAGLE-style modules; SGLang has it; TGI is improving.

# vLLM with a draft model
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --use-v2-block-manager

Verdict: turn it on, measure p50/p99/throughput, decide. It is not a free lunch but is a frequent win.

Multi-LoRA serving

LoRA adapters are tiny (10-200 MB) compared to the base model. Multi-LoRA serving loads one base model and N adapters; each request specifies which adapter to apply.

Why it matters: serving 50 fine-tuned variants of a base model used to require 50 separate deployments (and 50× the HBM). Multi-LoRA collapses that to one.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --max-loras 8 \
  --max-lora-rank 64 \
  --lora-modules \
    '{"name": "sql-coder", "path": "/loras/sql-coder"}' \
    '{"name": "support-bot", "path": "/loras/support-bot"}'

Knobs:

• --max-loras — number of adapters held in HBM concurrently. Each adapter that's "warm" is ready to serve without reload.
• --max-lora-rank — set to the largest rank across your adapters. Higher rank = more HBM per adapter.
• Dynamic load/unload at runtime via POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter (set VLLM_ALLOW_RUNTIME_LORA_UPDATING=True).

Performance tax:

• Per-batch overhead: kernels merge LoRA matmuls with base; throughput drops 5-15% versus base-only depending on rank and batch composition.
• HBM tax: max_loras × adapter_size permanent allocation.

For most inference platforms the trade is excellent — N LoRAs on one deployment is dramatically cheaper than N deployments.

Autoscaling on the right metric

The default impulse is to autoscale on DCGM_FI_DEV_GPU_UTIL. Don't. Continuous-batching servers run at 80%+ GPU util at very low load — they're keeping the GPU warm, not saturated. By the time util reaches 95%, your queue is already deep and p99 has blown out.

Better metrics:

A reasonable HPA pattern:

metrics:
- type: Pods
  pods:
    metric:
      name: vllm:num_requests_running
    target:
      type: AverageValue
      averageValue: "200"
- type: Pods
  pods:
    metric:
      name: vllm:gpu_cache_usage_perc
    target:
      type: AverageValue
      averageValue: "0.85"
behavior:
  scaleUp:
    stabilizationWindowSeconds: 60
  scaleDown:
    stabilizationWindowSeconds: 300

Two metrics so a single noisy signal doesn't trigger a scale event. Slow scale-down matters: model load takes 30-90 s for 70B; scaling down too fast then back up costs you a cold start during the recovery.

See: Inference vs training for why GPU util misleads here.

API gateway and routing

In front of the model server pods sits an API gateway. What it does:

• Auth — API keys, JWT, mTLS for service-to-service.
• Rate limiting — per-user / per-tenant tokens-per-minute and requests-per-minute. Token-aware rate limits (cost the user N tokens, not N requests) prevent abuse via long-prompt requests.
• Routing — model name → which deployment. Sometimes prefix-aware (route to the pod that has the KV cache for this prompt prefix).
• Traffic shaping — canary deploys, A/B, model rollback.
• Observability — request logs, billing events, SLO measurement.

Common choices:

For prefix-aware routing the rule of thumb: if your traffic has shared prefixes (chat, RAG with shared context), Dynamo or a LiteLLM-style sticky router pays off. If every request is unique (one-shot generation), simpler load balancing suffices.

Prefill/decode disaggregation

The disaggregated topology splits prefill and decode onto different pools:

        ┌──────────────┐  KV via NIXL ┌──────────────┐
client ─►  prefill pool ───────────────►  decode pool  ─► tokens out
        │  (compute-heavy) │             │ (memory-heavy) │
        │  big batches OK  │             │ long-running   │
        └──────────────┘                 └──────────────┘

When it pays:

• Workloads with very different prefill/decode ratios — e.g. RAG (long prompt, short answer) wants more prefill; chat (medium prompt, long answer) wants more decode.
• Bursty prefill — autoscale prefill independently of decode; latency for new requests stays low even as decode pool stays saturated.
• Different HW for each — prefill can run on H100s, decode on H200s with bigger HBM for KV.

Operational cost:

• More moving parts. Two deployments, a router (Dynamo), KV transport (NIXL).
• RDMA fabric required for cross-pool KV transfer at any reasonable rate.
• Failure modes multiply — what happens if the decode pod dies mid-stream after KV arrives?

Recommendation: don't disaggregate until co-located vLLM doesn't meet your p99 — then it's a real tool.

Quantization and dtype

Worth a paragraph because it changes everything in the stack:

• FP16/BF16 — baseline. 2 bytes per param. 70B = 140 GB.
• FP8 (H100, H200, B-series only) — 1 byte/param. 70B = 70 GB. Quality near-equivalent for most chat workloads.
• INT8 / INT4 — 1 / 0.5 bytes per param. Quality varies more; often paired with calibration (AWQ, GPTQ).
• MXFP4 (B-series) — 0.5 bytes/param with a hardware-supported low-precision format.

Operationally:

• The runtime must support the dtype. TensorRT-LLM has the broadest support; vLLM has FP8/INT8/INT4 with various flags.
• Quantization-aware deployments often use calibrated checkpoints — you can't just point at FP16 weights and ask for FP8.
• Model quality regression should be measured on your eval set, not the model card's.

Common production failure modes

• OOM at HBM peak. KV cache pool sized to nominal load, traffic spikes, KV exhausted, request errors. Mitigation: oversize pool, set --max-num-seqs conservatively, watch gpu_cache_usage_perc.
• First-token latency P99 cliff. New request waits behind a long prefill. Mitigation: chunked prefill, smaller --max-num-batched-tokens.
• Cold start on scale-up. Pod scheduled, model load takes 60 s, traffic was already redirected. Mitigation: longer initialDelaySeconds, pre-warmed warm pool, slower scale-down.
• Rolling update tail latency. New pod starts → old pods drained → traffic lands on warming pods. Mitigation: surge to extra replicas during deploy; don't replace > 25% at a time.
• Prefix cache cold across replicas. Each pod has its own prefix cache; traffic load-balanced round-robin → no shared prefix cache. Mitigation: sticky routing on user ID or prefix hash, Dynamo-style router.
• GPU got stuck in reset state. Bad kernel from one tenant (under MPS/time-slicing) bricks the GPU; pod restart doesn't fix; node needs nvidia-smi --gpu-reset or reboot. Mitigation: MIG isolation for hostile tenants; node-problem-detector to drain stuck nodes.

See: K8s GPU pod failures, Triage decision tree.

Choosing a framework: short opinions

• Off-the-shelf LLM, throughput matters, want it running today → vLLM.
• Squeezed vLLM, need 30-50% more, willing to compile → TensorRT-LLM in Triton.
• Many models, mixed framework zoo, ensemble pipelines → Triton.
• HF-centric, simple ops → TGI.
• Heavy structured output / agent workloads / shared prefixes → SGLang.
• Multi-tenant fleet of fine-tunes → vLLM with multi-LoRA.
• Need PD-disaggregation → vLLM + NIXL + Dynamo router.

There is no globally best framework; there is a best framework for your traffic shape. Benchmark on your traffic — public benchmarks lie when applied to a workload they didn't measure.

See also

• Inference vs training — why the inference cluster looks different
• GPU sharing: MIG vs MPS vs time-slicing vs vGPU — when to slice the GPU under inference
• NIXL — the KV cache transport for PD-disaggregation and offload
• NIM, NeMo, Triton — the NVIDIA stack and how it overlaps with the OSS field
• DCGM — the metric pipeline behind autoscaling
• GPU generations — the HBM-vs-FLOPs question per SKU
• K8s GPU pod failures — what to look at when a model server falls over
• vLLM docs: https://docs.vllm.ai/
• TensorRT-LLM: https://github.com/NVIDIA/TensorRT-LLM
• Triton Inference Server: https://github.com/triton-inference-server/server
• TGI: https://github.com/huggingface/text-generation-inference
• SGLang: https://github.com/sgl-project/sglang