NIM, NeMo, Triton: the NVIDIA inference + training stack

NVIDIA's AI software portfolio is wide and the names overlap. Operators get tickets like "deploy a NeMo on Triton via NIM" and have to untangle what was meant. This page is a clear partition: Triton is a model-serving runtime, NeMo is a training framework, NIM is a packaged inference microservice. They complement each other; they are not interchangeable.

The three products in one paragraph

You will often run all three in the same cluster: NeMo trains a model → checkpoint goes to a model repo → Triton serves it (or you wrap it as a private NIM). Or, for off-the-shelf models, just pull and serve a NIM and skip the rest.

Triton Inference Server

What it is

A C++ inference server that loads models from a model repository and exposes them over HTTP and gRPC. Backends plug in to support different frameworks:

Why operators care

Triton is the most general-purpose server in the NVIDIA stack. If you're not running off-the-shelf foundation models from NIM, you almost certainly have a Triton somewhere.

Reliable Triton deployment requires understanding:

• The model repository layout (filesystem or S3/Azure/GCS).
• Model configuration (config.pbtxt) per model — input/output tensors, dynamic batching, instance count, scheduler policy.
• Concurrency: instance groups (replicas of the model on different GPUs) vs dynamic batching (combining requests in flight).

Model repository

/models/
├── bert-classifier/
│   ├── config.pbtxt
│   └── 1/
│       └── model.onnx
├── llama-3-8b/
│   ├── config.pbtxt
│   └── 1/
│       └── model.plan         # TensorRT engine
└── preprocess/
    ├── config.pbtxt
    └── 1/
        └── model.py           # python backend

Triton watches the directory; adding a 2/ folder under bert-classifier/ automatically loads that as version 2 (with a configurable load policy: latest, all, specific).

Minimal config.pbtxt

name: "bert-classifier"
platform: "onnxruntime_onnx"
max_batch_size: 32
input  [{ name: "input_ids", data_type: TYPE_INT64, dims: [128] }]
output [{ name: "logits",    data_type: TYPE_FP32, dims: [3]  }]

dynamic_batching {
  preferred_batch_size: [8, 16, 32]
  max_queue_delay_microseconds: 5000
}

instance_group [{ count: 2, kind: KIND_GPU, gpus: [0] }]

This says: serve up to batch 32, prefer batches of 8/16/32 collected within 5 ms, run two model replicas on GPU 0 to overlap copies and compute.

Dynamic batching — why it's the killer feature

Inference workloads come in as small concurrent requests; the GPU is most efficient on big batches. Dynamic batching queues incoming requests for up to a small delay (microseconds), then runs them as one batch. With proper tuning you go from 30% GPU utilization to 80%+ at the same p99 latency.

Tuning knobs:

• preferred_batch_size — sizes the scheduler aims for.
• max_queue_delay_microseconds — how long it'll wait to fill a batch.
• priority_levels + default_priority_level — multi-class priority queues.

Deployment shape on K8s

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: triton
        image: nvcr.io/nvidia/tritonserver:24.10-py3
        args: ["tritonserver", "--model-repository=/models", "--strict-model-config=false"]
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 32Gi
        ports:
        - { name: http, containerPort: 8000 }
        - { name: grpc, containerPort: 8001 }
        - { name: metrics, containerPort: 8002 }
        readinessProbe:
          httpGet: { path: /v2/health/ready, port: 8000 }
        volumeMounts:
        - { name: models, mountPath: /models, readOnly: true }

Metrics on :8002 are Prometheus-format — scrape them. The nv_inference_request_success, nv_inference_queue_duration_us, nv_inference_compute_infer_duration_us histograms are what you want for SLO dashboards.

KV cache for LLM backends

tensorrt_llm and vllm backends maintain a KV cache for transformer attention — the dominant memory consumer for LLM inference. Knobs:

• kv_cache_free_gpu_mem_fraction (TRT-LLM): fraction of free VRAM the engine claims for KV.
• gpu_memory_utilization (vLLM): same idea.
• max_num_seqs / max_seqs: how many concurrent sequences the cache holds.

Get this wrong and you OOM at high concurrency, or you under-utilize VRAM. Sane defaults are 0.85-0.90 of free VRAM. Watch with DCGM DCGM_FI_DEV_FB_USED.

NeMo — the training framework

NeMo is a PyTorch-based framework for training and fine-tuning large neural networks: LLMs, speech (ASR/TTS), multimodal, vision-language. It is opinionated: it integrates Megatron-LM (model parallelism), TransformerEngine (FP8), Apex (optimizer state sharding), and ships configs / recipes for known model families.

Why operators care

You're rarely the one writing NeMo configs, but you are the one keeping NeMo training jobs alive. That means:

• NeMo containers are huge (8-15 GB) and pull from nvcr.io/nvidia/nemo. Use a pull-through cache mirror in your registry, or training jobs will be cold-start-bottlenecked.
• NeMo workloads use Megatron tensor + pipeline parallelism + sequence parallelism. They are unusually sensitive to NCCL tuning, NVLink topology files, and PCIe topology — see NCCL multi-node tuning.
• Checkpoints are big: a 70B-parameter model is ~280 GB on disk in FP16 + optimizer state ≈ 1 TB. NeMo does sharded checkpoints; storage backend needs to handle parallel write from N ranks. Lustre / Weka / DAOS are common; NFS struggles past 32 ranks.
• Resume semantics: NeMo+PTL supports periodic checkpoints + auto-resume. A node failure mid-run should resume from the last checkpoint without manual intervention if the storage and the launcher are configured right.

NeMo on K8s — typical shape

# simplified PyTorchJob (Kubeflow operator)
apiVersion: kubeflow.org/v1
kind: PyTorchJob
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 16          # 16 nodes × 8 GPUs = 128 GPUs
      template:
        spec:
          containers:
          - name: pytorch
            image: nvcr.io/nvidia/nemo:24.07
            command: ["bash", "-c"]
            args: ["torchrun --nproc_per_node=8 ... examples/nlp/language_modeling/megatron_gpt_pretraining.py ..."]
            resources:
              limits:
                nvidia.com/gpu: 8
            volumeMounts:
            - { name: data,        mountPath: /data,        readOnly: true }
            - { name: checkpoints, mountPath: /checkpoints }

NeMo is also commonly launched via NeMo Launcher (Slurm) or NeMo Curator (data prep). Operators see those mostly as "ops dashboards have lots of node-busy time".

Recent: NeMo Microservices

NVIDIA has been bundling individual stages (NeMo Curator, NeMo Customizer, NeMo Evaluator, NeMo Guardrails) as separate microservices. That makes the platform composable but also means more services to operate.

NIM — packaged inference microservices

What it is

NIM (NVIDIA Inference Microservice) is a container image per model + recipe, optimized for NVIDIA GPUs, exposing an OpenAI-compatible HTTP API. Think "Llama-3.1-70B-Instruct, but pre-tuned for H100 with TensorRT-LLM, served by Triton, packaged and shipped".

NIMs exist for:

• LLMs: Llama 3 / 3.1 / 3.2 / 3.3, Mistral, Mixtral, Phi, Gemma, Nemotron, DeepSeek (some)
• Speech: Riva ASR / TTS
• Vision: NV-CLIP, embedding models
• Retrieval: NVIDIA Retriever embeddings + reranker
• Safety: NeMo Guardrails

NIM containers are pulled from nvcr.io/nim/ and require an NVIDIA AI Enterprise entitlement (a license — talk to NVIDIA sales). Activation is via a key passed at run time.

Why use NIM over building your own

Deployment shape — NIM on K8s

NVIDIA ships a NIM Operator that handles model caching, helm releases, autoscaling. The basic pattern:

apiVersion: apps.nvidia.com/v1alpha1
kind: NIMService
metadata:
  name: llama3-8b-instruct
spec:
  image:
    repository: nvcr.io/nim/meta/llama3-8b-instruct
    tag: 1.0.3
  authSecret: ngc-api
  storage:
    nimCache: { name: llama3-cache }     # PVC with model weights
  replicas: 2
  resources:
    limits: { nvidia.com/gpu: 1 }
  expose:
    service: { type: ClusterIP, port: 8000 }

The Operator pulls model weights once into a shared PVC (NIM caches: nimCache), then pods mount read-only — saves repeated pulls of multi-GB weights at scale.

HPA on token throughput

Standard CPU/memory HPA doesn't make sense for inference. Real metrics:

• num_requests_running (current concurrency).
• time_to_first_token (latency SLO).
• tokens_per_second (throughput).

NIM exposes these via Prometheus. With prometheus-adapter + HPA on external metrics, you can scale on TTFT p99 or queue depth.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef:
    apiVersion: apps.nvidia.com/v1alpha1
    kind: NIMService
    name: llama3-8b-instruct
  minReplicas: 2
  maxReplicas: 16
  metrics:
  - type: Pods
    pods:
      metric: { name: num_requests_running }
      target: { type: AverageValue, averageValue: "8" }

When to use which — decision flow

Do you need to TRAIN or FINE-TUNE?
   yes → NeMo. Period.
   no  →
        Is it an off-the-shelf model in NVIDIA's catalog AND you have AI Enterprise?
           yes → NIM. Pull and run.
           no  →
                Custom / private model OR no NIM available?
                   yes → Triton. Pick a backend (TensorRT-LLM for LLM, ONNX for vision, etc.)

Common gotchas

Observability checklist

For each of these, you want at least one dashboard panel:

• Triton: nv_inference_request_duration_us (p50/p95/p99), nv_inference_pending_request_count, dynamic batch sizes histogram, GPU utilization (DCGM).
• NIM: time_to_first_token, time_per_output_token, num_requests_running, gpu_cache_usage_perc.
• NeMo: training step time, samples/sec, loss + grad-norm, NCCL all-reduce time, checkpoint write duration, GPU SM_ACTIVE / TENSOR_ACTIVE.

See also

• GPU generations — picking GPUs for inference vs training
• DCGM monitoring
• MIG for inference multi-tenancy
• NIXL for distributed inference
• NCCL multi-node tuning
• NIM docs: https://docs.nvidia.com/nim/
• Triton docs: https://github.com/triton-inference-server/server
• NeMo docs: https://docs.nvidia.com/nemo-framework/