NIXL (NVIDIA Inference Xfer Library): fast tensor transport for distributed inference

NCCL is for training: collective ops over the entire job's GPU set, latency-tolerant, throughput-optimal. Inference has a different communication pattern — point-to-point KV cache transfers between prefill and decode workers, on-demand tensor moves between heterogeneous memory tiers (GPU, host RAM, NVMe, remote object store), latency-sensitive, asynchronous. NCCL was designed before that problem existed at this scale. NIXL (NVIDIA Inference Xfer Library) is NVIDIA's answer.

Why a new library

The shape of LLM inference workloads changed fast in 2023-2024:

1. Prefill / decode disaggregation. Prefill (processing the prompt) is compute-bound; decode (generating tokens) is memory-bound. Running them on different GPU pools — sized differently, scaled independently — is now standard. But disaggregation requires moving KV cache from a prefill worker to a decode worker, fast, sometimes across nodes.
2. KV cache offload to host or NVMe. Long-context (100K+) workloads spill KV cache to slower tiers and pull it back on demand. That's not a collective; it's a point-to-point copy from "wherever the cache lives" to the GPU.
3. Tensor parallelism across heterogeneous compute. Some new inference architectures shard work across GPU + DPU + remote storage in patterns NCCL doesn't model.
4. Smart routers / KV-aware schedulers. NVIDIA Dynamo, vLLM, and similar systems route requests to whichever worker already has the KV cache for that prompt prefix — requiring cross-worker coordination on tensor moves.

The unifying theme: point-to-point, asynchronous, heterogeneous memory transport — for which NCCL is not a clean fit.

What NIXL is

NIXL is an open-source C++/Python library (under ai-dynamo/nixl on GitHub) that exposes:

• A uniform memory abstraction across GPU memory, CPU RAM, NVMe (via GPU Direct Storage), and remote storage (object stores, file systems).
• Asynchronous point-to-point reads and writes between any registered buffer.
• Pluggable backend plugins that select the best transport (UCX over RDMA / TCP, NVLink P2P, GDS, POSIX, Mooncake, S3-compatible).

A NIXL transfer abstracts over "what hardware path do I take" — UCX picks RDMA when both ends are on InfiniBand, falls back to TCP otherwise; GDS gives you direct GPU↔NVMe DMA without bouncing through host RAM. The application code is the same.

It is announced ~2024, evolving fast, and used in production by NVIDIA Dynamo. Treat it as "stable enough for production but expect API churn between minor versions" — pin your dependency.

Core concepts

Agent

A NIXL agent is the per-process handle. Two agents can talk to each other if they share a backend transport (UCX in the same fabric, GDS for the same shared filesystem, etc.). An agent registers buffers and posts transfers; remote agents receive notifications.

Backends (plugins)

UCX is the primary one and what most production deployments use today. NIXL is tested against specific UCX versions (currently the 1.21.x series) — version-skew with system UCX bites.

Memory descriptors

A NIXL transfer is described by memory descriptors that say "this region of bytes lives in this memory type at this address". The library handles registration with the appropriate hardware (pinning, MR registration for RDMA, GDS handle for NVMe).

# pseudo-Python — simplified for clarity
import nixl

agent = nixl.Agent(name="prefill-worker-1", backends=["ucx"])

# register a GPU buffer
gpu_buf = torch.empty(1024 * 1024 * 16, dtype=torch.float16, device='cuda')
gpu_desc = agent.register(gpu_buf, mem_type=nixl.MEM_GPU)

# discover the remote agent (out-of-band exchange — your own logic)
remote_agent_info = control_plane.get("decode-worker-3")
agent.connect(remote_agent_info)

# post an async write to remote agent's pre-registered buffer
xfer = agent.post_xfer(
    op=nixl.WRITE,
    local_desc=gpu_desc,
    remote_agent="decode-worker-3",
    remote_desc=remote_agent_info["kv_buffer_desc"],
)

# poll / wait for completion
while not xfer.completed():
    time.sleep(0.0001)

The actual API is more involved (handles, notifications, partial transfers); this captures the shape.

How NIXL differs from NCCL

It's not "NIXL replaces NCCL" — they coexist. A modern training+inference stack uses both: NCCL for training all-reduce, NIXL for inference KV transfer.

Use cases

vLLM PD-disaggregation

vLLM (and Dynamo on top of it) routes prefill to one pool of GPUs and decode to another. After prefill, the KV cache must move from a prefill worker to a decode worker before generation starts. NIXL does that move:

[Prefill worker]                              [Decode worker]
     │ runs prefill                                   │
     │ produces KV cache (~ 100 MB to several GB)     │
     │                                                │
     │  NIXL.post_xfer(WRITE, kv_buf → decode-N)      │
     ├────────────── UCX over IB/RoCE ────────────────►│
     │                                                │
     │ frees its KV; ready for next prefill           │ runs decode

For best efficiency, KV transfer overlaps with the start of decode: NIXL's async model lets the decode worker begin processing the first slabs of cache before the rest arrive.

Dynamo router with KV-aware routing

NVIDIA Dynamo's smart router knows which workers hold which KV prefixes. When a request matching a prefix comes in, the router prefers a worker that already has the cache. When it must move cache, NIXL is the transport.

Long-context KV offload

For 1M+ context windows, KV cache exceeds a single GPU's memory. The system spills cold cache to host RAM or NVMe (via GDS) and pulls it back on demand. NIXL's GDS backend does the GPU↔NVMe move directly, skipping the host RAM bounce that would otherwise saturate PCIe.

Custom inference frameworks

Any framework that needs to move tensors between processes (Ray Serve clusters, custom orchestrators) can use NIXL as the data-plane library, leaving service discovery / scheduling to the framework.

Operational realities

Installation

# Python wheel (matches DOCA / UCX of the host)
$ pip install nixl

# Or build from source for custom backend mix
$ git clone https://github.com/ai-dynamo/nixl
$ cd nixl && cmake -B build -DBACKENDS="UCX;GDS;POSIX" && cmake --build build
$ cmake --install build

NIXL needs UCX of a compatible version installed on the system (and matching on every node). UCX in turn needs OFED / DOCA-Host for full RDMA support.

Backend selection at runtime

agent = nixl.Agent(name="worker-1", backends=["ucx", "gds", "posix"])
# NIXL picks the best backend per (src_mem, dst_mem, peer) pair

The library has heuristics for which backend to use for which transfer type. You can force selection with hints when debugging.

What to monitor

NIXL doesn't ship its own Prometheus exporter (yet). You're scraping the underlying transports:

• UCX: ucx_perftest-derived counters; UCX has telemetry plugins. RDMA throughput per QP via mlnx_perf / NIC counters.
• GDS: gdscheck and the GDS stats in /proc/driver/nvidia-fs/stats.
• Application-level: instrument your prefill/decode code to histogram NIXL transfer sizes and durations.

Sizing — bandwidth and latency

For prefill/decode disaggregation, the KV cache transfer is on the critical path. Calculate:

KV size per layer per token = 2 (K, V) × num_heads × head_dim × dtype_bytes
KV size per request          = KV_per_layer × num_layers × num_tokens

For Llama-3-70B at 4096 tokens in FP16: ≈ 0.5 MB per layer × 80 layers × 4096 tokens ≈ 1.6 GB of KV per long prefill. Move that over 200 Gb RoCE (~25 GB/s realistic) ≈ 65 ms. Plan your fabric accordingly: 400 Gb fabrics double the budget.

Common gotchas

• UCX version mismatch. NIXL expects a specific UCX. System UCX from your distro is usually older. Solution: ship UCX in the container or build into NIXL.
• OFED / DOCA-Host mismatch on the host. UCX uses libibverbs which uses OFED; if the host kernel modules disagree with the userspace OFED libraries, UCX falls back to TCP silently. NIXL transfers will work but at 10x lower bandwidth. Test with ucx_perftest between two nodes before debugging NIXL itself.
• GDS not enabled. GDS requires nvidia-fs kernel module loaded + NVMe device on a supported topology. Without it, GDS backend silently falls back to host-staged copies (no benefit).
• MR registration cost on first transfer. NIXL registers memory regions lazily; first transfer to a new buffer can be 10× slower than steady-state because of MR pinning. Pre-register buffers at init time when latency-sensitive.
• Firewall / fabric ACLs. UCX's TCP fallback uses ephemeral ports; if you're locked-down, RDMA paths must come up. Test with ucx_info -d and confirm the IB / RoCE devices are visible to UCX.
• Out-of-band agent discovery. NIXL doesn't ship a service discovery mechanism — agents need to exchange connection info via your own control plane (etcd, K8s service, gRPC handshake). Don't expect a built-in "join this group of agents" call.

Versioning posture

NIXL is young. As of 2025 it's at major version 0/1, evolving fast. Operational implications:

• Pin the version: don't do pip install nixl in production; pin a tag.
• Read changelogs every minor release: APIs change, backend names change, performance defaults shift.
• Co-version with NIXL consumers: if Dynamo expects NIXL 0.x, deploying 1.x will break.

What NIXL is not

• Not a scheduler. It does the bytes-on-the-wire part; you decide which agent gets which work.
• Not a service discovery mechanism. Bring your own.
• Not a NCCL replacement for training. For collectives, use NCCL or a successor.
• Not a high-level KV cache manager. vLLM / Dynamo / your framework manage KV cache lifecycle and call NIXL when they need to move it.

Quick mental model

┌─────────────────┐   ┌───────────────────────┐   ┌────────────────┐
│ application     │ → │ NIXL                  │ → │ transport       │
│ (vLLM, Dynamo)  │   │  agent / register /   │   │ UCX, GDS, POSIX │
│                 │   │  post_xfer            │   │                 │
└─────────────────┘   └───────────────────────┘   └────────────────┘
                                                           │
                                                           ▼
                                                   ┌─────────────────┐
                                                   │ HW: IB/RoCE,    │
                                                   │ NVMe, NVLink,   │
                                                   │ TCP             │
                                                   └─────────────────┘

Application writes "move this tensor to that agent"; NIXL picks the best path; the transport moves the bytes; completion is notified back to the application.

See also

• NCCL multi-node tuning — the training-side counterpart
• NIM / NeMo / Triton inference stack — what NIXL is plumbing for
• DOCA + BlueField — the fabric NIXL often runs over
• GPU generations — affects available transports (NVLink 5, GDS support)
• NIXL upstream: https://github.com/ai-dynamo/nixl
• NVIDIA Dynamo overview: https://developer.nvidia.com/blog/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models/