DOCA + BlueField DPU: offloading network/storage from the host CPU

A DPU is the third leg of the modern compute stool: CPU runs your application, GPU runs accelerated math, DPU runs everything else — networking, security, storage, infrastructure services. NVIDIA's DPU line is BlueField; the SDK to program it is DOCA. This page is what BlueField is, why you'd deploy it, and what changes when you move from a "smart NIC" mental model to a real DPU.

What a DPU actually is

A DPU is not a smart NIC with extra silicon, even though that's how it gets sold. The right mental model:

A DPU is a standalone ARM-based server with its own memory, OS, and OOB management — that just happens to also be the NIC for the host it sits inside.

Concretely a BlueField-3 has:

• A ConnectX-7 NIC (400 Gb Ethernet / 400 Gb InfiniBand).
• 16 ARM Cortex-A78 cores.
• Up to 32 GB of DDR5 RAM (DPU-local, not host RAM).
• Hardware accelerators for: regex, crypto (IPsec, TLS, AES, SHA), compression, DPA (data-path accelerator).
• An eMMC for boot, plus optional onboard NVMe.
• A BMC for out-of-band management.
• PCIe Gen5 x16 connection to the host.

It runs its own Linux distribution (BlueField OS, Ubuntu-based) entirely independent of the host. From the host you see a NIC; the DPU itself sees the host as a "PF on the other side of PCIe".

Why DPUs exist

Three trends converged:

1. East-west traffic exploded — LLM training nodes do 400 Gbps per node of all-reduce, AI inference does cross-node KV transfer (see NIXL). Software OVS in the host kernel can't move that without burning every CPU core.
2. Multi-tenancy needs hard isolation — cloud providers can't trust the host to enforce tenant boundaries; the DPU does it from outside the host trust domain.
3. GPU servers should not waste cycles on infra — every CPU core spent on networking is a core that's not feeding the GPU. Offloading frees the host for what it's there for.

BlueField generations

(BlueField-1 existed pre-acquisition under Mellanox and is unsupported.)

For most operators today the relevant unit is BlueField-3. BlueField-2 is still in the field but newer features (DPA, advanced DOCA libraries) target -3+.

DOCA — what it gives you over raw OFED

Without DOCA, programming a BlueField means using:

• iproute2 / tc for OVS.
• IBverbs + librdmacm for RDMA.
• DPDK for fast-path packet processing.
• Custom code for crypto offload via kernel xfrm or hardware-aware userspace.

This is the same stack you'd use on any Mellanox NIC. It works but it's a kit of unrelated APIs.

DOCA replaces and unifies them with:

There are also reference applications (production-grade examples that customers actually deploy as-is): doca_ipsec_security_gw, doca_dpdk_forward, doca_url_filter, etc.

The main practical wins:

• One toolkit, one versioning story. OFED, DOCA, BlueField OS all version together.
• Forward-compat: DOCA apps written for -3 should run on -4 with rebuild.
• DPA programmable accelerator: lets you push code onto a small data-path accelerator core for sub-microsecond packet handling — without DOCA you can't access it.

Deployment modes — the most operationally important choice

A BlueField can run in three configurations. Picking the wrong one is the #1 source of "I bought DPUs and got nothing".

1. NIC mode (DPU disabled / passthrough)

DPU runs no offload; ARM is idle. The card behaves as a plain ConnectX-7. Use case: you bought a BlueField but you only need the NIC, you'll enable DPU later. From the host's view it's a normal NIC.

2. Embedded function (DPU mode) — common in cloud

The DPU runs OVS, IPsec, security stack on the ARM cores. Host OS sees VF representors rather than the NIC itself. Tenant traffic flows host VF → DPU OVS → wire, with the host having no ability to see other tenants' traffic. This is what AWS Nitro / Azure SmartNIC topologies do.

host kernel
  └─ VF representor on PCIe
     │  (host can only do what the DPU lets it)
     ▼
  BlueField OS
   └─ OVS-DPDK + DOCA flow + IPsec → physical wire

Operator tasks: install BFB image (BlueField boot image) on the DPU, configure DPU OS via cloud-init, monitor DPU as a separate "host" in your fleet inventory.

3. Separated host (DPU as Kubernetes node)

The DPU joins the K8s cluster as its own node alongside the host. Pods can be scheduled to the DPU. Use case: infrastructure pods (CNI, service mesh data plane, observability collectors) run on the DPU, application pods run on the host. The host CPU is fully reserved for tenant workloads.

                K8s control plane
                   │       │
                   ▼       ▼
              host-node    dpu-node    (both members of the same cluster)
              (CPU+GPU)    (DPU ARM, 16 cores)
              tenant pods  infra pods   (CNI, service mesh, telemetry)

This is the NVIDIA AI cloud reference design for new GPU clusters: each H100/H200 host has a paired BlueField-3 acting as the K8s infra plane, freeing host CPU for training.

What to actually offload

Storage: NVMe-oF target on the DPU

A common deployment: the DPU emulates a local NVMe namespace on the host's PCIe bus. Behind the scenes, the DPU forwards requests over RDMA to a remote storage backend (Weka, VAST, DAOS, custom).

host application
  read /dev/nvme0n1
        ↓ PCIe (native NVMe)
    BlueField (presents virtual nvme controller)
        ↓ RDMA over fabric
    Remote storage backend (anywhere)

To the host, it looks like a fast local SSD. To the storage team, all storage traffic is RDMA over the BlueField — host CPU does no I/O. NVIDIA calls this SNAP (Software-defined Network Accelerated Processing).

Image flashing & lifecycle

The DPU's OS is shipped as a BFB (BlueField Bundle) file: a single container of bootloader + kernel + rootfs + firmware + DOCA packages.

# from the host, with BlueField in DPU mode and rshim driver loaded
$ ls /dev/rshim*
/dev/rshim0/

# push a new BFB
$ bfb-install --rshim rshim0 --bfb DOCA_2.7.0_BSP_4.7.0_Ubuntu_22.04-3.20240625.bfb
Pushing bfb
2.34GiB 0:01:43 [23.1MiB/s]
INFO[FlashBFB]: Done
INFO[FlashBFB]: Reboot required

# reboot the DPU
$ ssh -p 22 ubuntu@bf3-mgmt sudo reboot

Operations:

• Pre-prod test the BFB. Reflashing a fleet of DPUs in production has the same blast radius as host kernel upgrades. Stage in a small ring first.
• BMC access is via the rshim console + an OOB Ethernet port on the BlueField. Get OOB networking right before you trust the DPU as your tenant data plane — if the DPU goes down, you may lose host networking entirely.
• Firmware vs BFB: NIC firmware is part of the BFB. Mismatch between NIC firmware and OFED/DOCA on the host shows up as "device not detected" or "QP create failed". mlxfwmanager reports versions.
• DOCA-OFED on the host: in DPU mode, the host needs DOCA-Host packages (a stripped-down OFED) instead of full MLNX_OFED. They co-exist poorly.

Telemetry

Two layers:

1. Host-visible NIC telemetry — same as ConnectX-7 — counters, RoCE health, link state. Scrape via mlnx_perf, sysfs counters, or dcgm-exporter if integrated.
2. DPU OS telemetry — the DPU is its own host. CPU usage of the ARM cores, memory pressure, OVS flow counts. Run a node_exporter on the DPU and scrape it as a separate Prometheus target.

# from inside the DPU OS
$ doca_telemetry_exporter --config /etc/doca-telemetry.yaml
# exposes per-flow counters, crypto throughput, DPA queue depths

When BlueField is and isn't worth it

Yes:

• Multi-tenant cloud / AI-cloud where you cannot trust the host with tenant separation.
• 400+ Gb fabrics where soft-OVS or kernel IPsec would burn unacceptable host CPU.
• Hyperconverged storage where you want NVMe-oF without a dedicated storage controller per host.
• Air-gapped HPC where you can't touch the host kernel for security but need policy enforcement at line rate.

No / questionable:

• Single-tenant clusters where the host is fully trusted — you pay for a card you don't need.
• < 100 Gb fabrics with light tenant counts; soft-OVS or DPDK on the host is enough.
• Workloads dominated by host CPU bottlenecks unrelated to networking.

Common operational gotchas

• DPU mode toggle is destructive. Switching between NIC mode and DPU mode reformats the eMMC. Plan a maintenance window; the host loses networking during the switch.
• DOCA-Host and MLNX_OFED conflict. Don't try to install both. Pick DOCA-Host on hosts that talk to BlueField-3 in DPU mode.
• Linux distro support is narrower than OFED. DOCA fully supports Ubuntu 22.04 / 24.04 and RHEL 8/9; other distros lag or are community-supported.
• Persistence of DPU OS configs. First-boot config (cloud-init / BFB customization) should write to /etc of the DPU rootfs. A reflash blows it away.
• Performance gotcha: by default OVS on BlueField-3 doesn't enable hardware offload. You must enable hw-offload in OVS for line-rate; otherwise the ARM cores process every packet and you're back to soft-switching at ~30 Gb max.
• Two mlx5 instances: in DPU mode, the host sees representor mlx5 devices, not the actual NIC. Many tooling assumptions about "the NIC at PCIe 0000:81:00.0 is the wire" break.

See also

• NVIDIA driver / OFED stack
• GPU generations
• NIXL — point-to-point transfer over UCX, often using BlueField
• Networking: NVLink, RoCE, and InfiniBand
• DOCA documentation: https://docs.nvidia.com/doca/sdk/index.html
• BlueField user guides: https://docs.nvidia.com/networking/category/bluefield