InfiniBand vs RoCE: when to pick which fabric

The first architectural decision in a new GPU cluster is: InfiniBand or RoCE. The hardware vendors will sell you both. The cluster will run training jobs either way. NCCL works on both. The difference shows up in operator effort, multi-tenancy story, ecosystem fit, and what your fabric can scale to without ripping out cables.

This page is the decision framework. It assumes you've read InfiniBand primer and RoCE v2 + DCB/PFC/ECN and want to know which to pick. The short answer at the end: it depends on who you are and what else is in your shop.

The honest comparison table

The asymmetry above is intentional: most lines lean IB-favorable on technical merits, RoCE-favorable on operational and economic merits. That's actually the choice you're making.

When you choose InfiniBand

Pick IB when these are true:

• Pure AI training / HPC focus. Your cluster's job is training, you don't share infra with general IT, the fabric is dedicated.
• You want SHARP. In-network reductions are a real 1.5-2x AllReduce improvement and there's no Ethernet equivalent in 2026. If you're memory-bandwidth bound on collective ops, this matters.
• Mature multi-tenant isolation matters. PKey-based tenant partitions are 20+ years mature on IB; the equivalent on RoCE (per-tenant DSCP queues + PFC profiles + L3 ACLs) is more brittle and easier to misconfigure.
• Single-fabric simplicity is a goal. One stack, one SM, one diagnostic toolchain. Less surface area to operate.
• Lower latency under load with less tuning. Native credit flow means an unconfigured IB fabric is already lossless. RoCE requires PFC + ECN configured everywhere or it falls over under congestion.
• You're following an NVIDIA reference architecture verbatim. DGX SuperPOD and DGX BasePOD reference architectures are IB-default.
• Latency-sensitive HPC workloads (CFD, lattice QCD, large-scale MD) where the few hundred nanoseconds of RoCE encap overhead matter.

If three or more of these are true, IB is the clearer pick.

When you choose RoCE

Pick RoCE when these are true:

• Mixed AI + general IP infrastructure. You want one fabric for both training and the rest of your DC. RoCE traffic sits alongside management, storage, and tenant Ethernet.
• Operator team is Ethernet-fluent. The team knows BGP, ECMP, VXLAN, EVPN. Adding IB means re-training (or hiring); adding RoCE-on-Eth means tuning a fabric they already understand.
• Multi-vendor procurement strategy. You don't want to be locked into one switch vendor for your whole AI fabric. Arista, Cisco, Juniper, Dell, NVIDIA all sell 400G/800G Ethernet switches that can do RoCE.
• Existing Ethernet management tooling. NetBox, Ansible network modules, Nautobot, your monitoring stack — all geared toward Ethernet, all extend cleanly to RoCE.
• Cost pressure is real. RoCE switches are typically 1.5-2x cheaper per port than IB at the same speed, and the cabling is more available off-the-shelf.
• You'll need to scale beyond what IB SM scales. At >5000 endpoints, opensm/UFM scaling becomes a real concern; standard L3 routing scales further with less custom infrastructure.
• Your storage stack is Ethernet (NVMe-oF over RoCE, S3-via-Eth, etc.). Keeping fabric and storage on the same protocol family is operationally simpler.

If three or more of these are true, RoCE is the right pick.

When the choice is hybrid

Some sites run both: IB for the compute fabric, Ethernet (with or without RoCE) for north-south, storage, and management.

   +----------------+         +----------------+
   |  GPU compute   |  -- IB -- |  GPU compute   |
   |  (rail 0..7)   |         |  (rail 0..7)   |
   +-------+--------+         +--------+-------+
           |                           |
           |                           |
       (IPoIB or                  (IPoIB or
        Eth NIC)                   Eth NIC)
           |                           |
           v                           v
   +-------------------------------------+
   |   Ethernet north-south / mgmt /     |
   |   storage / VPC / public            |
   +-------------------------------------+

This is the DGX SuperPOD reference pattern: IB for the training fabric, Ethernet for everything else (storage gateway, mgmt, OOB). Each node has IB HCAs (8x for compute) and Ethernet NICs (1-2x for control plane).

The hybrid pattern is common because it avoids the worst of both worlds — your compute fabric gets IB's predictable low latency and SHARP, your IT infrastructure stays Ethernet, and storage can be either depending on the storage system.

What you can't change later

This is the load-bearing point of the whole decision: the switch hardware is fixed once procured, and that decides your fabric type.

• HCA mode is selectable: mlxconfig -d /dev/mst/mt4129_pciconf0 set LINK_TYPE_P1=1 for IB, =2 for Eth. ConnectX-6/7/8 are physically dual-mode. So you can repurpose a NIC.
• Switch hardware is single-mode: a Quantum-2 QM9700 is an InfiniBand switch and only that. A Spectrum-4 SN5600 is an Ethernet switch and only that. You can't flip it.

So the decision is locked in at switch-procurement time. If you buy 64 Quantum-2 switches, you're running IB for the life of those switches (~5 years typical). If you buy 64 Spectrum-4 switches, you're running RoCE.

The corollary: mode-switching the HCAs without matching switches is a configuration error. We've seen people set LINK_TYPE_P1=2 to "try RoCE" on an IB-switch fabric and the link refuses to come up because the switch is speaking IB. The HCA and switch must agree.

Cost comparison (approximate, public-info only)

These are order-of-magnitude figures from published list prices and integrator quotes. Real cost varies hugely by volume, contract, and bundled support.

For a 1024-GPU cluster (64 switches + ~2000 cables), the fabric BOM difference is typically $1-3M IB vs RoCE. That's real money but small relative to the GPU spend ($30M+ for H100 NDR-class). Don't pick the fabric only on cost; pick it on the operational fit and let cost be a secondary input.

Real-world deployments

InfiniBand:

• NVIDIA DGX SuperPOD reference (every generation since DGX-1): IB-default for compute fabric.
• Selene (NVIDIA): HDR IB fat-tree, 4480 A100s.
• Eos (NVIDIA): NDR IB fat-tree, 4608 H100s.
• CINECA Leonardo: HDR IB Dragonfly+, 13,824 A100s (publicly described).
• Most academic and government HPC: IB-dominant for sustained-bandwidth HPC workloads.

RoCE:

• Many cloud GPU providers (AWS, Azure, GCP, CoreWeave variants) ship RoCE for "AI Ethernet" tiers.
• Meta's H100 fleets reportedly use RoCE on Spectrum-X-style designs (publicly described).
• xAI Colossus: 100k H100s on Quantum-2 IB (publicly stated), but other large fleets are RoCE.

There's no single "right answer" in deployment patterns — both fabrics ship at scale. The pattern is roughly: dedicated AI infrastructure tilts IB; cloud-style multi-purpose tilts RoCE.

Performance: how do they actually compare?

The published-spec view says they're equivalent at the same wire speed (400G NDR vs 400GbE). The real-world view has nuances.

Single-NIC bandwidth

Both fabrics hit ~95-97% of line rate on ib_write_bw-style point-to-point benchmarks once the basics are right. NDR400 sees ~388-395 Gb/s; 400GbE RoCE sees ~380-390 Gb/s after UDP/IP overhead. The ~5-10 Gb/s gap is RoCE encapsulation (UDP + IP + Eth headers add ~58 bytes of overhead per MTU; on 4096-byte payloads that's ~1.4%).

Practically the same. Don't pick a fabric on this number.

AllReduce at scale

This is where SHARP starts to matter. For a 1024-rank AllReduce:

• IB with SHARPv3 enabled: ~1.5-2x speedup vs no-SHARP.
• RoCE: no equivalent.

If your training profile is heavy on large AllReduce (most LLM training), IB-with-SHARP gives a real, measurable training-throughput improvement. If your profile is more compute-bound and AllReduce is a smaller fraction (e.g., MoE expert-parallel where AllToAll dominates), the SHARP advantage is smaller.

Tail latency under congestion

This is where RoCE tuning shows up. A well-tuned RoCE fabric (Spectrum-X with ConnectX-7+ and properly set DCQCN) holds tail latency tight. A poorly tuned one (default DCB profile, untuned PFC threshold, unprovisioned ECN marking) shows large tail spikes when the fabric goes from low to high load.

IB doesn't have this problem to the same degree because credit flow is operating at the link level and is automatic. The hardware isn't waiting for an operator to set a threshold.

In a green-field deployment with operators who haven't tuned RoCE before, IB will get to "good enough" performance faster. RoCE will eventually match it but takes more iterations.

AllToAll workload

AllToAll (used for MoE training) stresses the spine layer maximally — every rank to every other rank, no rail locality. Adaptive routing dominates here.

• IB: Quantum-2/X800 adaptive routing in hardware, mature.
• RoCE: Spectrum-X has telemetry-based adaptive routing; non-NVIDIA Eth switches may have ECMP only (which doesn't help when many flows hash to same path).

For MoE-heavy workloads, adaptive routing capability is a real differentiator. Both fabrics can do it, but the IB version has more years of operational maturity behind it.

The "RoCE is good enough now" argument

A common argument from cloud providers and some operators: "RoCE has caught up with IB by 2026, the practical performance gap is small, and the operational benefits of staying on Ethernet outweigh the few-percent of AllReduce."

There's truth to this. For the single-tenant, well-tuned, ~1000-node case, RoCE on Spectrum-X delivers training performance within a few percent of equivalent IB. NCCL works on both, GDR works on both, the data plane is identical. The arguments for IB are mostly about edge cases (peak SHARP performance, lighter operator burden, more mature multi-tenant isolation).

The counter-argument: those edge cases turn out to matter when you scale, when you multi-tenant, or when you change workloads from what you tuned for. A fabric that "works at the lab benchmark" but stutters under real-world bursty multi-job traffic costs more than the BOM premium of IB.

We've seen both choices succeed and both fail. The failure modes:

• IB failures: SM scaling problems past 5000 ports, partition complexity for multi-tenant, vendor lock-in to NVIDIA.
• RoCE failures: PFC misconfigured, ECN marking wrong, pause-storm under load, tail latency degrades over time as traffic patterns drift.

Migration notes

What if you picked one and want to move?

IB → RoCE

Painful. Requires:

• Replacing every switch (Quantum → Spectrum or alternative Eth).
• Recabling everything (OSFP-IB cables don't physically work in OSFP-Eth ports in all cases — verify per cable type).
• Reconfiguring HCAs (mlxconfig LINK_TYPE_P1=2).
• Standing up Ethernet management, BGP/OSPF, VLAN structure if you didn't have it.
• Rebuilding tenant isolation in RoCE terms (DSCP, ACLs).
• Tuning PFC, ECN, DCQCN before traffic ramps.

Plan 1-3 months of operator effort plus a multi-day downtime window per row. Realistically, you do this only at hardware-refresh time (every 4-5 years), not as a migration.

RoCE → IB

Same pain, mirror image. Replace switches, recable, reconfigure HCAs to IB mode, stand up SM, configure partitions.

The lesson: decide carefully at procurement, because the decision sticks.

Operator effort: a concrete comparison

What it actually takes to bring up and run each fabric, by the rough hour count.

IB bringup (1000-node fabric, experienced team)

Phase 0: design + BOM                    1-2 weeks (mostly procurement)
Phase 1: physical install                3-5 days (cabling)
Phase 2: switch firmware + mgmt setup    1 day
Phase 3: SM bringup                      4-8 hours
Phase 4: partitions + routing            2-4 hours
Phase 5: per-link verification           4-8 hours
Phase 6: pair-wise RDMA                  4-8 hours
Phase 7: NCCL scale-out                  1-2 days
Phase 8: handover + ops baseline         1 week
Total operator-active time:              ~3-4 weeks active

Most of the time is install (mechanical) and validation (waiting for tests). Active configuration work is small because IB's defaults are good.

RoCE bringup (same 1000-node fabric, experienced team)

Phase 0-2: same as IB                    2-3 weeks
Phase 3: routing (BGP/OSPF) bringup      1-2 days
Phase 4: VLAN / VXLAN / EVPN setup       1-2 days
Phase 5: DCB profile + PFC config        2-3 days (this is the painful one)
Phase 6: ECN marking + DCQCN tuning      2-3 days iterative
Phase 7: per-link verification           4-8 hours
Phase 8: pair-wise RDMA                  4-8 hours
Phase 9: NCCL + AllReduce + AllToAll     2-3 days
Phase 10: ongoing PFC retuning            (continuous, weeks)
Total operator-active time:              ~5-6 weeks active

The difference is mostly Phase 5-6 (PFC + ECN tuning) and Phase 10 (ongoing tuning). Real-world: a team familiar with RoCE adds maybe a week. A team new to RoCE can spend a month chasing PFC behavior under specific traffic patterns.

If your team has done RoCE before, this gap shrinks. If they haven't, IB's faster bringup is real.

What to actually decide first

The order of decisions:

1. Operator fit: does the team know IB or Ethernet better today? If you have to retrain a whole NOC, that's months of effort and bug-hunt cycles.
2. Tenant model: are you single-tenant (your own training jobs) or multi-tenant (you sell GPU slices)? PKey isolation in IB is mature; multi-tenant RoCE works but the isolation is built from L3 ACLs + DSCP queues which is more brittle.
3. Reference architecture: are you cloning a NVIDIA reference design (IB-default) or rolling your own (more flexible)?
4. Storage protocol: is your storage Ethernet (likely RoCE) or IB-native (likely IB)?
5. Cost ceiling: is the fabric cost a meaningful fraction of total cluster spend? Usually no, but if budget is tight, RoCE can save $1-3M per 1k-GPU cluster.

Walk through those five questions before drawing the BOM.

Common failure modes (specific to fabric choice)

Quick reference

See also

• InfiniBand fabric primer
• InfiniBand fabric architecture
• InfiniBand switch hardware
• InfiniBand implementation runbook
• RoCE v2 + DCB/PFC/ECN
• RDMA fundamentals
• NCCL multi-node tuning
• Subnet Manager + partitions

External authoritative references:

• NVIDIA Quantum-2 InfiniBand Platform overview
• NVIDIA Spectrum-X Ethernet Platform overview
• NVIDIA DGX SuperPOD H100 Reference Architecture
• InfiniBand Trade Association specifications
• IETF RFC 8915 — RoCE v2 considerations (informational reading)