InfiniBand switches: NVIDIA Quantum / Quantum-2 / Quantum-X800 family compared

When you're sizing a fabric you're choosing both the topology and the hardware to fill it. This page is the hardware reference: what NVIDIA ships in 2026, what each product is good for, what it costs you in power and rack U, and how to translate "1024 GPUs, 8-rail, full bisection" into a switch BOM.

For the architecture (rail-alignment, fat-tree math, oversubscription) see IB architecture. For implementation order see IB implementation. For verification commands see IB switches: L2 verification.

The current product family at a glance

Confirmed against NVIDIA product briefs and the official datasheets. Naming convention: MQM9700-NS2F is the SKU for QM9700 with internal management, OSFP, P2C airflow; suffix codes are airflow + management variants.

Quantum-2 (QM9700 / QM9790) — the workhorse

This is what 80% of GPU clusters built in 2024-2025 are running. Single Quantum-2 ASIC, 64 NDR-400G ports physically presented as 32 OSFP cages (each cage is a "twin-port OSFP" carrying two independent 400G links).

Specifications:

Port count:     64 NDR (or 128 NDR200 split, or mix)
Aggregate BW:   51.2 Tb/s (one direction) / 102.4 Tb/s bidirectional
Packet rate:    66.5 billion packets per second
Form factor:    1U, 19" rack
Power:          ~1720 W typical (per NVIDIA datasheet)
Cooling:        P2C (port-to-cable) or C2P (cable-to-port) airflow
PSUs:           Dual 1+1 redundant
ASIC:           Quantum-2 (single chip)
SHARP:          SHARPv3 (up to 64 parallel reduction streams)
SM:             Embedded supports up to 2000 nodes (QM9700 only)
                QM9790 has no embedded SM, externally managed
Routing:        Adaptive routing in hardware
Congestion:     Hardware-based congestion control
Telemetry:      Per-port counters streamed to UFM
Latency:        Sub-microsecond port-to-port (vendor publishes "ultra-low";
                xAI Colossus reports ~850 ns end-to-end across 3 tiers)

        +-------------------- QM9700 / QM9790 ----------------------+
        |                                                            |
   USB  | +--+ +--+ +--+ +--+   ...   +--+ +--+ +--+ +--+ +--+ +--+ |  console
   mgmt | |O1| |O2| |O3| |O4|         |  | |  | |  | |  | |  | |O32| |  serial
        | +--+ +--+ +--+ +--+         +--+ +--+ +--+ +--+ +--+ +--+ |  PSU bays
        |   front panel: 32 OSFP cages = 64 NDR-400G logical ports   |  fan tray
        +------------------------------------------------------------+
              ^                                                  ^
              |  port 1 = OSFP cage 1 (carries 1/1 + 1/2)         |  port 64 = OSFP cage 32 (carries 32/1 + 32/2)

When you'd pick QM9700 vs QM9790

The hardware is identical underneath. Pick managed if you're comfortable with embedded SM lifecycle (firmware tied to switch upgrades). Pick unmanaged if you want SM upgrade decoupled from switch upgrade.

SHARPv3 — the in-network compute that matters

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) lets the switch ASIC compute partial sums during AllReduce. Instead of every byte transiting end-to-end through every rank, the leaf and spine switches add streams together as they pass through.

SHARPv3 specifics:

• 64 parallel reduction streams per switch (SHARPv2 was 2)
• Multi-tenant — multiple independent jobs use SHARP simultaneously
• Built into the Quantum-2 ASIC; no extra hardware
• Activated through NCCL with NCCL_COLLNET_ENABLE=1 and SHARP daemons running on each switch + endpoint
• Provides up to 32x AI acceleration vs SHARPv2 (per NVIDIA marketing; in practice 1.5-2x AllReduce wall-clock improvement is what you actually measure)

To verify SHARP is active, look for SHARP trees in the SM logs and nccl-tests output:

$ NCCL_DEBUG=INFO NCCL_COLLNET_ENABLE=1 ./all_reduce_perf -b 8 -e 16G -f 2 -g 8
NCCL INFO Collnet/SHARP plugin loaded
NCCL INFO SHARP: tree depth 3, root at switch leaf-spine-core

If you don't see SHARP lines, either the daemon isn't running, the switch firmware doesn't have it enabled, or NCCL was built without SHARP support. See NCCL doc for the env vars.

Hardware-based congestion control on Quantum-2

Worth understanding because this is what makes IB feel "always lossless" without operator tuning. The mechanism:

• Switch egress queues track buffer occupancy in real-time.
• When occupancy crosses a threshold, the switch generates congestion notification packets (CNP-equivalent in IB-speak) back to the source HCA.
• Source HCA throttles the offending QP at the hardware level — not just signal to the OS, the NIC itself reduces its injection rate.
• Recovery is automatic; no operator tuning required for normal traffic patterns.

This is why a fresh-out-of-the-box IB fabric handles bursty traffic gracefully, while an equivalent fresh RoCE fabric falls over on the first heavy AllReduce until you tune PFC and ECN. The tuning is moved into the silicon.

You can still tune the thresholds for very specific workloads via congestion_control in opensm.conf, but the defaults work for almost everyone.

Quantum-2 telemetry counters worth alerting on

Each port reports >100 counters via the standard ibportstate / perfquery and via UFM Telemetry's structured streaming. The ones that matter operationally:

UFM's default alerting profile picks reasonable thresholds for these. Without UFM, write Prometheus exporters that scrape perfquery or use the gNMI streaming interface.

Quantum-X800 (Q3200-RA, Q3400-RA) — the B200/GB200 generation

Released 2024, paired with ConnectX-8 HCAs and B200/GB200 GPUs. Doubles per-port speed to 800G (XDR), uses 200G-PAM4 SerDes, four-lane port = 4 × 200G = 800G.

Q3200-RA — 2U air-cooled, dual-ASIC

Form factor:     2U
ASICs:           2 x Quantum-3 (each 36 ports of 800G = 18 OSFP cages)
Total ports:     144 effective 800G (logically two 36-port switches in one chassis)
Per-ASIC BW:     28.8 Tb/s
Total BW:        57.6 Tb/s
Power:           ~2 kW typical (significant jump from QM9700)
Cooling:         Air-cooled
NDR back-compat: yes (DAC, AOC, transceivers all work for 4x100G mode)
SHARP:           SHARPv4
PSUs:            Redundant
Use case:        Edge-of-cluster, smaller XDR builds, mixed NDR/XDR fabrics

Important: the Q3200-RA is two switches in one chassis. Each ASIC presents 36 of the 144 ports. From a topology perspective you wire it as two independent leafs (or two spines) sharing power and a single management address. They don't switch traffic between ASICs through the chassis backplane.

Q3400-RA — 4U director, single-fabric

Form factor:     4U
ASICs:           4 x Quantum-3 in internal fat-tree midplane
Total ports:     144 effective 800G (single switching domain)
Aggregate BW:    115 Tb/s
Power:           ~3-4 kW (8 x PSU for redundancy)
Cooling:         Air-cooled, C2P airflow
SHARP:           SHARPv4
2-tier fat-tree: connects up to 10,368 NICs per NVIDIA spec
                 (144-port leaf, 144-port spine, 1:1)
PSUs:            8 (redundant)
Use case:        Spine/core layer for 1k+ node fabrics

The Q3400 is the director-class option for the XDR generation. Internally it's structured as four ASICs arranged in a Clos — line-card-equivalent on the front, fabric-card-equivalent inside — but from outside it's one 144-port switch. That single-domain property is what makes it useful as a spine: the rest of the fabric sees 144 ports of equal-cost shortest path to anywhere, not "144 ports across 4 switching ASICs."

SHARPv4 — what's new

• More aggressive reduction trees (deeper, asymmetric topologies handled)
• Better adaptive routing integration
• Designed alongside ConnectX-8 HCAs (which natively offload more)
• Higher per-tenant scaling for multi-tenant SHARP

For most operators, SHARPv4 means "all the SHARPv3 benefits, more so" — you don't tune it differently, you just enable it and NCCL uses it.

Pairing with ConnectX-8 HCAs

The Quantum-X800 generation was designed alongside ConnectX-8 NICs. Worth noting the pairing because it changes some sizing assumptions:

ConnectX-8 capabilities:
  Ports:        Single-port 800G, or dual-port 400G
  PCIe:         Gen6 x16 (Gen5 x16 fallback)
  Max msg rate: ~1.2 Bpps (vs 800 Mpps on CX-7)
  Form factor:  PCIe HHHL or full-height; OCP3.0 SFF
  Connectors:   OSFP for the NIC side at 800G; some SKUs QSFP112
  In-network:   New offload engines for collectives and RAG-style retrieval
  GDR:          Native, with peer-mem in PCIe Gen6 territory

For an 8-rail B200 / GB200 node with CX-8 HCAs, per-node fabric BW is 8 × 800 Gb/s = 6.4 Tb/s — twice what CX-7 provides. Sizing the spine layer for that requires the doubled per-port speed, hence Q3400-RA director-class density becomes more attractive.

Older generation: Quantum (HDR / QM8700)

Still in many production fleets — A100 clusters, older H100 rollouts, and any fabric where the HCAs are CX-6 / CX-6Dx.

QM8700:
  Ports:        40 HDR (200G) on QSFP56 cages
  Aggregate:    16 Tb/s
  Form factor:  1U
  Power:        ~640 W
  SHARP:        SHARPv2 (2 reduction streams, no multi-tenant)
  Embedded SM:  yes

Two important interop facts:

• HDR is forward-compatible-ish with NDR: a CX-7 (NDR) HCA plugged into a QM8700 (HDR) port runs at HDR (200G), not NDR. The HCA negotiates down. Verify with ibstat | grep Rate.
• HDR cables and connectors are not directly compatible with NDR: QSFP56 (HDR) is physically different from OSFP (NDR). Mixing requires the right transceivers and splitters, sometimes adapter cables.

For a new build in 2026 you wouldn't choose QM8700. Stays in fleet because it's already racked and paid for.

Comparing across generations: where the speed comes from

Generation:  HDR              NDR              XDR
Year:        2018             2022             2024
Per-lane:    50 Gb/s PAM4     100 Gb/s PAM4    200 Gb/s PAM4
Lanes/port:  4                4                4
Per-port:    200 Gb/s         400 Gb/s         800 Gb/s
Connector:   QSFP56           OSFP / QSFP112   OSFP
ASIC:        Quantum          Quantum-2        Quantum-3 (X800)
Power/port:  ~16 W            ~27 W            ~14 W (despite higher BW;
                                                  more efficient SerDes)
SHARP:       SHARPv2          SHARPv3          SHARPv4

The "X" in XDR stands for "Extreme Data Rate." The doubling each generation is achieved by doubling per-lane modulation density (4×→8×→16× effective signaling vs SDR baseline) plus PAM4 since HDR.

Embedded vs dedicated subnet manager

Decision matrix:

A typical production setup for a 1000-node cluster:

• 2 × dedicated SM hosts (small x86, 8 cores, 32 GB RAM, dual-port HCAs into the fabric for redundancy)
• Master at priority 0, standby at priority 1
• UFM running alongside the master for telemetry and alerting
• Embedded SM on every leaf switch disabled (ib sm disable) to prevent split-brain

You can dual-purpose: one SM host runs UFM Enterprise (or UFM Cyber-AI for advanced anomaly detection), the second is a vanilla opensm standby. UFM telemetry collects 120+ counters per port (BER, temperature, retransmissions, histograms) and feeds them into Prometheus / Grafana via UFM's REST API or push integrations.

UFM tiers

NVIDIA UFM ships in three tiers in 2026:

• UFM Telemetry: collection only, push to your own observability stack (Prometheus, FluentD).
• UFM Enterprise: full management — discovery, provisioning, traffic monitoring, congestion discovery, automated partition lifecycle, dashboards.
• UFM Cyber-AI: Enterprise + ML-driven anomaly detection on the telemetry stream. Useful at scale where you can't manually sift counters.

For a 256+ GPU cluster customer-facing, Enterprise is roughly the minimum. Cyber-AI is overkill until you're past ~1000 nodes.

Spectrum-X (SN5600) — the RoCE alternative

If you've decided on RoCE instead of IB (see IB vs RoCE), the equivalent NVIDIA hardware is:

SN5600:
  ASIC:         Spectrum-4
  Ports:        64 x 800GbE (OSFP)
  Plus:         1 x 25GbE SFP28 (mgmt-ish)
  Aggregate:    51.2 Tb/s
  Form factor:  2U
  Buffer:       Fully shared, large packet buffer
  Use case:     Spectrum-X AI fabric (RoCE with PFC + ECN tuned for AI)

Pairs with BlueField-3 SuperNICs to deliver an "AI Ethernet fabric" with tuned PFC, telemetry-based congestion control, and adaptive routing for RoCE. NVIDIA markets a 1.6× improvement vs traditional Ethernet fabrics for AI workloads.

The architecture is similar to IB (leaf + spine fat-tree, rail alignment principles still apply). The differences are operational — see IB vs RoCE for the fabric-choice decision.

Implementing rail-aligned fabric — the practical mapping

Rail alignment is the architectural pattern from IB architecture. Here's how you actually realize it with these switches.

Naming and labeling

Adopt a consistent scheme on day one. Recommended:

Leaf switches:
  leaf-r0-l01   (rail 0, leaf 1)
  leaf-r0-l02   (rail 0, leaf 2)
  ...
  leaf-r7-l04   (rail 7, leaf 4)

Spine switches:
  spine-r0-s01
  spine-r0-s02
  ...

OOB management hostname matches:
  leaf-r0-l01.fabric.internal

The r<rail> token in the name is what saves you 6 months later when a port is misbehaving. Without it, "leaf-23 port 17" is opaque. With it, leaf-r4-l03 port 17 immediately tells you what rail it serves.

Per-rail fabric design

For 8-rail nodes (most DGX H100 / H200 / B200), each rail is its own fat-tree:

   Rail 0 fabric:
   +----------+ +----------+ +----------+ +----------+
   |spine-r0-1| |spine-r0-2| |spine-r0-3| |spine-r0-4|     1 spine for SU-32, 4 for SU-128
   +----------+ +----------+ +----------+ +----------+
        | full mesh |
        v           v
   +----------+ +----------+
   | leaf-r0-1| | leaf-r0-2|     2 leaves for SU-32 (32 nodes / 16 nodes per leaf)
   +----------+ +----------+     ... or 4 leaves for SU-128
        |    |    |    |
       node mlx5_0 of every node in this rail

   Repeat 8 times (rails 0..7), independently.

Each rail being independent means: a failure on rail 3's spine doesn't affect rails 0, 1, 2, 4, 5, 6, 7. NCCL's per-rail rings continue working on the other 7 rails; only rail 3's traffic is degraded.

This is a real operational property — a switch failure in a non-rail-aligned design takes down a fraction of every job, while in a rail-aligned design it concentrates damage to one rail and most jobs continue.

Shared-spine variant

For smaller fabrics (<256 endpoints) you can collapse the per-rail spines into a single shared spine layer, with port grouping to keep rails distinct:

   Shared spine for 32-node fabric:
   +-----------------------+
   |  spine-shared-01      |     64 ports total
   |    ports 1-8: rail 0  |     8 ports per rail
   |    ports 9-16: rail 1 |
   |    ...                |
   |    ports 57-64: rail 7|
   +-----------------------+

The discipline: each leaf's uplinks all land on its rail's reserved port range on the spine. A misplaced uplink crosses rails. ftree routing handles this fine; just be careful with cabling.

For larger fabrics (>256 endpoints) per-rail spines win on isolation and scale.

Cabling per switch type

Maximum reaches:

Practical takeaway: DAC inside a rack, AOC between racks in adjacent rows, transceivers + fiber for long runs. Plan transceiver count based on your floor plan.

Power and rack planning

A real rack of an AI fabric:

   Compute rack:
   +------------------+
   |  leaf-N (QM9700) |  1U  ~1.7 kW
   |  ---------------- |
   |  cable manager   |  1U
   |  ---------------- |
   |  GPU node 1      |  4U  ~10.4 kW (DGX H100 typical)
   |  GPU node 2      |  4U  ~10.4 kW
   |  GPU node 3      |  4U  ~10.4 kW
   |  ...              |
   |  GPU node 8      |  4U  ~10.4 kW
   +------------------+
                       Total:
                       8 nodes x 10.4 kW = 83.2 kW
                     + 1 leaf x 1.7 kW    =  1.7 kW
                     = ~85 kW per rack (PDU sizing)

   Switch row (separate):
   +------------------+
   |  spine-1 (QM9700)|  1U  ~1.7 kW
   |  spine-2 (QM9700)|  1U  ~1.7 kW
   |  ...              |
   |  spine-8 (QM9700)|  1U  ~1.7 kW
   +------------------+
   = ~14 kW for 8 spines (much lower than compute racks)

Switch row power is comparatively trivial. The compute racks are where the watts live, but cooling is also where switch placement matters: hot-aisle / cold-aisle has to flow consistently for both compute and switches. Mixed P2C/C2P airflow in the same row is a common rookie mistake — pick one airflow direction per row.

For Quantum-X800 Q3400 directors, you're looking at ~3-4 kW per chassis, and they're 4U, so a switch rack with 8 directors plus management gear sits around 30-40 kW. Plan PDU and cooling accordingly.

BOM math: real cluster examples

256-GPU H100 cluster (32 nodes, 8 HCAs each)

Endpoints: 32 nodes x 8 HCAs = 256 ports of NDR 400G

Rail-aligned, full bisection, single Scalable Unit (matches DGX SuperPOD H100):
  Per rail: 32 endpoints
  Leafs per rail: 32 / 32 = 1 leaf (32 down + 32 up)
  Spines per rail: 1 spine
  Total leafs: 8 (one per rail)
  Total spines: 8 (one per rail)
  Cross-rail: at this size the rails share spines through 1-tier fat-tree

Switch BOM:
  - 8 x QM9700 (leaf, with embedded SM on one for bringup, then disable)
  - 8 x QM9790 (spine, externally managed)
  - 1-2 x x86 SM host running opensm or UFM

Cable BOM:
  - 256 x OSFP-to-QSFP112 (3m DAC if same-rack, AOC if not) for HCAs
  - 256 x OSFP-to-OSFP (5-30m AOC) for leaf-to-spine

Power: 16 switches x 1.7 kW = 27.2 kW switch power
Rack U: 16 * 1U + cable management

This sizing matches the DGX SuperPOD H100 reference architecture for one Scalable Unit (SU = 32 DGX H100 systems). NVIDIA's published reference uses 32 leaves total across multiple SUs but the per-SU math is the same.

1024-GPU H100 cluster (128 nodes, 8 HCAs each)

Endpoints: 128 nodes x 8 HCAs = 1024 ports of NDR 400G

Rail-aligned, full bisection, 4 Scalable Units (4 SU x 32 nodes):
  Per rail: 128 endpoints
  Leafs per rail: 128 / 32 = 4
  Spines per rail: 4
  Total leafs: 32
  Total spines: 32
  In a 2-tier fat-tree, leafs uplink directly to spines (no core layer needed)

Switch BOM:
  - 32 x QM9790 (leaf)
  - 32 x QM9790 (spine)
  - 2 x dedicated SM host (master + standby) running UFM Enterprise

Cable BOM:
  - 1024 x OSFP-to-QSFP112 for HCAs (mostly 3m DAC + some AOC)
  - 32 x 32 = 1024 OSFP-to-OSFP for leaf-to-spine (5-30m AOC)

Power: 64 switches x 1.7 kW = ~109 kW switch power
Total fabric power: ~115 kW including SM hosts and management

8192-GPU H200 cluster (1024 nodes, 8 HCAs each, XDR)

Endpoints: 1024 x 8 = 8192 ports of XDR 800G

Rail-aligned, 3-tier (leaf / spine / core), full bisection or near-full:
  Per rail: 1024 endpoints
  Leafs per rail: 1024 / 36 = 29 (round up to 32 for symmetry)
  Spines per rail: 32 (matching leafs in 2-tier per rail)
  Core: shared across rails for cross-rail traffic
  Total leafs: 32 x 8 = 256 x Q3200 (or fewer Q3400s)
  Total spines: 32 x 8 = 256 x Q3200 (or 16 x Q3400 directors)

Director option (Q3400-RA spine):
  - Replace 256 x Q3200 spines with ~16 x Q3400 directors
  - Cuts switch count significantly, simplifies cabling but introduces single-point density risk
  - Each Q3400 connects all 8 rails through its 144 ports

Power: ~500-800 kW for the fabric alone, depending on director vs fixed

This is in the territory of NVIDIA's reference architectures for large SuperPOD clusters and the Eos-class supercomputer designs. If you're building one, you're working with NVIDIA's reference team and don't need this doc; if you're sizing one in a spreadsheet, the math here is the right starting point.

Reference designs (publicly documented)

A few designs you can study without an NDA:

• DGX SuperPOD H100 Reference Architecture (NVIDIA, 2023): 4 SU × 32 DGX = 1024 GPUs, NDR fat-tree using QM9700/QM9790, rail-aligned. The canonical reference; everything else borrows from this.
• DGX SuperPOD GB200 Reference Architecture (NVIDIA, 2024): GB200 NVL72 with Quantum-X800 XDR fabric, scaling pattern updated for NVL72 racks.
• DGX SuperPOD B300 Reference Architecture (NVIDIA, 2025): Latest, B300 + Q3400 directors + AC power, smaller scale tier.
• Selene (NVIDIA, 2020): 4480 A100 GPUs, HDR Quantum fat-tree, in-house testbed, publicly described in NVIDIA's blog posts. Good for understanding HDR-era design.
• Eos (NVIDIA, 2023): 4608 H100 GPUs, NDR Quantum-2 fabric, evolution of Selene.
• CINECA Leonardo (2022): 13,824 A100 GPUs, HDR Quantum fabric in Dragonfly+ topology — a reference for non-fat-tree designs at scale.

The Reference Architecture documents at docs.nvidia.com/dgx-superpod/ are the most useful — they include cable counts, port maps, and rack diagrams that you can adapt. They're free and authoritative.

Common failure modes

Quick reference: command and SKU cheat sheet

See also

• InfiniBand fabric architecture (rail-alignment, fat-tree, bisection)
• InfiniBand implementation runbook
• InfiniBand vs RoCE decision guide
• InfiniBand switches: L2 verification
• Subnet Manager + partitions
• InfiniBand fabric primer
• NCCL multi-node tuning

External authoritative references:

• NVIDIA Quantum-2 QM9700 Datasheet
• NVIDIA Quantum-X800 Platform overview
• NVIDIA Q32xx and Q34xx XDR Switch User Manual
• NVIDIA Unified Fabric Manager
• DGX SuperPOD reference architectures