InfiniBand fabric architecture: building a working AI cluster fabric

You can buy 1024 NDR-capable HCAs, ten Quantum-2 switches, a few thousand transceivers, and still end up with a fabric that runs NCCL allreduce at 60% of what the spec sheet says. The reason is almost never a single broken component. It's the architecture: oversubscription on the wrong layer, rails crossed at cabling time, partitions that don't match tenant boundaries, an SM that can't keep up. The hardware is the easy part.

This page is the design view. It assumes you've read the InfiniBand primer and SM + partitions, and walks through what an AI fabric actually looks like as a layered system, why rail alignment matters more than bandwidth, and what numbers to target when sizing.

Hardware specifics (which switch SKU, which transceiver, what power per rack) live in IB switch hardware. Implementation (the runbook from rack delivery to NCCL pass) lives in IB implementation.

The six architectural layers

Top to bottom, an AI-grade IB fabric is six layers. Most people think about three (compute / leaf / spine) and miss the others.

+------------------------------------------------------------+
| 6. UFM / monitoring (optional but recommended at scale)    |
+------------------------------------------------------------+
| 5. Subnet Manager (embedded in switch OR dedicated host)   |
+------------------------------------------------------------+
| 4. Super-spine (only for very large clusters > ~4k nodes)  |
+------------------------------------------------------------+
| 3. Spine switches (aggregation between leafs)              |
+------------------------------------------------------------+
| 2. Leaf switches (one per rail-group or rack-group)        |
+------------------------------------------------------------+
| 1. Compute nodes with HCAs (1-rail to 8-rail per node)     |
+------------------------------------------------------------+

What each layer is responsible for:

The SM is shown above the data-plane layers because it talks to all of them but isn't on the data path. That's the right mental model: data plane is layers 1-4, control plane is layer 5, observability/management is layer 6.

Single-rail topology — small reference fabric

The simplest case: one HCA per node. You see this in storage clusters, CPU-only HPC clusters, or test/staging GPU clusters where you've cheaped out on NICs. The diagram is straightforward.

                  +------------+    +------------+
                  |  spine-1   |    |  spine-2   |
                  +------------+    +------------+
                   /    |   \  \    /  /   |    \
                  /     |    \  \  /  /    |     \
                 /      |     \  \/  /     |      \
                /       |      \ /\ /      |       \
               /        |       X  X       |        \
              /         |      / \/ \      |         \
             /          |     /  /\  \     |          \
            /           |    /  /  \  \    |           \
       +-------+    +-------+    +-------+    +-------+
       |leaf-1 |    |leaf-2 |    |leaf-3 |    |leaf-4 |
       +-------+    +-------+    +-------+    +-------+
        | | | | ...  | | | | ...  | | | | ...  | | | |
       n01-n08      n09-n16      n17-n24      n25-n32

 Each leaf:  64 ports total, 32 down to nodes + 32 up to spines
 Each spine: 64 ports, all 64 down to leafs (16 per leaf x 4 leafs)
 Bisection:  2 spines x 64 ports x 400G = 51.2 Tb/s -> full bisection
 Nodes:      32 nodes x 1 HCA each = 32 endpoints

A few things to notice:

• Every leaf connects to every spine. That's a Clos / two-tier fat-tree. There is no leaf-to-leaf link. All leaf-to-leaf traffic transits a spine.
• Uplink count = downlink count per leaf. 32 down + 32 up = full bisection. If you cut the leaf-to-spine cables you'd lose connectivity to half the world; the fabric is sized so any leaf can drive its full 32×400 = 12.8 Tb/s downstream demand into the spine layer simultaneously.
• You can get away with 4 leafs and 2 spines for 32 nodes because 32 ports of downlink fits on a single 64-port leaf with room left over for uplinks. In practice you'd usually want more headroom, so 4 leafs is the comfortable answer.

This single-rail design is fine when you only have one NIC per node. The minute you have 8 NICs per node — i.e., a real AI training node — single-rail stops scaling and you move to rail-aligned.

Multi-rail / rail-aligned topology — the AI standard

Modern GPU servers (DGX H100, DGX H200, HGX, OEM equivalents) ship with 8 HCAs per node, one per GPU. The reason is bandwidth: 8 × 400 Gb/s NDR = 3.2 Tb/s of network bandwidth per node, sized to roughly match the per-GPU NVLink and HBM bandwidth so the network doesn't bottleneck NCCL out-of-server collectives.

But "8 HCAs per node" only works as designed if the fabric is rail-aligned: HCA index N on every node connects to "the rail-N plane" of the fabric, which is a separate fat-tree. Each rail is its own fabric, and they share spines (or super-spines) only at the top.

   Node N (8 GPUs, 8 HCAs)        Node M (8 GPUs, 8 HCAs)        ... 32 nodes total
   +-------------------+          +-------------------+
   | GPU0 - HCA0 (m0)  |          | GPU0 - HCA0 (m0)  |
   | GPU1 - HCA1 (m1)  |          | GPU1 - HCA1 (m1)  |
   | GPU2 - HCA2 (m2)  |          | GPU2 - HCA2 (m2)  |
   | GPU3 - HCA3 (m3)  |          | GPU3 - HCA3 (m3)  |
   | GPU4 - HCA4 (m4)  |          | GPU4 - HCA4 (m4)  |
   | GPU5 - HCA5 (m5)  |          | GPU5 - HCA5 (m5)  |
   | GPU6 - HCA6 (m6)  |          | GPU6 - HCA6 (m6)  |
   | GPU7 - HCA7 (m7)  |          | GPU7 - HCA7 (m7)  |
   +-------------------+          +-------------------+
        |    |    |    |               |    |    |    |
        m0   m1   m2  ...m7             m0   m1   m2  ...m7
        |    |    |                     |    |    |
        |    |    |    +----------------+    |    |
        |    |    |    |                     |    |
        v    v    v    v                     v    v
  +---------+ +---------+ +---------+ +---------+ +---------+ +---------+ +---------+ +---------+
  | leaf-0  | | leaf-1  | | leaf-2  | | leaf-3  | | leaf-4  | | leaf-5  | | leaf-6  | | leaf-7  |
  | (rail 0)| | (rail 1)| | (rail 2)| | (rail 3)| | (rail 4)| | (rail 5)| | (rail 6)| | (rail 7)|
  +---------+ +---------+ +---------+ +---------+ +---------+ +---------+ +---------+ +---------+
       \         |         /     |         |         |         |         /
        \        |        /      |         |         |         |        /
         \       |       /       |         |         |         |       /
          v      v      v        v         v         v         v      v
                       +-------------+ +-------------+
                       |  spine-0    | |  spine-1    |   (one spine pair per rail,
                       |  (rail 0)   | |  (rail 0)   |    or shared spines that
                       +-------------+ +-------------+    keep rail traffic distinct)
                          ... rail 1, 2, 3, ... 7 spine planes ...

  Each leaf-N: 32 downlinks (one to HCA-N of each of the 32 nodes)
               + 32 uplinks to rail-N spines
  Total leafs: 8 (one per rail)
  Total spines: 16 (2 spines x 8 rails) for full bisection per rail

Read that diagram carefully. The critical detail: leaf-0 sees every node's HCA m0 and only HCA m0. Leaf-1 sees every node's m1. And so on.

That means traffic from node-A.HCA0 to node-B.HCA0 stays inside leaf-0 (one switch hop). Traffic from node-A.HCA0 to node-B.HCA1 has to go up to a spine, across, and back down — three hops, plus it leaves the rail-0 fabric.

Why rail alignment matters

NCCL — and any well-written collective library — knows about this layout. By default it pairs GPU N on one node with NIC N on the same node, and the corresponding NIC N on the peer node. For an AllReduce across 32 nodes, the collective decomposes into 8 parallel rings, one per rail. Each ring's traffic stays inside its own leaf:

   AllReduce across 32 nodes, 8 GPUs each:

   Rail 0:  GPU0 of all 32 nodes form a ring. Traffic stays in leaf-0.
   Rail 1:  GPU1 of all 32 nodes form a ring. Traffic stays in leaf-1.
   ...
   Rail 7:  GPU7 of all 32 nodes form a ring. Traffic stays in leaf-7.

   Result:  bytes never traverse the spine. 8 leafs each carry 1/8 of the traffic.
            Aggregate fabric utilization: 8 x 400 Gb/s per node = 3.2 Tb/s.

If rails are misaligned (cabling error: HCA0 of node-1 plugged into leaf-3 instead of leaf-0), NCCL's rail assumption breaks. Now node-1.HCA0 to node-2.HCA0 has to hop leaf-3 → spine → leaf-0, for every byte. The spine layer suddenly carries traffic it was never sized to carry. You get:

• Effective bandwidth drops by ~30-50% on AllReduce
• Spine port congestion produces PFC backpressure (RoCE) or credit stalls (IB)
• Jitter increases; tail latency on small collectives spikes

This is the most common implementation bug we see. It looks fine in ibstat (the link is up at 400 Gb/s), it looks fine in ibping (you can reach the peer), it just runs slow under load. See common pitfalls in the implementation doc.

Cross-rail traffic isn't free, but it's necessary

Some collectives genuinely need cross-rail traffic — AllToAll for MoE expert routing being the canonical example. Each GPU sends a different chunk to every other GPU, regardless of rail. This is when the spine layer earns its keep.

The architectural rule: size the spine layer for full bisection across the full HCA count, not just the per-rail bandwidth. If you have 32 nodes × 8 HCAs = 256 endpoints at 400 Gb/s, full bisection wants 102.4 Tb/s of spine capacity. That's 256 ports at the spine layer (256 × 400 Gb/s = 102.4 Tb/s), which is 4 × Quantum-2 64-port switches. With the rail-aligned design, you split that into 8 rail planes of 32 spine ports each, but the math is the same.

Fat-tree (k-ary n-tree)

The topology underneath rail-aligned design is a fat-tree, sometimes called a Clos. The "fat" comes from the property that the upper layers have at least as much aggregate bandwidth as the lower layers — a normal tree has a thin trunk; a fat tree has a trunk fat enough to carry every leaf simultaneously.

   2-tier (k=64) fat-tree, k/2 = 32 down, 32 up per leaf:

                      +-----+ +-----+ ... +-----+
                      | s0  | | s1  |     | s31 |     32 spines (each 64-port)
                      +-----+ +-----+     +-----+
                       |||...   |||...      |||...
                       | full mesh from each leaf to every spine
                       v                   v
        +------+ +------+ ... +------+ ... +------+
        | l0   | | l1   |     | lm   |     | l63  |  64 leafs
        +------+ +------+     +------+     +------+
         ||...                 ||...
         32 ports down to nodes, 32 up to spines

     Endpoints: 64 leafs x 32 ports = 2048 endpoints @ 400 Gb/s
     Total bandwidth: 2048 x 400 Gb/s = 819.2 Tb/s ingress
     Spine bandwidth: 32 spines x 64 ports x 400 Gb/s = 819.2 Tb/s
     => Full bisection

The same pattern recursed gives you 3-tier fat-trees for very large clusters. The DGX SuperPOD H100 reference architecture uses this for 4 SU's (4 × 32 = 128 nodes × 8 rails = 1024 endpoints) with a leaf / spine / core layout.

Oversubscription

Full bisection isn't always built. "2:1 oversubscribed" means leafs have twice as many downlinks as uplinks — 64-port leaf with 42 down to nodes and 21 up to spines. Cheaper, fewer cables, fewer spine switches.

   Full bisection vs 2:1 oversubscription on a 64-port leaf:

   +-------------+-------------+-------------+
   | Layout      | Down to     | Up to       |
   |             | nodes       | spines      |
   +-------------+-------------+-------------+
   | Full        | 32          | 32          |
   | 2:1 over    | 42          | 21 (rounded)|
   | 4:1 over    | 51          | 13 (rounded)|
   +-------------+-------------+-------------+

For AI training, 2:1 oversubscription is already a meaningful hit on AllReduce, and 4:1 is a disaster. Reasoning: AllReduce across N nodes with M-byte buffers wants the bisection bandwidth to carry roughly 2M(N-1)/N bytes for ring-AllReduce. At 2:1 oversubscription that takes twice as long, and nothing else fills in for the missing capacity. Some traffic patterns (AllToAll) hurt even worse.

The exception: if your workload is rail-confined (e.g., tensor parallel only across same-server GPUs, or you've designed your job to keep AllReduce inside one rail) then oversubscription on the spine layer doesn't matter. But for a general-purpose AI cluster, build full bisection. The cost difference is real but small relative to the GPU spend, and you avoid an unfixable architectural ceiling.

Real numbers: a 64-node H100 cluster

Concrete sizing:

   64 nodes x 8 GPUs = 512 GPUs (H100 SXM)
   64 nodes x 8 HCAs (NDR 400G) = 512 endpoints

   Rail-aligned: 8 rail planes
   Per rail: 64 endpoints @ 400 Gb/s

   Per rail leaf count: 64 endpoints / 32 ports per leaf = 2 leafs per rail
   Total leafs: 8 rails x 2 leafs = 16 leafs
   Per rail spine count: 2 leafs x 32 uplinks / 64 ports per spine = 1 spine per rail
   Total spines: 8 (one per rail)

   Bisection per rail: 1 spine x 64 ports x 400 Gb/s = 25.6 Tb/s per rail
   Aggregate bisection: 8 rails x 25.6 Tb/s = 204.8 Tb/s

   Total switch count: 16 leafs + 8 spines = 24 Quantum-2 switches

That's the order-of-magnitude switch BOM for a 512-GPU H100 fabric. Every additional leaf or spine you don't need is wasted money; every one you skipped becomes a bottleneck under load.

Dragonfly+ — the alternative for very large clusters

Fat-trees scale, but cabling cost grows linearly with port count and you eventually hit a wall on how many spines + super-spines you can rack. Dragonfly+ is the answer some HPC sites picked.

   Dragonfly+ structure: groups (intra-group fat-tree) connected all-to-all between groups.

           +-----------------+        +-----------------+
           | group A         |   |   | group B         |
           |  (intra fat     |  ===  |  (intra fat     |
           |   tree, ~256    |  ===  |   tree, ~256    |
           |   endpoints)    |  ===  |   endpoints)    |
           +-----------------+   |   +-----------------+
                  ||   ||             ||   ||
                  ||   ||  global   ||   ||
                  ||   ||  links    ||   ||
                  ||   ||  (long    ||   ||
                  ||   ||   AOC)    ||   ||
                  ||   ||             ||   ||
           +-----------------+        +-----------------+
           | group C         |  ===   | group D         |
           |  (intra fat     |  ===   |  (intra fat     |
           |   tree, ~256)   |  ===   |   tree, ~256)   |
           +-----------------+        +-----------------+

   Each group: fat-tree internally. Each pair of groups: small set of "global" links.
   Total endpoints: groups x intra-group endpoints, scales to 10s of thousands.

Dragonfly+ properties (vs equivalent fat-tree):

• Cheaper cabling at scale: the long, expensive global links are sparse — just enough for all-to-all between groups, not full bisection.
• Smaller diameter: 3 hops max (intra → global → intra) for any pair across groups.
• Adaptive routing required: with sparse global links, "best path" depends on instantaneous load. Quantum-2 + UFM with AR enabled handles this; static routing degrades fast.
• Add a group, no recabling: a key operational advantage — you can stretch a Dragonfly+ to bigger sizes without ripping up existing cables.

Used at: U Toronto Niagara (the original public Dragonfly+ deployment), CINECA Leonardo (HDR Dragonfly+, exascale-class), ECMWF (full-bandwidth Dragonfly+).

For an AI cluster you'd consider Dragonfly+ once you're past ~4000-5000 endpoints, when the fat-tree's spine and core layer dominates BoM. Below that, fat-tree is simpler operationally and makes better sense.

Switch-internal architecture

Worth knowing because two "64-port" switches can have very different internals.

Single-ASIC fixed switches

Quantum-2 QM9700: one Quantum-2 ASIC, 64 NDR ports as 32 OSFP cages, 51.2 Tb/s aggregate. Internally, every port can talk to every other port at line rate; the ASIC has enough crossbar capacity for full meshing.

   QM9700 internal:

   +----+ +----+ +----+ ... +----+      32 OSFP cages
   | 1/1| | 1/2| | 1/3|     |1/32|      (each splits into 2 NDR ports = 64 logical)
   +----+ +----+ +----+     +----+
      |     |     |            |
      v     v     v            v
   +-------------------------------+
   |         Quantum-2 ASIC        |    Single chip, 51.2 Tb/s crossbar
   |  - SHARPv3 in-network compute |
   |  - Adaptive routing engine    |
   |  - Congestion control HW      |
   |  - Telemetry counters         |
   +-------------------------------+
   |            x86 CPU            |    Management plane,
   |            (mgmt)             |    optional embedded SM
   +-------------------------------+

Multi-ASIC chassis-internal fabric

Quantum-X800 Q3200-RA in 2U: two Quantum-3 ASICs in the same enclosure, each with 18 OSFP cages. Each ASIC is logically an independent switch with 36 ports of 800G. That's why marketing says "144 ports of 800G in 2U" — but you're looking at two switches inside one chassis. From a topology perspective, you wire it as two leafs (or two spines) sharing only power and management.

Q3400-RA director-class: 4U, four Quantum-3 ASICs internally arranged in their own fat-tree midplane. Externally presents 144 ports at 800G that all switch to each other. This is what you use when you need spine-class density and don't want to manage many 1U switches.

Director / modular chassis (older generation)

Older Mellanox director switches (CS8500-class HDR, 800-port chassis) used a midplane Clos internally — line cards plus fabric cards arranged as an internal fat-tree. From outside it looked like a single huge switch; inside, the fabric cards were spines and the line cards were leafs. Those are still in production fleets but new builds are mostly fixed 1U/2U/4U from the Quantum-2 / Quantum-X800 families.

Subnet design

One subnet per fabric, one SM, even at thousands of nodes. That's the default. The reasoning:

• LID space: classic 16-bit LIDs gave 49152 usable per subnet; modern extended-LID (24-bit) extends this to ~16 million. Fabric size is no longer the LID-space limit it once was.
• Single-SM scaling: opensm comfortably handles 2000-3000 ports. UFM scales further. Beyond that you size the SM host (cores + RAM) and tune sweep intervals; you don't multi-subnet.
• Multi-subnet introduces routers. IB-to-IB routers exist but are rare and add hops.

You'd consider multi-subnet only when:

• Crossing data centers (each DC is its own subnet, bridged by IB router or by application-layer gateway).
• Hard-isolating tenants for compliance reasons that PKey can't satisfy.
• A single flat fabric exceeds what your SM can sweep within your latency budget.

For the 99% case: one subnet, one master SM, one standby SM. PKeys for tenant boundaries. See SM + partitions for the partition syntax.

LID budget planning

Even with extended LIDs, plan your address space:

   Default opensm config:
   - LID range 0x0001 - 0xBFFF (49152 usable)
   - SM at LID 1
   - Endpoints get LIDs sequentially as discovered

   If you set LMC > 0 (LID Mask Count), each port consumes 2^LMC LIDs.
   LMC=2 (4 LIDs per port) on a 1024-port fabric = 4096 LIDs consumed.
   Still fits in 16-bit, but eats budget.

   Modern extended LIDs (enable_extended_lids true in opensm.conf):
   - 24-bit LID space, ~16M usable
   - Required by switches that support it (most Quantum-2+)
   - Enable from day one if you might grow past 32k ports

Don't enable extended LIDs after the fact on a running fabric without a planned reconfiguration window — the SM has to re-LID the whole subnet, briefly disrupting connectivity. Set it correctly at bringup.

Partition / tenant design

   Default partition (0x7fff): SM + all-to-all control traffic
   Tenant A (0x8001): full mesh inside, no leak to tenant B
   Tenant B (0x8002): full mesh inside, no leak to tenant A
   Storage partition (0x8100): full members = both tenants + storage gateway

   ALL=limited in default => endpoints can talk to SM but not to each other through default

This pattern lets storage be addressable from both tenants while keeping tenant compute traffic isolated. See the partition config example in SM + partitions.

Cabling realities

This section saves people weeks of work if they read it before procurement.

Optical vs copper

For a typical AI cluster floor plan (compute rows + switch row), the breakdown is roughly:

• Node-to-leaf (in-rack): DAC, 1-2m
• Leaf-to-spine (row-to-row): AOC, 5-30m
• Spine-to-super-spine (if present): AOC or transceivers, 30-100m

Connector form factors

NDR (400 Gb/s, 4x100G PAM4):
  Switch side: OSFP cages (each cage = 2 logical 400G ports = "twin-port OSFP")
  HCA side:    OSFP or QSFP112
  Cables:      OSFP-to-OSFP, OSFP-to-2xQSFP112 splitters

XDR (800 Gb/s, 4x200G PAM4):
  Switch side: OSFP cages (twin-port)
  HCA side:    OSFP (ConnectX-8 era)
  Cables:      OSFP-to-OSFP, OSFP-to-2xOSFP splitters

HDR (200 Gb/s, 4x50G PAM4):
  Switch side: QSFP56
  HCA side:    QSFP56
  Cables:      QSFP56-to-QSFP56, QSFP56-to-2xQSFP56 splitter

The "twin-port OSFP" on Quantum-2 is worth understanding: one physical OSFP cage carries two independent 400G ports. So a 32-cage QM9700 is a 64-port switch. When you order cables for a leaf-to-spine link you're typically using OSFP-to-2xQSFP112 splitters at the HCA end (since CX-7 NICs are single-port QSFP112) and OSFP-to-OSFP at the switch-to-switch end.

Bend radius and labeling

Two operational hazards that look trivial until you've debugged them at 3am.

Bend radius: optical fibers have a minimum bend radius (typically 30 mm for AOC). Tight bends produce signal loss that doesn't show up at install but appears as slow link degradation over weeks. Stay above the published minimum. Use cable management arms, not zip-ties pulled tight.

Labeling: every cable, both ends, with destination port. When you have 1000+ cables in a fabric, "trace this cable" is a 30-minute job per cable if you can't read labels. The discipline:

Format: <src-switch>:<src-port> <-> <dst-device>:<dst-port>
Example: SW-LEAF-03:25/1 <-> NODE-014:HCA-mlx5_2

Label both ends, stick the label 10 cm from the connector (not on it),
print durably (printed labels last; handwritten do not).

When a cable is replaced, the new one gets new labels. If labels disagree with the partition config or ibnetdiscover output, the labels are wrong, fix them immediately — the next person debugging will trust the labels and waste hours.

Sizing rule of thumb

For a rail-aligned AI fabric with 8 HCAs per node:

nodes * 8 = endpoints
endpoints / (ports_per_leaf / 2)  =  leaves_needed_for_full_bisection
leaves_per_rail = leaves_needed / 8
spines_per_rail = leaves_per_rail (in 2-tier full-bisection)
total_switches = leaves_needed + spines_per_rail * 8

Example: 128-node, 8-HCA, NDR (Quantum-2 64-port):
  endpoints = 1024
  leaves = 1024 / 32 = 32  (32 down + 32 up per leaf)
  leaves_per_rail = 32 / 8 = 4
  spines_per_rail = 4
  total_switches = 32 + 4 * 8 = 64 switches

64 Quantum-2 switches for a 1024-GPU fabric is roughly the right order of magnitude. Throw in 2-4 management switches (1G/10G ethernet for OOB), an SM host (or 2 for redundancy), and you have a switch BOM.

Beyond ~2048 endpoints you start needing 3-tier (leaf / spine / core) and the math compounds; consider Quantum-X800 Q3400-RA director-class for the spine layer to cut switch count by a factor of 2-3.

See IB switch hardware for per-product detail and a fuller BOM example.

Common failure modes

See also

• InfiniBand fabric primer
• InfiniBand switches: L2 verification
• InfiniBand switch hardware (Quantum / Quantum-2 / Quantum-X800)
• InfiniBand implementation runbook
• InfiniBand vs RoCE decision guide
• Subnet Manager + partitions
• NCCL multi-node tuning
• GPUDirect / GDR
• PCIe topology

External authoritative references:

• NVIDIA DGX SuperPOD H100 Reference Architecture
• NVIDIA DGX SuperPOD GB200 Reference Architecture
• NVIDIA Quantum-2 InfiniBand Platform overview
• InfiniBand Trade Association specifications