InfiniBand topology design — fat-tree, dragonfly+, rail-optimized for GPU clusters

Topology is the architectural decision you cannot undo without re-cabling the entire data center. Buy the wrong switches or lay the cable wrong, and every allreduce in every training job will pay the penalty for the lifetime of the cluster. This page is the design reference: when to choose each topology, what the math looks like, and what operators consistently get wrong.

Why topology choice is load-bearing

A fabric built on NDR Quantum-2 switches runs 400 Gb/s per port. A fabric built on XDR Quantum-X800 switches runs 800 Gb/s per port. Whether either one actually delivers that bandwidth to your training jobs depends almost entirely on topology — specifically:

• Bisection bandwidth: the minimum bandwidth between any two equal halves of the network. Training collectives (allreduce, allgather) move data in patterns that look like a worst-case bisection test.
• Cost vs. non-blocking: a fully non-blocking (1:1) fabric requires the most switches and cables. An oversubscribed fabric is cheaper but slower under load.
• Blast radius: when a switch fails, how many GPUs go dark? Topology determines this.
• Cabling complexity: a 3-tier fat-tree connecting 4096 GPUs requires order-of-magnitude more cables than a 2-tier connecting 512. Operations teams have to live with this.

Real numbers to keep in mind throughout:

"NDR" is the dominant tier for clusters built in 2023–2025. "XDR" is entering production in 2025–2026 (Quantum-X800 / ConnectX-8). XDR doubles per-port BW but the switch count, cabling, and cost roughly halve for the same aggregate.

Fat-tree (Clos) topologies

The mental model

A fat-tree — also called a Clos network — is a multi-tier switching fabric where every device connects to every device at the tier above, and the result is that any leaf can communicate with any other leaf at full speed with zero contention. It's called "fat" because the inter-tier links are as wide as the intra-tier links.

Fat-trees come in 2-tier and 3-tier variants. The choice is almost entirely a function of cluster size.

2-tier fat-tree — ≤ 512 to ≤ 1024 GPUs

In a 2-tier fat-tree, you have leaf switches (which connect to compute nodes) and spine switches (which connect to leafs). Every leaf connects to every spine. Every spine connects to every leaf. No direct leaf-to-leaf links.

2-TIER FAT-TREE — 16 leaf × 16 spine (64-port Quantum-2, NDR 400G)
Maximum endpoints: 16 leafs × 32 downlinks = 512 HCA ports

                 ┌───────┐  ┌───────┐  ┌───────┐  ┌───────┐
SPINE            │ sp-01 │  │ sp-02 │  │ sp-03 │  │ sp-04 │  ...  16 spines total
                 └───┬───┘  └───┬───┘  └───┬───┘  └───┬───┘
                     │ (every spine connects to every leaf — only 4 shown)
  ┌──────────────────┼──────────┼──────────┼──────────┼────── ... ─┐
  │                  │          │          │          │             │
  ▼                  ▼          ▼          ▼          ▼             ▼
┌──────┐          ┌──────┐  ┌──────┐  ┌──────┐  ┌──────┐      ┌──────┐
│lf-01 │          │lf-02 │  │lf-03 │  │lf-04 │  │lf-05 │  ... │lf-16 │
└──┬───┘          └──┬───┘  └──┬───┘  └──┬───┘  └──┬───┘      └──┬───┘
   │                 │         │          │         │              │
 (32 down)                                                        (32 down)
   │
gpu-001..032       gpu-033..064     ...                        gpu-481..512

┌─────────────────────────────────────────────────────────────────┐
│ Per switch: 64 NDR ports total                                  │
│ Leaf:  32 ports down to nodes  +  32 ports up to spines         │
│ Spine: 32 ports down to leaf-1 (2 per leaf × 16 leafs)         │
│        ... one "virtual cable" per leaf-spine pair              │
│ Bisection bandwidth:                                            │
│   16 spines × 32 uplink ports × 400 Gb/s = 204.8 Tb/s          │
│   16 leafs  × 32 downlink ports × 400 Gb/s = 204.8 Tb/s  (1:1) │
│ Non-blocking: YES at 1:1 leaf-to-spine ratio                    │
│ Total cables: 16 leafs × 32 uplinks = 512 leaf-spine cables     │
│               + 512 node-to-leaf cables = 1024 cables total     │
└─────────────────────────────────────────────────────────────────┘

Key sizing rules for a 2-tier fat-tree with 64-port NDR switches:

• Each leaf has K ports total. Use K/2 for downlinks (to nodes) and K/2 for uplinks (to spines).
• With K=64: 32 downlinks × N leafs = max nodes.
• For full bisection: need N_spine × (K/2) ≥ N_leaf × (K/2), so N_spine = N_leaf.
• At 16 leafs + 16 spines (64-port each): 512 node ports, 512 spine uplink ports. Full bisection.
• Practical upper bound for 2-tier with 64-port switches: 512 endpoints non-blocking, ~1024 with 2:1 oversubscription.

When does 2-tier make sense?

• Cluster ≤ 64 servers × 8 GPUs = 512 GPU endpoints (with 8-rail, that's 512 HCA ports × 1 rail, i.e., 64 servers × 8 NICs = 512 total ports).
• You want the simplest possible topology: two hops max between any two nodes.
• Budget: 32 switches (16 leaf + 16 spine) instead of ~100+ for a 3-tier.
• You don't expect to scale much beyond current size. Two-tier doesn't grow gracefully — adding more endpoints requires adding both leafs and spines.

3-tier fat-tree — ≤ 4096 endpoints

A 3-tier fat-tree adds a super-spine (sometimes called "core") layer above the spines. Leafs connect to spines; spines connect to super-spines.

3-TIER FAT-TREE — up to 4096 GPU endpoints (64-port Quantum-2, NDR 400G)

┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│ ssp-01  │ │ ssp-02  │ │ ssp-03  │ │ ssp-04  │  ... 16 super-spines
└────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘
     │           │           │           │
     └─────┬─────┴──┬────────┴─────┬─────┘
           │        │              │
      ┌────┴───┐ ┌──┴─────┐ ┌─────┴──┐
      │ sp-01  │ │ sp-02  │ │ sp-03  │  ... 16 spines (per pod) × N pods
      └────┬───┘ └───┬────┘ └───┬────┘
           │         │          │
      ┌────┴──┐ ┌────┴──┐ ┌────┴──┐
      │ lf-01 │ │ lf-02 │ │ lf-03 │  ... 128 leafs total
      └───┬───┘ └───┬───┘ └───┬───┘
          │         │         │
       nodes      nodes     nodes   (32 nodes each = 4096 total)

┌──────────────────────────────────────────────────────────────────┐
│ MATH (64-port NDR, equal splits at each tier):                   │
│                                                                  │
│ Leaf:       32 down to nodes + 32 up to spines                   │
│ Spine:      32 down to leafs + 32 up to super-spines             │
│ Super-spine: all 64 ports down to spines                         │
│                                                                  │
│ 128 leafs × 32 downlinks = 4096 node ports (endpoints)          │
│ 16 super-spines × 64 ports = 1024 inter-tier paths at full BW   │
│                                                                  │
│ Switch count:                                                    │
│   128 leafs + 64 spines + 16 super-spines = 208 switches        │
│                                                                  │
│ Bisection (1:1): matches downlink bandwidth at every tier        │
│ Total cables (approx): ~12,000–14,000 links for full deployment  │
└──────────────────────────────────────────────────────────────────┘

When does 3-tier make sense?

• More than ~512–1024 endpoints (when 2-tier starts requiring too many switches anyway).
• Scale to 4096 GPU ports non-blocking, or 8192+ with 2:1 oversubscription.
• You can grow incrementally — add leafs within a spine pod, then add spine pods under the existing super-spine layer.
• You need partition-level fault isolation: a spine-pod failure affects only its leaf group.

The trade-off: three tiers means three-hop max path (leaf → spine → super-spine → spine → leaf = 4 hops including downlinks). Latency is slightly higher than 2-tier. For most training workloads the extra hundred nanoseconds don't matter — the fat-tree bisection bandwidth matters far more.

Rail-optimized topologies — the dominant H100/H200/B200 pattern

Why rails exist

An H100 SXM5 server has 8 GPUs. Each GPU needs to communicate with GPUs on other nodes. If you give the whole server a single NIC, that single 400 Gb/s link is shared by all 8 GPUs — you get 50 Gb/s per GPU instead of 400 Gb/s. That's a 8x underprovisioning of the network.

The solution is one NIC per GPU, one NIC per leaf switch — each called a "rail." Rail-aligned means GPU 0 on every server connects to leaf-0, GPU 1 to leaf-1, ..., GPU 7 to leaf-7. NCCL traffic for a given allreduce step mostly stays within one rail, so each allreduce stream has the full 400 Gb/s available.

ASCII reference of the per-server side of the same layout:

RAIL-OPTIMIZED LAYOUT — 8-GPU H100 SXM5 server, 8 rails

  SERVER gpu-01
  ┌────────────────────────────────────────────────────────────┐
  │  GPU-0 ──── NIC-0 (CX-7 NDR) ─────────────── rail-0 link  │
  │  GPU-1 ──── NIC-1 (CX-7 NDR) ─────────────── rail-1 link  │
  │  GPU-2 ──── NIC-2 (CX-7 NDR) ─────────────── rail-2 link  │
  │  GPU-3 ──── NIC-3 (CX-7 NDR) ─────────────── rail-3 link  │
  │  GPU-4 ──── NIC-4 (CX-7 NDR) ─────────────── rail-4 link  │
  │  GPU-5 ──── NIC-5 (CX-7 NDR) ─────────────── rail-5 link  │
  │  GPU-6 ──── NIC-6 (CX-7 NDR) ─────────────── rail-6 link  │
  │  GPU-7 ──── NIC-7 (CX-7 NDR) ─────────────── rail-7 link  │
  └───────────────────────────────────────────────────────────┘
       ↓             ↓             ↓             ↓
  leaf-rail-0   leaf-rail-1   leaf-rail-2   ...  leaf-rail-7

  Each leaf-rail-N connects:
    - All NIC-N ports from every server in this fabric segment
    - Uplinks to spine switches (all spines)
  
  Result: 8 independent rails, each with its own leaf, each running
  full NDR 400 Gb/s per endpoint. NCCL uses all 8 in parallel.

NCCL's multi-rail mode (enabled by NCCL_NET_GDR_LEVEL=2 and the right topology plugin) recognizes the 8 independent NICs and stripes allreduce operations across all 8 rails simultaneously. Peak aggregate per-server bandwidth: 8 × 400 Gb/s = 3.2 Tb/s.

Rail layout in a 64-GPU cluster (8 servers)

64-GPU RAIL-OPTIMIZED CLUSTER — 8 servers × 8 GPUs = 64 GPUs, 8 rails

  Rail 0 (leaf-r0): NIC-0 from srv-01..srv-08 = 8 ports down + 8 uplinks
  Rail 1 (leaf-r1): NIC-1 from srv-01..srv-08 = 8 ports down + 8 uplinks
  ...
  Rail 7 (leaf-r7): NIC-7 from srv-01..srv-08 = 8 ports down + 8 uplinks

  Spine layer (example, 4 spines):
  ┌──────────────────────────────────────────────────────────────┐
  │ sp-01  sp-02  sp-03  sp-04                                   │
  │   │      │      │      │                                     │
  │   └──────┴──────┴──────┘── connect to ALL 8 leaf-rails      │
  │                             (each spine has 8 uplinks, one   │
  │                              to each leaf-rail)              │
  └──────────────────────────────────────────────────────────────┘

  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
  │leaf-r0 │ │leaf-r1 │ │leaf-r2 │ │leaf-r3 │ │leaf-r4 │ │leaf-r5 │ │leaf-r6 │ │leaf-r7 │
  └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘ └────┬───┘
       │          │          │          │          │          │          │          │
  srv01.nic0  srv01.nic1  srv01.nic2  srv01.nic3  srv01.nic4  srv01.nic5  srv01.nic6  srv01.nic7
  srv02.nic0  srv02.nic1  srv02.nic2  ...
  ...
  srv08.nic0  ...                                                                srv08.nic7

  Total switches: 8 leaf-rails + 4 spines = 12 switches for 64 GPUs
  Bisection BW per rail: 8 servers × 400 Gb/s = 3.2 Tb/s per rail
  Aggregate bisection: 8 rails × 3.2 Tb/s = 25.6 Tb/s

How rail alignment pairs with SHARP

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs allreduce operations in the switches rather than at the endpoints. The SHARP tree is rooted at a spine switch and has leaf switches as intermediate aggregation nodes.

For SHARP to work with rail-aligned topology:

• Each SHARP aggregation manager (AM) node must be reachable by the SM on a dedicated port.
• The tree structure must match the fabric topology: one AM per leaf, aggregating up through spines.
• Rail-aligned layouts make SHARP natural: each rail's leaf is an aggregation point for that rail's portion of the allreduce.

A misconfigured SHARP deployment concentrates all aggregation managers on a single leaf, which becomes a hotspot and caps SHARP throughput to that leaf's uplink bandwidth rather than the full fabric. Spread AM assignments evenly: one AM per leaf switch, never more than 2 per physical switch.

Dragonfly+ topology

When fat-tree starts losing

A 3-tier fat-tree for 4096 nodes at NDR uses approximately 208 switches (as computed above). At 8192 nodes, you'd add a fourth tier — super-super-spines — and the switch count exceeds 400. The cable count runs to tens of thousands. Installation time and operational risk scale with cabling complexity.

Dragonfly+ is an alternative that provides similar or better bisection bandwidth with fewer switch hops and fewer cables at very large scale (typically ≥ 2000 endpoints).

Dragonfly+ structure

DRAGONFLY+ — groups connected by global links

  Group G0                  Group G1                  Group G2
  ┌─────────────────┐       ┌─────────────────┐       ┌─────────────────┐
  │  local switches │       │  local switches │       │  local switches │
  │  ┌──┐ ┌──┐ ┌──┐│       │  ┌──┐ ┌──┐ ┌──┐│       │  ┌──┐ ┌──┐ ┌──┐│
  │  │R0│ │R1│ │R2││       │  │R0│ │R1│ │R2││       │  │R0│ │R1│ │R2││
  │  └┬─┘ └─┬┘ └┬─┘│       │  └┬─┘ └─┬┘ └┬─┘│       │  └┬─┘ └─┬┘ └┬─┘│
  │   └──┬──┘   │  │       │   └──┬──┘   │  │       │   └──┬──┘   │  │
  │ nodes│      │  │◄──────►     │      │  │◄──────►     │      │  │
  │      │  global │       │     │ global│  │       │     │      │  │
  └──────┴─ links ─┘       └─────┴─ links─┘       └─────┴──────┴──┘

  Topology properties:
  - Nodes attach to local switches within a group
  - Local switches are fully connected within the group (all-to-all)
  - Groups are connected via global links (typically one global link per pair of groups)
  - Traffic within a group: 1 hop
  - Traffic between adjacent groups: 3 hops (local → global → local)
  - Traffic to non-adjacent groups: may require 2 global hops + local = 5 hops

  At scale (example, 512 groups × 48 nodes per group = 24,576 nodes):
    - ~512 groups, each with ~6 local switches
    - ~3072 local switches
    - ~130,000 global links between groups
    - Compared to fat-tree for same scale: ~5000+ switches with 5 tiers

Fat-tree vs Dragonfly+ trade-off

The crossover: most GPU clusters in the 2024–2026 timeframe that are ≤ 4096 endpoints use 2-tier or 3-tier fat-tree. Dragonfly+ is relevant for clusters of 10,000+ endpoints where the cabling cost of fat-tree becomes prohibitive. The xAI Colossus (100,000+ GPUs) uses Dragonfly+ globally with fat-tree within each pod.

For most operators reading this, fat-tree is the answer. Use dragonfly-style reasoning to pick fat-tree if you're building at ≥ 8k GPUs and want to model alternatives.

Oversubscription ratios

Oversubscription is the ratio of edge bandwidth (nodes to leafs) to spine bandwidth (leafs to spines). A 1:1 (non-blocking) fabric has as much spine bandwidth as edge bandwidth. A 2:1 fabric has half as much spine bandwidth.

OVERSUBSCRIPTION EXAMPLES (64-port NDR leaf):

1:1 (non-blocking):
  Leaf: 32 downlinks × 400G = 12.8 Tb/s edge
         32 uplinks × 400G = 12.8 Tb/s spine
  Result: all 32 nodes can simultaneously saturate their NIC

2:1 oversubscription:
  Leaf: 48 downlinks × 400G = 19.2 Tb/s edge
         16 uplinks × 400G =  6.4 Tb/s spine
  Result: if all 48 nodes push simultaneously, spine is bottleneck
  Typical: bursty workloads, mixed compute/inference, dev clusters

3:1 oversubscription:
  Leaf: 48 downlinks × 400G = 19.2 Tb/s edge
         16 uplinks × 400G =  6.4 Tb/s spine (same 16)
  ... you'd use larger port splits to achieve 3:1
  Result: significant congestion during allreduce in large training jobs

When each ratio is acceptable

1:1 non-blocking: Required for large-scale pretraining (100B+ parameter models). NCCL allreduce in large jobs is essentially a fabric-wide bisection stress test. Any oversubscription directly reduces allreduce bandwidth and increases step time. Rule of thumb: 1% oversubscription → approximately 0.5–1% slower allreduce (varies by collective algorithm and job size).

2:1 oversubscription: Acceptable for fine-tuning, inference clusters, heterogeneous clusters where not all nodes run collectives simultaneously. Also acceptable for dev/QA clusters where budget matters more than peak throughput. Expect 10–20% NCCL bandwidth reduction vs. non-blocking for typical allreduce patterns.

3:1 oversubscription: Only acceptable for truly bursty workloads — storage clusters, general HPC jobs that are compute-bound rather than network-bound, or inference serving where the bottleneck is GPU compute. For any serious distributed training at ≥ 64 GPUs, 3:1 will be noticeable in allreduce performance.

Cabling and transceiver constraints

NDR 400G cable types

At NDR 400G (technically a 4-lane SerDes at 100G per lane), signal integrity is tight. NVIDIA specifies:

• DAC: max 3 m for passive at 400G. Length and flex radius matter — coiled cables introduce loss.
• AOC: verified to 100 m at NDR. Use for inter-rack links (typically 3–30 m in a pod).
• Optical modules: required for anything over 100 m. OSFP form factor (not QSFP-DD) at NDR.

XDR 800G (Quantum-X800)

XDR 800G (8-lane × 100G NRZ, or 4-lane PAM4) has stricter signal integrity than NDR. At time of writing:

• Passive DAC: max 1–2 m (varies by vendor, treat as in-chassis only).
• AOC: 30–100 m depending on vendor qualification.
• Optical: required for inter-rack. FR4 modules are the common choice.

In practice: XDR fabrics will use predominantly optical. Budget for optical modules on every OSFP port except intra-chassis connections.

Routing cables at scale

A 3-tier fat-tree for 4096 GPUs requires approximately 12,000–14,000 cables. Cabling discipline matters:

• Label every end of every cable during installation — switch + port number.
• Document the topology in a cable management tool (most operators use Netbox or a spreadsheet) before a single cable is run.
• Leave 20% slack on cable length for routing flexibility. Running taut AOCs increases the chance of signal degradation from tight bends.
• Color code by rail where possible: all rail-0 cables one color, rail-1 another. Makes debugging crossed rails much faster.

SHARP tree planning

SHARP requires a tree structure overlaid on the fat-tree. Each leaf switch participates as a SHARP aggregation node; spines act as higher-level aggregation points.

Root distribution

The SHARP aggregation manager (AM) is software (runs on a dedicated host or embedded in UFM) that builds and manages the reduction trees. There are typically 1–4 AM roots per fabric.

Common operator mistake: placing all SHARP AM hosts on the same leaf switch. When a large allreduce spans all 128 leafs, traffic must all converge through that one leaf to reach the AM, saturating the leaf's uplinks. The correct placement:

• For ≤ 32 leafs: one AM root, attached to a spine switch (never a leaf).
• For 33–128 leafs: two AM roots on different spine switches.
• For > 128 leafs: consider one AM root per spine-pod, plus a top-level AM on the super-spine.

SHARP tree depth and aggregation efficiency

SHARP achieves best efficiency when:

1. The allreduce spans the full fabric (all nodes participate — SHARP is most efficient at 100% participation).
2. The tree depth matches the fabric tiers (2-tier fat-tree = 2 aggregation levels, 3-tier = 3 levels).
3. Each aggregation node has balanced children — avoid trees where one branch has 64 endpoints and another has 2.

For rail-aligned topologies with 8 rails, run a separate SHARP tree per rail. This maps naturally to the rail structure and means each SHARP operation stays within its rail's leaf-to-spine subgraph.

Subnet Manager placement and HA

Master and standby

Each IB subnet needs exactly one active SM. In small fabrics (≤ 2000 nodes), the embedded SM in a Quantum-2 switch (QM9700) is sufficient. For larger fabrics or multi-partition deployments, UFM (Unified Fabric Manager) on a dedicated host is standard.

HA deployment:

• Primary SM runs on dedicated management host (2 vCPUs, 8 GB RAM is sufficient per NVIDIA docs for ≤ 4096 nodes).
• Standby SM on a second management host, same subnet. Priority set lower than primary.
• SM takeover on primary failure: typically completes in 30–90 seconds (fabric size dependent). During takeover, routing tables are stale but data plane continues for existing QPs.

SM placement rules

• Never place the primary SM on a compute node. A compute node reboot during training is a hard failure mode. Management hosts are more stable.
• Do not co-locate SM with SHARP AM on the same host if the SHARP tree is large. Both can stress-test the SM during fabric events.
• Use dedicated SM ports: on director-class switches, reserve one management port for the SM that is not in the data-plane path.

Migration paths: 2-tier to 3-tier

If you build a 2-tier fat-tree today and expect to grow to 3-tier later, you can reserve ports at the time of initial build:

1. Leaf uplink reservation: in a 64-port leaf, use only 16 downlinks (to nodes) and 16 uplinks (to current spines), leaving 32 ports unconnected. When you add the 3rd tier, repatch those 32 ports to a new spine pod.
2. Spine as future leaf: some operators install more switches than needed initially, calling them "future leafs." They sit idle until the cluster grows. This wastes switch cost but removes the need to re-cable.
3. Additive growth: each time you add 32 nodes, add 1 leaf and its uplinks to existing spines. Grow the spine pool when you hit leaf count × (uplink-per-leaf) > spine port capacity.

Growth example:

Initial:    8 leafs  + 8 spines,   256 GPUs, 2-tier, NDR 400G
Step 1:    16 leafs  + 16 spines,  512 GPUs, still 2-tier
Step 2: Add 16 super-spines, re-route existing spines as "spine pods"
           = 16 leafs × 4 pods + 64 spines + 16 super-spines = 4096 GPUs, 3-tier

The re-cabling at Step 2 is the painful part: every existing spine-to-leaf cable must be checked and some re-patched as the fabric tier structure changes. Budget 3–5 days of skilled cabling labor per 1000 cables.

Cost models (approximate MSRP / list pricing)

These are order-of-magnitude estimates based on publicly available list pricing as of mid-2025. Actual contract pricing is typically 30–50% below list. Use these numbers for budget planning and architecture comparison, not for procurement.

NDR (Quantum-2, QM9700) fabric

For a 512-GPU non-blocking 2-tier fat-tree:

• 16 leaf + 16 spine = 32 Quantum-2 switches: ~$1.9M list
• 512 ConnectX-7 NICs (one per GPU): ~$1.3–1.8M list
• ~1024 cables (mix of DAC intra-rack + AOC inter-rack): ~$250K–$400K list
• Total fabric cost per GPU (approximate MSRP): ~$7,000–$9,000 per GPU endpoint

XDR (Quantum-X800) fabric

For comparable 512 GPU non-blocking 2-tier at XDR:

• Fewer switch ports needed (144 per switch vs 64), so fewer switches
• NIC cost ~2x NDR
• Total fabric cost per GPU (approximate MSRP): ~$12,000–$16,000 per GPU endpoint

The XDR premium is real but the total-cost-per-bit comparison is more favorable: XDR delivers 2x the bandwidth per NIC, so the cost-per-Gbps-per-GPU is competitive or better than NDR for new builds.

Common operator mistakes

1. Undersized spine layer

The most common error: adding more leafs without adding spines. If a 2-tier fabric has 16 leafs and 8 spines (2:1 oversubscription at the spine), that oversubscription is tolerable. If it's allowed to drift to 32 leafs × 8 spines (4:1), allreduce performance degrades significantly. Always track the leaf-to-spine ratio as you grow.

2. Mismatched HCA generations on the same fabric

An NDR fabric can tolerate HDR HCAs on a subset of nodes — they'll just link at 200G. The problem: NCCL's allreduce is as slow as the slowest participant. A single server with HDR NICs in a job that spans HDR + NDR nodes bottlenecks the entire collective at HDR speed. Enforce HCA generation consistency per NCCL process group. If you must mix, keep HDR and NDR on separate Slurm partitions or Kubernetes node pools.

3. Mixing rail-aligned and non-rail-aligned compute

If half your servers are rail-aligned (8 NICs each, one per leaf-rail) and the other half have a single NIC per server (connected to leaf-0 only), NCCL will see asymmetric topology and perform poorly. The SM will route traffic from 8-NIC servers through rail-0's leaf even for rails 1–7, which saturates rail-0's leaf-to-spine uplinks while other rails sit idle. Make rail alignment consistent across all nodes that share a Slurm partition or Kubernetes node pool.

4. Not reserving uplink ports for growth

See the migration path section above. Build in the uplink reservation from day one. The cost of reserving 16 ports per leaf is zero — you're not buying extra hardware, just not plugging cables into ports you'll need in 18 months.

5. Ignoring SHARP tree consistency after fabric changes

When you add nodes, the SM re-computes routing tables and potentially moves SHARP tree nodes. If you haven't re-validated the SHARP AM assignments, you can end up with a SHARP tree that routes through a now-overloaded leaf or has broken aggregation paths. After any significant fabric change (≥ 10% of endpoints), run ibdiagnet --sharp or equivalent and re-validate.

6. Single-homed SM in a dual-spine topology

If the primary SM lives on a host connected to leaf-0, and leaf-0 has a partial failure (some ports down), the SM can still run but can't reconfigure switches it can't reach. Connect the SM host to at least two separate leafs (or to a spine directly) so SM connectivity is independent of any single leaf failure.

Routing algorithms

The SM does more than assign LIDs — it computes and distributes routing tables (LFTs, Local Forwarding Tables) to every switch in the fabric. The routing algorithm matters for training jobs.

Minhop (default)

The SM computes minimum-hop paths for all source-destination pairs. Simple and fast to compute. Works well for fat-tree topologies where minimum hop is usually unique. Drawback: in a fat-tree with identical path lengths, minhop must break ties arbitrarily, which can create hot-spots.

# OpenSM: set routing engine
# In /etc/opensm/opensm.conf:
routing_engine = minhop    # options: minhop, updn, ftree, pqft, lash, ar_updn

Fat-tree routing (ftree)

A routing algorithm specifically designed for fat-tree topologies. Instead of breaking ties arbitrarily, it distributes flows across available equal-cost paths using a deterministic spreading algorithm. Results in more even utilization across spine switches.

For any fat-tree topology (2-tier or 3-tier), ftree or pqft is the recommended routing engine:

# /etc/opensm/opensm.conf
routing_engine = ftree     # fat-tree routing; requires all paths be equal-length
# OR
routing_engine = pqft      # PQFT: a more modern fat-tree variant, handles asymmetric fabrics better

ftree requires a strictly fat-tree topology with all paths equal-length. If you have any nodes with different hop counts (e.g., a server directly connected to a spine, bypassing a leaf), ftree will fall back to minhop. Keep the topology clean.

Adaptive routing (hardware, Quantum-2 / Quantum-X800)

Quantum-2 and Quantum-X800 support hardware adaptive routing — the switch ASIC reroutes individual packets in real-time based on current queue depths, without SM involvement. This provides near-optimal load balancing under all traffic patterns.

Adaptive routing is configured per switch (via UFM or mlxconfig) and works in parallel with the SM's static routing tables. The static LFTs provide the baseline path; adaptive routing overrides the egress port selection per-packet:

# Enable adaptive routing on a Quantum-2 switch (via mlxconfig on the switch itself)
# Or via UFM: Routing → Enable Adaptive Routing → Per-switch

# Verify adaptive routing status via ibnetdiscover output or UFM dashboard
ibnetdiscover -l | grep -i "adaptive"

# Check per-switch AR status:
saquery -C mlx5_0 -P 1 NodeInfo | grep AdaptiveRouting

Adaptive routing is recommended for all NDR/XDR fabrics. It provides 15–25% allreduce throughput improvement over static routing on a loaded fabric (operator rule of thumb; varies with job size and traffic pattern).

SHARP and routing interaction

SHARP aggregation paths are static trees computed at SHARP AM startup. They do not change with adaptive routing decisions. If adaptive routing moves traffic off the path that SHARP is using for a given reduction, SHARP performance is unaffected — it uses its own SHARP-specific VL (Virtual Lane).

Set SHARP_ALLREDUCE_ENABLE=1 and verify the SHARP VL is not shared with data-plane VLs. Use VL 15 for SHARP (the SHARP AM reserves it by default).

Virtual lanes and quality of service

An IB switch supports multiple Virtual Lanes (VLs). Think of them as hardware queues on every port — traffic assigned to VL 0 is queued and forwarded independently from VL 1, etc. Quantum-2 supports up to 8 data VLs (VL 0–7) plus VL 15 (management).

VL assignment for GPU cluster traffic

VL assignment (recommended):
  VL 0  : default NCCL data plane (reduce-scatter, allgather, allreduce)
  VL 1  : storage traffic (Weka, NFS-over-RDMA) — if shared fabric
  VL 2  : SHARP reduction traffic
  VL 15 : IB subnet management (SM, SA queries, MADs)
  VL 3–7: reserved or tenant isolation

VL assignment is done by the SM via SL-to-VL mapping and by the application via the Service Level (SL) set in the queue pair. NCCL sets SL=0 by default. If you want NCCL on VL 1 (for example, to separate it from storage), you must set NCCL_IB_SL=1 in NCCL's environment.

# NCCL environment to set SL (and therefore VL mapping):
export NCCL_IB_SL=0         # Default; maps to VL 0 via SM's SL2VL table
export NCCL_IB_TRAFFIC_CLASS=0   # DSCP/TC for IP-based congestion marking (RoCE)

Congestion control VL usage

Quantum-2's hardware congestion control uses dedicated VLs for notification messages. Do not assign NCCL data to the same VL as congestion notification messages. Verify the SL-to-VL mapping with:

# Query SM's SL-to-VL mapping for a specific port
smpquery -D SLtoVLMap <lid>
# Output shows which VL each SL maps to; verify no collision

Fabric verification tooling

After building or changing a fabric, verify it before running training jobs. These are the commands that matter:

ibnetdiscover — topology map

# Dump full fabric topology to a file
ibnetdiscover > /tmp/fabric-$(date +%Y%m%d).topo

# Verify node count
grep "^CA " /tmp/fabric-$(date +%Y%m%d).topo | wc -l
# Should match expected HCA count

# Check for missing switches
grep "^Switch " /tmp/fabric-$(date +%Y%m%d).topo | wc -l

# Detect links at wrong speed (should all be Rate: 400 for NDR)
grep "Rate 400" /tmp/fabric-$(date +%Y%m%d).topo | wc -l
grep -v "Rate 400" /tmp/fabric-$(date +%Y%m%d).topo | grep "Rate "
# Any non-400G links: investigate

ibdiagnet — fabric diagnostics

ibdiagnet is the comprehensive fabric diagnostic tool. Run it after initial fabric build and after any significant change:

# Full fabric diagnosis (takes several minutes on large fabrics):
ibdiagnet --pc -r --sharp --extended_speeds all -o /tmp/ibdiag-$(date +%Y%m%d)

# Key output files:
#   .log         — overall result, errors, warnings
#   .pm          — port error counters
#   .cables      — cable and optic health
#   .sm          — subnet manager status
#   .sharp       — SHARP tree status

# Quick-check: errors in the log file
grep -E "^ERR|^WRN" /tmp/ibdiag-$(date +%Y%m%d)/*.log | head -50

Common findings and what they mean:

perfquery — per-port error counters

After any training job run, check for error counters. Non-zero PortXmitDiscards is the most important signal:

# Check all ports on a switch (by LID):
perfquery -x 1    # LID 1 is typically the SM's switch

# Check a specific port with extended counters:
perfquery -xS <lid> <port>

# Key counters to watch:
#   PortXmitDiscards     — packets dropped due to congestion or full queues
#   PortRcvErrors        — link-level receive errors (CRC, etc.)
#   SymbolErrorCounter   — physical layer errors
#   VL15Dropped          — dropped management messages (SM can't keep up)

# Clear counters after baseline (after a clean run):
perfquery -x -R <lid>    # -R resets counters

Non-zero PortXmitDiscards after a training run: the fabric has congestion. Investigate which ports show discards — it will reveal which uplinks are saturated and whether the topology is the right size for the workload.

NCCL bandwidth test for end-to-end validation

After topology and cabling are verified with ibdiagnet, confirm application-layer throughput:

# all_reduce_perf: the definitive NCCL topology test
# Run from all nodes simultaneously with MPI

# 2-node test (fastest sanity check):
mpirun -np 16 -host gpu-01:8,gpu-02:8 \
  -x NCCL_DEBUG=INFO \
  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  all_reduce_perf -b 1G -e 4G -f 2 -g 1

# Full cluster test (N nodes):
mpirun -np $(( N * 8 )) -hostfile hostfile \
  all_reduce_perf -b 1G -e 8G -f 2 -g 1 2>&1 | tee /tmp/nccl-bw-$(date +%Y%m%d).log

# Expected output (NDR, 8-GPU server, 8 rails, non-blocking):
# Size(B)  ..  out_of_place_algbw  ..  busbw
# 1073741824   ..  380 GB/s             ..  713 GB/s
# "busbw" is the adjusted bandwidth (accounts for ring algorithm overhead)
# Target: busbw > 350 GB/s for NDR full-fabric test

busbw below 80% of expected indicates: topology misconfiguration, routing issue, SHARP not enabled (if expected), or NIC misconfiguration (incorrect HCA list in NCCL_IB_HCA).

NDR vs XDR: topology implications

Migrating from an NDR fabric to XDR (or building new with XDR) changes the topology math in two important ways:

1. Port density changes

Quantum-X800 switches have 144 ports at XDR 800G vs. Quantum-2's 64 ports at NDR 400G. For the same number of endpoints:

NDR (64-port switches): 512 endpoints = 16 leaf + 16 spine switches = 32 switches
XDR (144-port switches): 512 endpoints = 4 leaf (72 down + 72 up) + 4 spine = 8 switches

4x fewer switches at XDR for the same endpoint count (144-port vs 64-port)

This dramatically simplifies cable runs and reduces per-switch management overhead. A 512-GPU XDR cluster needs fewer patch panels, fewer switch management IPs, and fewer SM LID assignments.

2. NDR–XDR mixed fabrics

ConnectX-7 (NDR) and ConnectX-8 (XDR) can coexist on a Quantum-X800 switch — the X800 supports NDR ports when connected to NDR-capable transceivers. The XDR port negotiates down to NDR if the other end is NDR-only.

This enables incremental migration:

• Build an XDR switch layer; connect new XDR servers with CX-8 at 800G.
• Connect existing NDR servers with CX-7 at 400G (via NDR-compatible transceivers in the XDR port).
• Both generations operate simultaneously.

Performance implication: NCCL treats the mixed cluster as a single fabric. If a job spans NDR-only nodes and XDR nodes, the NDR nodes limit the allreduce speed for the ports they participate in. Keep XDR-only jobs on XDR-only nodes and NDR-only jobs on NDR-only nodes during a mixed-generation transition.

Quick sizing reference

Use this table as a starting point before doing the full math. All numbers assume 64-port NDR Quantum-2 switches, 8-GPU H100/H200 servers, rail-aligned, non-blocking 1:1.

Column meanings:

• Rails: number of independent leaf-to-spine sub-fabrics (= NICs per server)
• Leaf switches: total across all rails
• Spine switches: total across all rails (one spine pool per tier, shared across rails at the top tier)

For XDR Quantum-X800 (144-port switches), divide switch counts by approximately 3–4 (higher port density, fewer switches needed).

Refine these estimates with the full math from the fat-tree and rail sections above before purchasing switches.