Implementing an InfiniBand fabric: from rack delivery to production traffic

This is the runbook. If you've read IB architecture and IB switch hardware, you know what to build and what to buy. This page is the order of operations to bring it up and the gotchas at each step. The goal: a fabric that passes nccl-tests at 95%+ of theoretical bandwidth on day one, and stays there.

The structure is eight phases. Each phase is gated — don't proceed to the next until the current one is clean. Most "fabric ran fine in lab and broke under real load" stories come from skipping verification on an early phase and discovering the problem at the NCCL stage.

Phase 0: design

Before the first switch ships you need a finished BOM and rack plan. Inputs:

Outputs:

Switch BOM:
  - 32 x QM9790 leaf
  - 32 x QM9790 spine
  - 2 x SM host (8 cores, 32 GB RAM, dual-port CX-7)
  - 2 x mgmt switch (1G ethernet for OOB)
Cable BOM:
  - 1024 x OSFP-to-QSFP112 3m DAC (HCA in same rack as leaf, mostly)
  - 256 x OSFP-to-QSFP112 5m AOC (HCA in adjacent rack, edge nodes)
  - 1024 x OSFP-to-OSFP 15m AOC (leaf to spine)
Transceivers (built into AOCs / cables): 0 separate
Rack layout:
  - 16 racks of 8 GPU nodes + 1 leaf each (compute racks)
  - 4 racks of 8 spine switches + 2 SM hosts + 1 mgmt switch (switch row)

Tools that help at this stage

• NVIDIA's DGX SuperPOD reference architecture documents have detailed cable counts and rack diagrams; adapt rather than design from scratch.
• A spreadsheet that lists every endpoint, every leaf port, every spine port, and the cable that connects them. This becomes your install checklist and your audit baseline. Format:

src_device       src_port    dst_device      dst_port    cable_type   cable_id
node-001         mlx5_0      leaf-r1-l01     1/1         OSFP-QSFP112-3m  C-0001
node-001         mlx5_1      leaf-r2-l01     1/1         OSFP-QSFP112-5m  C-0002
...
leaf-r1-l01      33          spine-s01       1/1         OSFP-OSFP-15m    C-1025
...

• A topology generator script that takes node count + rail count + leaf/spine counts and outputs the wiring matrix. Saves you from off-by-one errors at 1024-cable scale.

Sanity checks before signing the BOM

• Port count math closes: nodes * rails == total leaf downlinks == total leaf-to-spine uplinks at full bisection.
• Power per rack is under PDU capacity: typical is 60-80 kW per rack for AI, but verify your DC supports it.
• Cable lengths cover the worst-case run: measure your floor plan; add 20% slack.
• Switch row has its own PDUs: don't share with compute racks; a tripped breaker shouldn't take both compute and fabric down.

Phase 1: physical install

When the racks land, the work is mechanical: rack the gear, run the cables, label everything.

Switch placement

Two patterns:

• Top-of-rack (ToR) leaf: leaf switch at the top of each compute rack, 1m DAC down to each node. Compact, less cable, but you need 16+ leaf SKUs distributed across 16+ racks.
• End-of-row leaf: a few high-density racks contain all leafs in one row, with longer (10m+) AOC down to each node. Less leaf SKU diversity, but more cable cost.

For NDR, ToR is overwhelmingly the right call — DAC is cheap and avoids the per-cable cost premium of AOC at scale. Quantum-2 1U leaf at the top of the rack, 8 nodes below, all DAC.

For XDR, the calculation shifts: DAC reach drops to 2m and per-port density doubles (144 ports per leaf), so end-of-row designs with a smaller number of director leafs become competitive again.

Cable management

Run them in three passes:

1. Compute → leaf cables first (in-rack, DAC). Quick, no fiber to bend.
2. Leaf → spine cables (row-to-row, AOC). Run through cable trays. Bend radius matters here — minimum 30 mm for AOC. Don't over-tighten zip-ties; use velcro.
3. Spine → super-spine cables (if applicable). Longest runs, often through structured fiber distribution.

Label every cable end at the time you connect it. Don't batch labeling for "later" — later is when you've forgotten which cable went where.

Validate physical link counts

Before powering on the fabric, walk the rows and count:

• Cables in each leaf cage: should equal downlinks + uplinks per the BOM.
• Cables in each spine cage: should equal downlinks per the BOM (no uplinks in 2-tier).
• Empty cages: should match planned spare ports.

A mismatch at this stage costs you 30 minutes to fix. The same mismatch found at NCCL stage costs you days.

Phase 2: switch firmware and boot

Switches arrive with whatever firmware NVIDIA shipped. Before you build a fabric, lock everyone to a known version.

Bringing up management network first

Every switch has an OOB ethernet management port. Plug those into your management VLAN, configure DHCP or static IPs, label them in DNS:

leaf-r1-l01.fabric.internal   10.50.1.11
leaf-r1-l02.fabric.internal   10.50.1.12
...
spine-s01.fabric.internal      10.50.2.1
spine-s08.fabric.internal      10.50.2.8
sm-master.fabric.internal      10.50.3.1
sm-standby.fabric.internal     10.50.3.2

Set initial passwords + SSH

Default credentials are documented per vendor. Change them on every switch immediately. Push your SSH public keys via Ansible / your config-management tool of choice.

switch-leaf-01> enable
switch-leaf-01# configure terminal
switch-leaf-01(config)# username admin password <newpass>
switch-leaf-01(config)# ssh server enable
switch-leaf-01(config)# write memory

Pin firmware versions

Pick a target firmware (e.g., latest stable for your switch generation as of build start), download from NVIDIA, push to all switches.

switch-leaf-01# show version
NVOS version:  1.5.2
ASIC version:  Quantum-2 firmware 31.2010.5046

# Fetch new firmware
switch-leaf-01# image fetch scp://user@10.50.5.1/fw/nvos-1.5.4.bin
switch-leaf-01# image install nvos-1.5.4.bin
switch-leaf-01# image boot next
switch-leaf-01# reload

After reboot, verify:

switch-leaf-01# show version
NVOS version:  1.5.4

Repeat across every switch. Mismatched firmware in one fabric is the source of mysterious routing inconsistencies — different firmware versions interpret some adaptive-routing and SHARP edge cases differently. Lock the version and treat firmware upgrades as a rolling fleet operation.

Enable management API + telemetry

For UFM or your own monitoring:

switch-leaf-01(config)# protocol nvue
switch-leaf-01(config)# nv set system api gnmi enabled
switch-leaf-01(config)# nv set system api restapi enabled
switch-leaf-01(config)# write memory

NVOS exposes telemetry over gNMI / REST. UFM can scrape this; your own Prometheus exporter can too.

Phase 3: subnet manager bringup

The fabric does nothing until an SM is running. Decide where it lives now (you should already know from Phase 0).

Path A: dedicated opensm host

On the SM host (Ubuntu 22.04 or RHEL 9 typical):

$ apt install opensm libibmad-utils infiniband-diags ibutils
$ systemctl enable --now opensm

Edit /etc/opensm/opensm.conf minimally:

sm_priority 0
guid 0x506b4b03000abcde       # this SM host's HCA port GUID

routing_engine ftree
log_max_size 4096
log_file /var/log/opensm.log
sweep_interval 10

partition_config_file /etc/opensm/partitions.conf

# Modern fabric: enable extended LIDs for headroom
enable_extended_lids true

# Adaptive routing on Quantum-2+
ar_enable true

Restart and verify:

$ systemctl restart opensm
$ tail -f /var/log/opensm.log
SUBNET UP
INITIALIZING_TO_MASTER: master priority=0 GUID=0x506b4b03000abcde
Routing time : 0.045 seconds
LFT setup time : 0.078 seconds
SWEEP DONE: 0.234 sec

$ sminfo
sminfo: sm lid 1 sm guid 0x506b4b03000abcde, activity count 5 priority 0 state 3 SMINFO_MASTER

Repeat on the standby SM host with sm_priority 1. Verify both SMs see each other and one is master:

$ saquery SMInfoRecord
[SMInfoRecord 1]   priority=0 SMINFO_MASTER  lid=1   guid=0x506b...
[SMInfoRecord 2]   priority=1 SMINFO_STANDBY lid=42  guid=0x506b...

See Subnet Manager + partitions for full config.

Path B: embedded SM on a switch

switch-leaf-01(config)# ib sm enable
switch-leaf-01(config)# ib sm priority 0
switch-leaf-01(config)# ib sm routing-engine ftree
switch-leaf-01(config)# ib sm partition file partitions.conf
switch-leaf-01(config)# write memory

Disable embedded SM on every other switch, even if they could run it:

switch-leaf-02(config)# ib sm disable

Multiple embedded SMs at the same priority cause split-brain. Be deliberate.

Initial discovery — sanity checks

Once an SM is running, the fabric should come up to "Active" state on every link. Verify:

$ ibhosts | wc -l
128                 # should match node count

$ ibswitches | wc -l
64                  # should match leaf + spine count

$ iblinkinfo -l | grep -c "Active.*LinkUp"
2048                # one per directional link, depends on topology

$ ibnetdiscover > /tmp/topo.txt
$ wc -l /tmp/topo.txt
4500ish             # spot-check the topology

If counts don't match: physical links are missing, or some HCAs aren't powered on, or a switch firmware is wedged. Resolve every counted-but-not-found device before proceeding — half the fabric "looks fine" while one rack is dark, and you find out under load.

Phase 4: partition and routing config

Default fabric is one partition (default 0xFFFF/0x7FFF), all ports as full members. For multi-tenant or production safety, set up partitions early.

partitions.conf

# Default — required, all members limited so they can talk to SM but not each other
Default=0x7fff,ipoib,defmember=full : ALL=limited;

# Tenant A
TenantA=0x8001,ipoib,mtu=5,rate=12,defmember=full :
        0xa288c2fffeabcde1,
        0xa288c2fffeabcde2,
        ...
        0xa288c2fffeabcdf0;

# Storage (full members of both tenant partitions for shared filesystem access)
Storage=0x8100,ipoib,defmember=full :
        0xa288c2fffe111111,    # storage-gw-01
        0xa288c2fffe111112;    # storage-gw-02

mtu=5 = 4096 byte IB MTU. rate=12 = NDR 400G. Both clamps; the SM enforces.

After editing:

$ kill -HUP $(pgrep opensm)
$ smpquery PKeyTable <node-lid> 1     # verify partition pushed to a node

Routing engine choice

Default ftree is right for fat-tree fabrics. Verify the SM picked it up:

$ grep "routing engine" /var/log/opensm.log
SM: routing engine: ftree

$ grep "Bipartite Fat Tree" /var/log/opensm.log
SM: ftree topology: detected balanced k-ary n-tree

For Dragonfly+ (rare for AI clusters at the size most operators run), use dor or lash per the topology. For irregular topologies, updn.

Adaptive routing

Quantum-2 and Quantum-X800 support hardware adaptive routing. In opensm.conf:

ar_enable true
ar_lid_top_priority 0   # default

On the switch:

switch-spine-01(config)# ib ar enable
switch-spine-01(config)# write memory

Verify on the SM:

$ grep -i "adaptive routing" /var/log/opensm.log
SM: AR is enabled, mode=2

Adaptive routing improves AllReduce in real-world traffic by spreading flows when the static path congests. Without it, you'll see periodic spine-port hot-spots and predictable congestion under heavy load.

SHARP enable

If your fabric is SHARP-capable (Quantum-2+) and your NCCL build supports it, enabling SHARP gets you in-network reductions. Two pieces to install:

1. SHARP Aggregation Manager (sharp_am) on the SM host or a dedicated host:

$ apt install sharp
$ systemctl enable --now sharp_am
$ sharp_smx_ucx_topology -o /tmp/sharp_topo.txt

2. SHARP daemon (sharpd) on every compute node:

$ systemctl enable --now sharpd
$ sharp_query --version

3. NCCL launches with NCCL_COLLNET_ENABLE=1:

$ NCCL_DEBUG=INFO NCCL_COLLNET_ENABLE=1 ./all_reduce_perf -b 8 -e 16G -f 2 -g 8
NCCL INFO Collnet/SHARP plugin loaded
NCCL INFO SHARP: tree depth 3

Without SHARP you're leaving 1.5-2x AllReduce performance on the table for large reductions. With it enabled but misconfigured, you can see worse performance than without — verify the tree depth makes sense and the daemon logs show clean reductions.

Phase 5: per-link verification

Before any data plane traffic, every link should be at expected speed with zero errors.

Per-port physical health (mlxlink)

On each compute node, for each HCA:

$ mlxlink -d $(ls /dev/mst | head -1) --port_module_state
Operational Info
        State                              : Active
        Physical state                     : LinkUp
        Speed                              : NDR
        Width                              : 4x
        FEC                                : Standard RS-FEC

Module Info
        Vendor Name                        : Mellanox
        Vendor Part Number                 : MFA7U10-H030
        Cable Length                       : 30m

Lane Diagnostics
        Effective Physical Errors          : 0
        Effective Physical BER             : <1e-15
        Raw Physical BER                   : <1e-9
        Height Eye Opening [mV]            : 24, 24, 25, 24
        Phase Eye Opening [psec]           : 27, 27, 28, 27

What to look for:

• State Active, Phys LinkUp: the basic case.
• Speed NDR / Width 4x: full speed and lane count. If 1x or 2x, lanes failed to train — re-seat or replace.
• Eye Opening: should be balanced across the 4 lanes (within 20% of each other). Outlier lane = developing failure.
• Raw BER: should be at or below 1e-9; effective (after FEC) should be 1e-15 or better. Higher = optical degradation.

Script this across every HCA in the cluster. Anything that flunks gets fixed before going further.

Per-port state via ibportstate

$ ibportstate <lid> <port>
PortInfo:
# Port info: Lid 589 port 1
LinkState:................Active
PhysLinkState:............LinkUp
LinkWidthSupported:.......1X or 2X or 4X
LinkWidthEnabled:.........1X or 2X or 4X
LinkWidthActive:..........4X
LinkSpeedSupported:.......NDR (or HDR, SDR, etc.)
LinkSpeedActive:..........NDR

LinkWidthActive < LinkWidthSupported = lane training problem. LinkSpeedActive < LinkSpeedSupported = speed negotiation failed.

ibdiagnet full sweep

Run from any host with fabric connectivity:

$ ibdiagnet -pc -ls --get_phy_info -o /var/tmp/ibdiagnet2
-I- Discovery
-I-   Discovered 64 switches and 1024 hosts
-I- Connectivity
-I-   No errors detected
-I- Link Width Check
-I-   No errors detected
-I- Link Speed Check
-I-   No errors detected
-I- Total Errors:        0
-I- Total Warnings:      0

Expected: zero errors, zero warnings on a freshly-built clean fabric. Anything else is a problem to fix before proceeding to Phase 6. Common Phase-5 hits:

• Symbol errors > 0 on a few ports (developing bad cable; replace before they get worse).
• Link width reduced (2X instead of 4X) on a port (re-seat or replace cable).
• Multiple SMs detected (fix priorities).

PFC counters baseline (RoCE sites)

If you're on RoCE (Spectrum-X), grab a clean baseline of PFC counters before any traffic. They should all be zero. After traffic, growth tells you where backpressure is happening.

$ for i in $(seq 0 7); do
    echo "Priority $i:"
    cat /sys/class/net/ens$((3+i))np0/qos/pfc-pause-tx-priority-$i
  done

For pure IB, this doesn't apply (IB uses credit flow, not PFC).

Phase 6: pair-wise RDMA verification

Now check that the fabric actually carries data at expected throughput.

Pair-wise ib_write_bw matrix

For a clean validation, every HCA should be able to push line rate to every other HCA. The easy version is a few hand-picked pairs; the thorough version is a full N×N matrix script.

host-A$ ib_write_bw -d mlx5_0 -F --report_gbits
host-B$ ib_write_bw -d mlx5_0 -F --report_gbits host-A

---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000           388.50             388.46               0.741
---------------------------------------------------------------------------------------

Reference numbers for healthy single-QP:

Anomalies to flag: any pair below 90% of expected. Common causes:

• MTU mismatch (re-check active_mtu on both ends).
• Wrong QP count for line rate (try -q 4 for 4 QPs).
• Adaptive routing routing some packets the long way (latency variance shows up as lower throughput on small messages).
• Cable degradation discovered now under load (re-run mlxlink post-test, eye opening may have shifted).

Multi-QP line rate

For a full check including parallelism:

$ ib_write_bw -d mlx5_0 -F --report_gbits -q 4 -s 1048576

-q 4 = 4 QPs. Hits closer to true line rate for NDR/XDR. Should reach ~395 Gb/s for NDR.

Cross-rail validation

In a rail-aligned fabric, also test cross-rail explicitly to verify spine routing:

host-A$ ib_write_bw -d mlx5_0 -F --report_gbits   # rail 0
host-B$ ib_write_bw -d mlx5_3 -F --report_gbits host-A   # rail 3 to rail 0

This forces the traffic over the spine. Should still hit ~95% of line rate on a healthy fabric.

Phase 7: scale-out NCCL

The data plane is verified. Now exercise it with the actual collective workload.

2-node NCCL test

# On rank 0:
$ mpirun -np 2 -H node-001:1,node-002:1 \
    -x NCCL_DEBUG=INFO -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
    -x NCCL_TOPO_DUMP_FILE=/tmp/topo.xml \
    /path/to/all_reduce_perf -b 8 -e 16G -f 2 -g 8

Expected output (healthy):

   16777216    4194304     float     sum    -1   3245.7   5170.8       N/A
   33554432    8388608     float     sum    -1   6321.2   5304.7       N/A
   67108864   16777216     float     sum    -1  12451.3   5388.7       N/A
   ...
   busbw: 363.5 GB/s          (close to 90% of 8 NIC * 400 Gb/s = 400 GB/s)

busbw is the bus bandwidth — what NCCL achieves accounting for the algorithm's data movement. For 8-rail NDR nodes, target is 350-380 GB/s in 2-node AllReduce.

Scale up: 4 nodes, 8, full

# 4 nodes:
mpirun -np 4 -H node-001:1,node-002:1,node-003:1,node-004:1 ...

# 8 nodes:
mpirun -np 8 -H node-001:1,...,node-008:1 ...

# Full cluster:
mpirun -np 128 -hostfile /etc/hostfile.full ...

Watch busbw at each scale. It should stay flat or only modestly degrade as you scale (rail-aligned design's whole point). Sharp drop at a particular scale = something architectural — likely spine bottleneck or partition not propagated to all hosts.

AllToAll test (the fabric stress test)

AllToAll is the worst case for spine layer pressure — every GPU to every GPU, no rail locality.

$ mpirun -np 128 -H ... ./alltoall_perf -b 8 -e 16G -f 2 -g 8

If AllReduce is healthy but AllToAll is much worse than expected, suspect:

• Adaptive routing not enabled (predictable spine congestion).
• Oversubscribed spine layer (hits hard on AllToAll).
• SHARP hurting performance in AllToAll paths (it's tuned for reductions, can interfere with all-to-all).

See NCCL multi-node tuning for env var tuning at this stage.

Phase 8: ongoing operations

Once the fabric passes phase 7, it's "in production." The ops cadence:

Routine checks

Alerting on the right things

Don't alert on every counter twitch. Alert on:

• PFC pause time growing (RoCE only) — indicates congestion patterns shifting.
• Symbol error rate exceeding threshold (e.g., >100 errors/hour on a port).
• Link state changes outside of planned maintenance.
• SM failover events (whether successful or not).
• Port flapping (>3 link-down events in 1 hour on a port).
• busbw regression (>10% drop from baseline at a fixed message size).

UFM Telemetry feeds these into Prometheus with little additional work; standalone you'll write some Prometheus exporters.

Firmware upgrade strategy

Never all-at-once. The pattern:

1. Soak test on one switch: pick a leaf, drain its nodes (cordon in K8s), upgrade firmware, run NCCL workload through that path for 24 hours.
2. Rolling upgrade by row: upgrade one row at a time, with verification between rows.
3. Spine layer last: spines carry cross-leaf traffic; upgrade them only after all leafs are stable.
4. SM upgrade separately: the SM's opensm package and the switch firmware are different lifecycles. Don't bundle them.

Time budget: a 64-switch fabric should be 2-4 hours of total wall-clock for a careful rolling upgrade, mostly in soak time.

Adding new nodes / racks

When the cluster grows:

1. Validate cabling matches the BOM extension.
2. Power on the new HCAs; SM should pick them up automatically.
3. Add their GUIDs to partitions.conf and SIGHUP opensm.
4. Run ib_write_bw from each new HCA to a representative existing HCA.
5. Run a small NCCL test that includes the new nodes.
6. Add to your monitoring + alerting baseline.

A common trap: people add nodes without updating partitions.conf, and the new nodes land in default-only partition (limited members), can't talk to anyone, looks like "the node is broken." Fix the partition file first.

Common implementation pitfalls

Vendor-letter template (cabling delivery error)

When fabric is delivered with cabling errors, the comms back to the vendor follow the same pattern as other vendor cases — plain prose, factual, specific. Sample:

The cabling installed on April 28 has a rail-alignment error affecting 14 nodes in racks R5 through R8. The expected wiring per the BOM was that each node's HCA index N connect to the leaf labeled rail-N. Specifically, on the 14 affected nodes, HCAs at indices 4 and 5 are swapped — HCA-4 is connected to leaf-rail5 and HCA-5 to leaf-rail4.

ibnetdiscover output for an affected node, node-r5-007:

node-r5-007 mlx5_4 -> leaf-rail5 port 7   (expected: leaf-rail4)
node-r5-007 mlx5_5 -> leaf-rail4 port 7   (expected: leaf-rail5)

The same swap pattern appears on 13 other nodes in the same rack range. NCCL allreduce on the cluster currently runs at 220 GB/s busbw vs the expected 360 GB/s, consistent with rail-misalignment forcing traffic over the spine.

Could you re-cable the 28 affected ports (14 nodes x 2 cables each)? We have a maintenance window April 30 from 03:00-07:00 UTC. If a tech can be on-site at start of window, we'll cordon the affected nodes ahead of time.

That's the format. State what's wrong, where, what was expected vs what's there, what you're asking for, and when.

See also

• InfiniBand fabric architecture
• InfiniBand switch hardware
• InfiniBand vs RoCE decision guide
• InfiniBand switches: L2 verification
• Subnet Manager + partitions
• InfiniBand fabric primer
• NCCL multi-node tuning
• RDMA fundamentals
• GPUDirect / GDR
• Network performance tuning

External authoritative references:

• NVIDIA InfiniBand Cluster Bring-up Procedure
• DGX SuperPOD H100 Reference Architecture (network bring-up sections)
• NVIDIA UFM telemetry guide
• opensm man page (current)