InfiniBand: a layer-2 fabric primer for ethernet people

If you grew up on ethernet, InfiniBand is a culture shock. There's no IP. There's no ARP. There's no routing protocol you've heard of. There's a thing called a "subnet manager" that's centralized and assigns addresses. And the speeds have names like "HDR" and "NDR" instead of "100GbE." Welcome.

This page is a primer for ethernet operators dropped into an IB cluster. It'll get you to the point where ibstat, ibv_devinfo, and ibnetdiscover output makes sense.

The mental model: IB is not IP

InfiniBand is a complete networking stack — physical layer, link layer (L2), network layer (L3 within the IB subnet), and transport. It does not speak IP natively. It uses its own addressing (LIDs, GIDs), its own routing (managed by the subnet manager), and its own transport (RC/UC/UD/RD QPs at the verbs layer).

IP can ride on top via IPoIB (IP-over-IB), but in modern GPU clusters IPoIB is usually only used for management and DNS/SSH — never for the data plane. NCCL talks the IB transport directly via verbs; IP is irrelevant to the bulk path.

So when someone says "the IB fabric is down," they mean: the subnet manager isn't routing, or LIDs aren't assigned, or the link is physically down. Not "ICMP fails."

Three components

This is fundamentally different from ethernet:

• No spanning tree: SM computes loop-free routes (minhop, fat-tree, dragonfly, etc.) and pushes them to switches.
• No MAC learning: forwarding is by LID, which is centrally assigned and pushed.
• No broadcast domain hassles: there's just the subnet, partitioned by PKey.
• No L3: routers exist (between subnets) but rarely used in a single GPU cluster.

Reading ibstat and ibv_devinfo

$ ibstat
CA 'mlx5_0'
        CA type: MT4129
        Number of ports: 1
        Firmware version: 28.39.2048
        Hardware version: 0
        Node GUID: 0xa088c20300abcdef
        System image GUID: 0xa088c20300abcdef
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 400
                Base lid: 589
                LMC: 0
                SM lid: 1
                Capability mask: 0xa651e848
                Port GUID: 0xa288c2fffeabcdef
                Link layer: InfiniBand

Read order:

1. State: Active — port is up, training, configured. Other states: Initializing (waiting for SM), Down (no link), Armed (intermediate).
2. Physical state: LinkUp — the wire is good. If this says Polling or Disabled, the cable, transceiver, or port is the problem.
3. Rate: 400 — current speed in Gb/s.
4. Base lid: 589 — your endpoint's address on this subnet. Assigned by SM.
5. SM lid: 1 — the SM lives at LID 1. It's almost always 1. If this is 0, no SM is running.
6. Link layer: InfiniBand — confirms IB mode. RoCE cards in IB mode would say InfiniBand here too if you flipped them; check mlxconfig -d <pci> q | grep LINK_TYPE.

ibv_devinfo -v gives more detail (active_mtu, GIDs, max_mr_size, etc.):

$ ibv_devinfo -d mlx5_0 -v | head -40
hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        max_mr_size:                    0xffffffffffffffff
        page_size_cap:                  0xfffffffffffff000
        ...
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               589
                        port_lmc:               0x00
                        link_layer:             InfiniBand
                        GID[  0]:               fe80:0000:0000:0000:a288:c2ff:feab:cdef

LIDs vs GIDs

Both are addresses, but at different layers and different scopes.

Within one subnet, packets use LIDs. The subnet manager pushes Linear Forwarding Tables (LFTs) to every switch keyed by LID. Look up dst LID, find egress port, forward.

GIDs become relevant in two cases:

1. Multi-subnet IB (you'd need IB-to-IB routers — rare).
2. RoCE (which always uses GIDs because there's no LID — the GID encodes the IPv4/v6 address).

In practice, on a single-subnet IB cluster, you can ignore GIDs. On RoCE, GIDs are everything (see RoCE doc).

LMC (LID Mask Count)

Looks like LMC: 0 in ibstat. Lets the SM assign 2^LMC consecutive LIDs to one port for path diversity (different LIDs → different routes through the fabric). LMC=2 means 4 LIDs per port. Used by some adaptive-routing schemes; default is LMC=0 in most clusters.

Speed generations

A few practical notes:

• IB ports are usually 4x (four physical lanes per port), so a "400 Gb/s NDR port" is actually 4 × 100 Gb/s lanes. Some director-class switches support 1x/2x/4x splits.
• Backward compatibility is strict: an HDR (200) HCA in an EDR (100) port runs at 100. Verify with Rate: in ibstat.
• The wire link rate after encoding is what Rate: reports.
• 800 Gb/s XDR is rolling into Blackwell-era clusters paired with NVIDIA Quantum-X800 switches.

A quick sanity check: an H100 SXM with 8x ConnectX-7 NDR NICs can ingest 8 × 400 = 3.2 Tb/s = 400 GB/s, which roughly matches NVLink 4 (900 GB/s per GPU bidirectional). With Blackwell + ConnectX-8 / XDR, you double that.

MTU: IB MTU is not IP MTU

ibv_devinfo shows both max_mtu and active_mtu. 5 means 4096 bytes. All endpoints on the path must agree — if the SM sees mismatched advertised MTUs, it negotiates down to the minimum. Validate with:

$ ibv_devinfo -d mlx5_0 | grep mtu
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)

If active_mtu < max_mtu, something on the path is forcing a smaller MTU — usually a misconfigured switch or a node with IB_PMA_ATTR_PORT_INFO set wrong. Different from IP MTU. Don't confuse them. On RoCE the IB-MTU is bounded by IP-MTU (you can't fragment IB inside an under-sized IP packet).

IPoIB — and why NCCL doesn't use it

IPoIB is a kernel module (ib_ipoib) that exposes an IP-style netdev (ib0, ib1, ...) on top of an IB port. It tunnels IP packets through IB UD (Unreliable Datagram) by default, or RC if you set mode=connected. So you get eth0-like semantics — DHCP, ICMP, TCP — over IB.

When you'd see it:

• The control plane: SSH into IB-attached storage gateways, NFS mounts over IPoIB, monitoring agents.
• Bootstrap traffic for NCCL/MPI rank-0 rendezvous (often via IPoIB if there's no separate management network).

When you wouldn't:

• The data plane. NCCL goes verbs → RDMA WRITE directly. IPoIB adds kernel IP processing and discards the zero-copy advantage. Kills throughput.

A common confusion: someone benchmarks iperf3 over ib0 and sees ~50 Gb/s on a 400 Gb/s port and thinks IB is broken. It's not — IPoIB just doesn't scale that high. The benchmark to run is ib_write_bw, not iperf3, if you want to test IB itself.

ibnetdiscover — topology dump

$ ibnetdiscover -p
Switch  36 0x506b4b03000abcde   # "Quantum-X800 leaf-1"
[1]     "H-a288c2fffeabcdef"[1] (a288c2fffeabcdef) # "host gpu-01 mlx5_0" lid 589 4xNDR
[2]     "H-a288c2fffe123456"[1] (a288c2fffe123456) # "host gpu-02 mlx5_0" lid 590 4xNDR
[3]     "S-506b4b03000fedcba"[5]                   # "Quantum-X800 spine-1" port 5
...

Reads as: switch port → connected device + port. H- prefix = HCA, S- prefix = switch. Cross-reference port states with iblinkinfo to find degraded links.

$ iblinkinfo -l
CA: gpu-01 mlx5_0:
       0xa288c2fffeabcdef       589    1[  ] ==( 4X         100.0 Gbps Active/  LinkUp)==>     1    1[  ] "leaf-1" ( )

If you see ==( 1X 25.0 Gbps Active)==> on a line that should be 4X NDR, you've got a partial link (some lanes failed to train). Replace the cable.

Common operations

Check SM is running

$ sminfo
sminfo: sm lid 1 sm guid 0x506b4b03000abcde, activity count 1234567 priority 0 state 3 SMINFO_MASTER

SMINFO_MASTER = healthy. SMINFO_NOTACTIVE = nothing's running. SMINFO_STANDBY = there's a master elsewhere. See Subnet Manager doc.

Run health check across the fabric

$ ibdiagnet -pc -ls --get_phy_info

-pc clears port counters first, -ls checks link speeds. Report goes into /var/tmp/ibdiagnet2/ibdiagnet2.log. Look for "Errors" and "Warnings" sections — symbol error counts > 0 mean cable degradation.

Validate end-to-end with a small RDMA test

host-A$ ib_write_bw -d mlx5_0 -F --report_gbits
host-B$ ib_write_bw -d mlx5_0 -F --report_gbits host-A

Should hit ~96-97% of line rate. Anything less, something's misconfigured.

When IB things go wrong

See also

• InfiniBand switches: L2 verification
• Subnet Manager + partitions
• RDMA fundamentals
• NCCL multi-node tuning
• GPUDirect / GDR