InfiniBand switches: L2 verification and why it matters

The first time you SSH into an InfiniBand director switch you'll notice something missing: there's no show ip route. There's no BGP, no OSPF, no L3 anything. That's because everything in IB is L2. When we say "verify a switch" in IB land, we mean: check ports are up, check the SM is reachable, check the partition keys match, check the routing tables the SM pushed look right, check the buffer/congestion counters aren't melting.

This page is the operational checklist for "is this switch healthy and doing the job."

Why "L2 verification" is the right framing

In ethernet, when something's broken you instinctively reach for show ip route and traceroute. In IB those don't exist. The closest equivalents:

If you internalize "everything is L2 and the SM owns the routing tables," IB debugging stops being mysterious.

How to log into a switch and what to check

NVIDIA Quantum (and Mellanox SX-series) switches run MLNX-OS (or NVOS on newer Quantum-X800). SSH in with the management IP, default user admin. The CLI is Cisco-style.

switch-leaf-01> enable
switch-leaf-01# show interfaces ib 1/1
Interface ib1/1 status:
  Description:           link to gpu-01 mlx5_0
  Type:                  IB
  Width:                 4X
  Speed:                 NDR (400 Gbps)
  State:                 Active
  Phys state:            LinkUp
  PKey:                  default 0xffff
  LID:                   589
  GID:                   fe80::a288:c2ff:feab:cdef
  Symbol errors:         0
  Link error recovery:   0
  Link downed:           0
  Port rcv errors:       0
  Port xmit discards:    0
  Port xmit constraints: 0
  Port rcv constraints:  0
  Port xmit data:        12.4 TiB
  Port rcv data:         12.5 TiB

The fields you care about, in order:

1. State: Active = traffic OK. Initializing = SM hasn't finished. Down = wire problem.
2. Phys state: LinkUp/Polling/Disabled. LinkUp + state Initializing = SM not running or PKey mismatch.
3. Width / Speed: should match what the HCA wants. 4X NDR for ConnectX-7. If it's 1X or 2X, lanes failed to train.
4. Symbol errors: should be 0 or growing very slowly. Anything more than a handful per hour = cable or transceiver going bad.
5. Link downed: counts how many times the link bounced. > 0 in steady state = flapping link.
6. Port xmit discards: should be 0. Non-zero in IB land usually means buffer credits exhausted (congestion).

Subnet manager state on the switch

switch-leaf-01# show ib sm
sm running                          : yes
sm priority                         : 0
sm state                            : master
sm sweep interval                   : 10
sm subnet-prefix                    : fe80::
sm log level                        : INFO

sm running yes / sm state master = this switch IS the active SM. If you don't want it to be (because you have a dedicated SM elsewhere), set priority to 0 and let the other SM take over with priority > 0.

If you have multiple director switches each thinking they're master, that's a split-brain. SM should always be exactly one master in steady state. See Subnet Manager doc.

Partition view

switch-leaf-01# show ib pkey
PKey index  PKey      Membership
0           0x7fff    full       <-- default partition, unrestricted
1           0x8001    full       <-- tenant A
2           0x8002    full       <-- tenant B

The 0x prefix is the partition key. 0x7fff is the default partition (limited to 0xFFFF in classic IB, but high-bit-stripped to 0x7FFF in some encodings). Customers' tenant traffic carries non-default PKeys.

Subnet manager elections

When multiple SMs see each other on the same fabric, they negotiate by priority (0-15). The highest priority wins; ties broken by lowest GUID. The losers go to standby and only take over if the master drops.

$ saquery SMInfoRecord
SMInfoRecord dump:
        Lid.....................1
        SM_Key..................0x0000000000000001
        ActCount................1234567
        PriorTime...............100
        SMState.................SMINFO_MASTER
        Priority................0
        Reserved................0x00

Multiple SMs printed here = multi-SM cluster. Only one should be in SMINFO_MASTER state.

Best practice for production:

• Two SMs at minimum: one master (priority 0) on a dedicated SM node, one standby (priority 1) on a director switch. Failover takes ~5-30s.
• Big clusters (>1k ports) typically use UFM (Unified Fabric Manager from NVIDIA), which is opensm plus a rich monitoring/automation stack. Worth the license cost when you have hundreds of nodes.
• Don't run the SM on a node that's also a workload host — SM CPU work spikes during fabric reconfiguration.

Partition keys (PKeys) for tenant isolation

PKeys are 16-bit values that tag IB endpoints into "partitions." Two endpoints with non-overlapping PKey membership cannot communicate.

0x8001 = full member of partition 1. 0x0001 = limited member of partition 1. Two limiteds in the same partition can't talk — useful for hub-and-spoke designs (clients limited, server full).

The default partition is 0xffff (full) / 0x7fff (limited). If you bring up a new HCA without configuring custom partitions, it lands in the default and talks to everyone — fine for single-tenant clusters.

For tenant isolation: dedicate 0x8001..0x80NN per tenant, both ends configured as full. Configure in /etc/opensm/partitions.conf:

Default=0x7fff,ipoib,defmember=full : ALL=limited;
TenantA=0x8001,ipoib,defmember=full : 0xa288c2fffeabcdef, 0xa288c2fffe123456 ;
TenantB=0x8002,ipoib,defmember=full : 0xa288c2fffe555555, 0xa288c2fffe666666 ;

Restart opensm. Verify with smpquery PKeyTable <lid>:

$ smpquery PKeyTable 589 1
# PKey table: Lid 589 port 1
# PKey block 0:
0x8001 0xffff 0x0000 0x0000 ...

Endpoint sees its tenant PKey + default. Cross-tenant traffic gets dropped at the source HCA.

Routing algorithms

The SM computes paths and pushes per-switch LFTs. Algorithms:

In opensm.conf:

routing_engine ftree
fat_tree_root_friendly_update_interval 0
log_file /var/log/opensm.log

For a clos / fat-tree (the standard for ML clusters), ftree is the right default. It looks at the topology, identifies leaves vs spines, and assigns LIDs so that ECMP-style spreading works. The downside: it requires symmetric, full-bisection topology — if one cable drops, ftree falls back to minhop and your bandwidth profile changes.

Adaptive Routing (AR) on Quantum-2/Quantum-X800 switches lets the switch hardware pick a different output port per packet when its primary route is congested, somewhat similar to ECMP but at the switch hardware. Has to be enabled both in the SM config and on each switch.

Real-world: debugging a port stuck "Initializing"

Symptoms in ibstat on the host:

Port 1:
        State: Initializing
        Physical state: LinkUp

Wire is good (LinkUp), but SM hasn't promoted to Active. Run through this checklist:

1. Is an SM actually running and reachable?

   $ sminfo
   sminfo: sm lid 0 — no SM responded
   

   If sm lid 0, your SM is down or unreachable. Start opensm somewhere or check why the embedded switch SM stopped.

2. PKey mismatch: SM expects this port to have a PKey it doesn't have, or vice versa. Check /etc/opensm/partitions.conf includes this port's GUID, or the host's ib_uverbs PKey index.

3. Stuck in INIT due to congestion control negotiation: rare, but opensm.log will say so. Restart SM.

4. Cable seated correctly but transceiver faulty: on the host run mlxlink -d <pci> --port_module_state to see optical module diagnostics. Check Tx/Rx power.

5. Switch port admin-disabled: show interfaces ib 1/1 will say Phys state: Disabled instead of LinkUp. interface ib 1/1; no shutdown.

Reading ibdiagnet output

ibdiagnet is the single most useful fabric-wide diagnostic. Run it from any host with a fabric-wide IB connection:

$ ibdiagnet -pc -ls --get_phy_info -o /var/tmp/ibdiagnet2

It produces ~10 files in /var/tmp/ibdiagnet2/:

The "warnings/errors" summary at the top is your trip report. Common entries:

• Symbol Errors > 0 on links → degrading cable, replace.
• Link Width Reduced → cable trained at 1X or 2X instead of 4X. Re-seat or replace.
• Multiple SMs detected → split-brain, fix SM priorities.
• Port not in active state → unhealthy ports listed by name/GUID.

Identifying a flapping link

Flapping = link goes Down then back Up repeatedly. Causes: marginal cable, dirty fiber, mismatched FEC settings.

$ iblinkinfo --switch <switch_guid>
... port 5: 0xa288c2fffeabcdef 1[  ] ==( 4X 100.0 Gbps Active/  LinkUp)==> ...

That snapshot doesn't show flapping. To detect it, compare counters across two snapshots ~30s apart:

$ perfquery -a <switch-lid> 5         # port 5
PortCounters:
        LinkDownedCounter:..............3
        ...
$ sleep 30
$ perfquery -a <switch-lid> 5
PortCounters:
        LinkDownedCounter:..............7

LinkDowned grew by 4 in 30s = flapping. Reset counters with perfquery -R <lid> <port> after fixing.

Switch firmware: mft and flint

Mellanox Firmware Tools (MFT) is how you update HCA and switch firmware:

$ mst start
$ mst status -v
DEVICE_TYPE         MST                    PCI       RDMA       NET
ConnectX7(rev:0)    /dev/mst/mt4129_pciconf0  d7:00.0  mlx5_0   net-ens3np0

$ flint -d /dev/mst/mt4129_pciconf0 query
Image type:            FS4
FW Version:            28.39.2048
FW Release Date:       12.10.2025
PSID:                  MT_0000000838

Updates:

$ flint -d /dev/mst/mt4129_pciconf0 -i fw-ConnectX7-rel-28_40_2300-MCX755106AS-HEAT_Ax-UEFI-14.36.21-FlexBoot-3.7.500.bin burn

Switch firmware via flint from the host through the management connection or via the switch CLI directly:

switch-leaf-01# image fetch scp://user@scp-server/path/fw.img
switch-leaf-01# image install fw.img
switch-leaf-01# image boot next
switch-leaf-01# reload

Regression risks to know:

• Routing engine changes: a firmware bump may change default ftree behavior. If your cluster relied on a specific traffic pattern, expect a perf change.
• AR (Adaptive Routing) toggles: some firmware versions enable AR by default. If your tenant doesn't have AR-aware NCCL versions, you can hit out-of-order delivery costs.
• PFC/QoS defaults: shouldn't matter for IB, but on RoCE switches sharing infra it matters a lot.

Always: upgrade one switch first, run a 24h soak with NCCL workloads, then rolling upgrade the rest. Never upgrade the entire fabric at once.

Quick-reference command cheat sheet

See also

• InfiniBand fabric primer
• Subnet Manager + partitions
• RDMA fundamentals
• Network performance tuning
• NCCL multi-node tuning