Subnet Manager + partitions: tenant isolation in InfiniBand

In ethernet, no single component "owns" the topology. Switches independently learn MACs, run STP, exchange BPDUs, and converge. In InfiniBand, the Subnet Manager owns everything: which port gets which LID, which switch forwards which destination through which port, which endpoints can talk to which others (PKey enforcement). The fabric is dumb without it. So understanding the SM is understanding how the fabric actually behaves.

This page covers what the SM does, where it usually lives, how partitions work, and the failure modes you'll have to debug.

What the SM is responsible for

Five jobs, in approximate order of frequency:

1. Topology discovery — sweep the fabric every N seconds, build a graph of switches and HCAs.
2. LID assignment — pick a 16-bit LID for every endpoint port.
3. Routing — compute paths (minhop / ftree / etc.) and push Linear Forwarding Tables to switches.
4. Partition enforcement — distribute PKey tables to endpoints and switches; isolation lives here.
5. Multicast — manage Multicast Forwarding Tables (MFTs) for IB multicast groups (used by IPoIB ARP, MPI broadcast in some cases).

Plus event handling: when a port goes down, when a new node joins, the SM gets a trap, runs a partial sweep, recomputes routes, pushes deltas. This is the "fabric reconfiguration" event you sometimes see in opensm.log causing brief stalls.

A single subnet has exactly one master SM at any time. Standby SMs exist for redundancy and elect a master via priority + GUID tiebreak.

Where the SM lives

Three deployment patterns:

For a 16-32 node cluster, embedded SM on a leaf switch is fine. For 64+ nodes or when you want isolation between clusters, dedicate a small server (4 cores, 16 GB RAM is plenty) and run opensm. For 256+ nodes or anything customer-facing, use UFM.

Finding your SM

$ ibstat | grep "SM lid"
                SM lid:                 1

$ sminfo
sminfo: sm lid 1 sm guid 0x506b4b03000abcde, activity count 1234567 priority 0 state 3 SMINFO_MASTER

SM lid 1 is the canonical master location — the master almost always claims LID 1 for itself. If sm lid 0 shows up, no SM is responding. If sm guid differs across nodes, you have multiple SMs and they aren't all seeing each other (likely partition or network problem).

For more detail, query the SA (Subnet Administrator, lives alongside SM):

$ saquery -o all | head
SAQuery type: SAQuery
SubnAdmGet 0x10 SMInfoRecord
SMInfoRecord dump:
        Lid.....................1
        SM_Key..................0x0000000000000001
        ActCount................1234567
        PriorTime...............100
        SMState.................SMINFO_MASTER
        Priority................0

saquery SMInfoRecord lists all SMs that announced themselves on the subnet — useful for finding standbys.

Configuring opensm

Minimal /etc/opensm/opensm.conf:

# Identity / priority
sm_priority 0
guid 0x506b4b03000abcde       # bind to a specific HCA port (by GUID)

# Routing
routing_engine ftree
log_max_size 4096
log_file /var/log/opensm.log

# Partitions
partition_config_file /etc/opensm/partitions.conf

# Sweep interval
sweep_interval 10

# QoS (optional)
qos TRUE
qos_policy_file /etc/opensm/qos-policy.conf

Start with systemctl start opensm. Logs in /var/log/opensm.log. A healthy startup looks like:

SUBNET UP
INITIALIZING_TO_MASTER: master priority=0 GUID=0x506b4b03000abcde
Routing time : 0.045 seconds
LFT setup time : 0.078 seconds
SWEEP DONE: 0.234 sec

When something is wrong:

SM: error - port has no GID                    -> stuck in INIT, won't go Active
SM: PKey table mismatch on lid 589              -> partitions misconfigured
SM: link state mismatch                          -> fabric flapping during sweep

Partitions explained with example

Two tenants on the same fabric, completely isolated.

partitions.conf:

# Default — required, all members limited (so they can talk to SM but not each other)
Default=0x7fff,ipoib,defmember=full : ALL=limited;

# Tenant A: research team, 4 nodes
TenantA=0x8001,ipoib,mtu=5,rate=12,defmember=full :
        0xa288c2fffeabcde1,
        0xa288c2fffeabcde2,
        0xa288c2fffeabcde3,
        0xa288c2fffeabcde4 ;

# Tenant B: production training, 8 nodes
TenantB=0x8002,ipoib,mtu=5,rate=12,defmember=full :
        0xa288c2fffe111111,
        0xa288c2fffe222222,
        0xa288c2fffe333333,
        0xa288c2fffe444444,
        0xa288c2fffe555555,
        0xa288c2fffe666666,
        0xa288c2fffe777777,
        0xa288c2fffe888888 ;

What this gives you:

• Default partition still exists; everyone is a "limited" member, so they can't talk to each other through the default. Required for SM/SA traffic.
• TenantA endpoints get PKey 0x8001 with full membership — they can talk to each other.
• TenantB endpoints get PKey 0x8002 with full membership.
• A TenantA host trying to send to a TenantB host gets its packet dropped at its own HCA (PKey enforced by sender HCA on egress — yes, there's some trust here, see below).
• ipoib flag tells the SM to also create an IPoIB multicast group for that partition.
• mtu=5 = MTU 4096, rate=12 = NDR. Both clamps are advisory.

After editing, signal opensm:

$ kill -HUP $(pgrep opensm)

It re-reads partitions.conf and re-pushes PKey tables to all switches and endpoints. Verify with smpquery PKeyTable <lid> 1:

$ smpquery PKeyTable 589 1
# PKey table: Lid 589 port 1
0x8001 0x7fff 0x0000 0x0000 ...

This endpoint sees its tenant PKey (0x8001 full) and the default (0x7fff full because we set defmember=full in Default). Anything else gets PKey-mismatch dropped.

A note on PKey enforcement

PKey enforcement on IB is "trust the source HCA." A malicious tenant with root on a node could send packets with arbitrary PKeys, and the switch ingress would have to enforce. Modern switches do enforce ingress PKey checks (configurable per port), but the default behavior was historically lenient. For real multi-tenant isolation:

• Set pkey_enforce per port in switch config.
• Restrict access to /dev/infiniband/issm and /dev/infiniband/umad on hosts (these allow setting PKey directly).
• In Kubernetes, RDMA Shared Device Plugin / SR-IOV with separate VFs per tenant is what you actually want.

Configuring a switch-embedded SM

On NVOS / MLNX-OS:

switch-leaf-01# configure terminal
switch-leaf-01(config)# ib sm enable
switch-leaf-01(config)# ib sm priority 0
switch-leaf-01(config)# ib sm routing-engine ftree
switch-leaf-01(config)# ib sm partition file partitions.conf
switch-leaf-01(config)# write memory

The partition file syntax is the same as opensm. You upload it via image fetch or paste through the CLI.

Multi-SM redundancy

Pattern: priority-0 master on a dedicated SM node, priority-1 standby on a switch.

# Node SM
$ cat /etc/opensm/opensm.conf
sm_priority 0

# Switch SM
switch-leaf-01(config)# ib sm priority 1

Failover steps when the master dies:

1. Standby's keepalive timer expires (~10s).
2. Standby SM re-elects itself. It sweeps the fabric, recomputes routes, pushes new LFTs.
3. Existing QPs survive the LID re-assignment in most cases (IB endpoints cache LIDs but reload on SA event).
4. Total disruption: usually 5-30s; some workloads see a tail of "vendor err 81" if their RC retry timeout is shorter than the failover.

To pre-emptively migrate (e.g., for maintenance):

switch-leaf-01(config)# ib sm priority 15        # demote
# wait for standby to take over
$ saquery SMInfoRecord                            # confirm new master
switch-leaf-01(config)# ib sm disable             # take it offline

Failure modes

SM not running

Symptom: all ports stuck in Initializing. sminfo returns sm lid 0.

Fix: start opensm somewhere, or enable embedded SM on a switch. If opensm is running but not master:

$ systemctl status opensm
Active: active (running)

$ tail -f /var/log/opensm.log
SM: another master detected — going to standby

Either it's seeing a stale SM record, or someone else's SM is louder. Check saquery SMInfoRecord for who else is announcing. Bump priority or stop the other.

Two SMs fighting (split-brain)

Symptom: LIDs change unexpectedly, fabric reconfigures every few seconds, NCCL workloads die randomly with vendor err 81.

$ saquery SMInfoRecord
[SMInfoRecord 1]   priority=0 SMINFO_MASTER  lid=1   guid=0xa1...
[SMInfoRecord 2]   priority=0 SMINFO_MASTER  lid=42  guid=0xb2...

Two masters both at priority 0 — they don't see each other (network-partitioned, or one's SMP traffic is being filtered). Find why they can't communicate, fix the path, then one will demote.

Fix in the meantime: stop one of them, raise its priority, restart.

Partition not propagated to a node

Symptom: a single endpoint can't talk to its tenant peers. smpquery PKeyTable <its-lid> shows only the default partition.

Causes:

• Its GUID isn't in partitions.conf (typo, recently moved).
• SM pushed but the endpoint missed it (rare; SIGHUP the SM to retry).
• Endpoint went down/up between sweeps, missed the broadcast.

Fix: SIGHUP opensm. If it persists, re-add the GUID in partitions.conf (sometimes a stray comma or whitespace breaks parsing — opensm.log will show parser errors).

LID exhaustion

Symptom: SM log says out of LIDs after expanding the fabric. Default LID range is 0x0001-0xBFFF (49152 usable). Reduce lmc if you set it >0, or expand to 24-bit LIDs (modern SMs support enable_extended_lids true and switches that do too).

SA database bloat causing slow sweeps

Symptom: Sweep took N.XXX sec growing. SA DB hits memory limits on the SM host.

Fix: more RAM on the SM host, or move to UFM which keeps SA DB tuned. For very large fabrics (4k+ ports), opensm starts to sweat.

Quick reference

See also

• InfiniBand fabric primer
• InfiniBand switches: L2 verification
• RDMA fundamentals
• NCCL multi-node tuning
• GPUDirect / GDR