fabric-manager: the NVSwitch coordinator nobody tells you about

If you have an NVSwitch — and on H100, H200, B100, B200, B300, GB200 reference HGX/DGX boards you do — then nvidia-fabricmanager.service is part of your driver stack whether you knew it or not. The kernel driver alone is not enough to bring up an 8-GPU NVLink fabric: you need a userspace daemon that walks the topology, programs each NVSwitch's routing tables, hands every GPU a fabric address, and maintains health. That daemon is fabric-manager.

This page covers what FM is, why running an NVSwitch board without it gives you a "GPU island" where every GPU works individually but nccl-tests shows ~50 GB/s instead of ~450 GB/s busbw, and the operational rules that keep it healthy.

What it actually is

NVSwitch is a chip on the HGX/DGX baseboard that crossbar-switches NVLink lanes between the 8 GPUs. On H100 SXM5 there are 4 NVSwitches per board. On B200/GB200 NVL72 there are 18 NVSwitches per rack (an entire rack-scale fabric). They are not configured by the GPU driver alone — they need a control-plane process that:

1. Enumerates GPUs and switches over PCIe.
2. Validates the cabling/topology against an expected map.
3. Programs routing tables on each NVSwitch so any GPU can talk to any other at full NVLink bandwidth.
4. Assigns each GPU a "fabric address" used for cross-GPU memory accesses.
5. Monitors the fabric for link errors / training failures / degradation.
6. On NVL72: coordinates with peer FMs across the rack to form a 72-GPU domain.

That control-plane process is nvidia-fabricmanager, started as nvidia-fabricmanager.service. Without it, NVLinks may train physically but cross-GPU traffic does not get routed → CUDA P2P, NCCL all-reduce, NVLS, NVLink SHARP all fail or fall back to PCIe.

When you need fabric-manager

Rule of thumb: if nvidia-smi nvlink -s shows links between GPUs at 26 GB/s+ and there are more than 2 GPUs, you have an NVSwitch and you need FM.

The MUST-MATCH rule

This is the single most important operational fact about fabric-manager:

nvidia-fabricmanager package version must equal the NVIDIA driver minor version. Exactly. Down to the patch.

If the driver is 570.124.06, FM must be 570.124.06. If the driver is 550.144.03, FM must be 550.144.03. There is no slack. The protocol between the kernel driver and the FM daemon changes between minor releases, and a mismatch means FM refuses to start.

Failure looks like:

$ systemctl status nvidia-fabricmanager --no-pager
× nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/lib/systemd/system/nvidia-fabricmanager.service; enabled)
     Active: failed (Result: exit-code) since Tue 2025-09-30 14:02:11 UTC; 1min ago
    Process: 4123 ExecStart=/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg (code=exited, status=1/FAILURE)
       PID: 4123 (code=exited, status=1/FAILURE)

$ tail /var/log/fabricmanager.log
[Sep 30 2025 14:02:11] [ERROR] [tid 4123] fabric manager NVIDIA driver 570.124.06 detected, fabric manager binary version 570.86.16 is not compatible
[Sep 30 2025 14:02:11] [ERROR] [tid 4123] fabric manager initialization failed

The fix is always to align them — usually pin both in your config management:

# Ansible / Salt / whatever
nvidia_driver_version: "570.124.06"
nvidia_fabricmanager_version: "570.124.06"
nvidia_peermem_version: "570.124.06"   # ships with driver but worth pinning

Why this matters during driver upgrades

The most common way operators get bitten is upgrading driver via apt without updating FM in lockstep:

# DON'T DO THIS — partial upgrade
apt install --only-upgrade nvidia-driver-570

# DO THIS
apt install --only-upgrade nvidia-driver-570 nvidia-fabricmanager-570 libnvidia-nscq-570

libnvidia-nscq-XXX (NVIDIA System Control Query) is the third version-locked piece — it's the library FM uses to talk to NVSwitches. Mismatch there gives you a vaguer "failed to query NVSwitch" error.

If you already have a mismatch on a node, the safe sequence is:

systemctl stop nvidia-fabricmanager
apt install --reinstall nvidia-fabricmanager-570=570.124.06-1   # exact version
systemctl start nvidia-fabricmanager
journalctl -u nvidia-fabricmanager -f

The service in detail

$ systemctl cat nvidia-fabricmanager
# /lib/systemd/system/nvidia-fabricmanager.service
[Unit]
Description=NVIDIA fabric manager service
After=network-online.target nvidia-persistenced.service
Wants=network-online.target

[Service]
Type=forking
ExecStart=/usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg
PIDFile=/var/run/nvidia-fabricmanager/nv-fabricmanager.pid

[Install]
WantedBy=multi-user.target

Key things to know:

• It depends on network-online.target because in shared NVSwitch mode (NVL72) it talks to peer FMs over the management network.
• It depends on nvidia-persistenced.service because the driver must be initialized on all GPUs before FM can enumerate them.
• It writes its log to /var/log/fabricmanager.log (configurable in fabricmanager.cfg).
• Default config is /usr/share/nvidia/nvswitch/fabricmanager.cfg — read it once, the comments explain everything.

Notable config knobs

# /usr/share/nvidia/nvswitch/fabricmanager.cfg

LOG_FILE_NAME=/var/log/fabricmanager.log
LOG_LEVEL=4                          # 0=fatal, 4=info, 6=debug
LOG_APPEND_TO_LOG=1
LOG_FILE_MAX_SIZE=1024               # MB
LOG_MAX_FILE_ROTATE=3

# Standalone (single-node 8-GPU) vs Shared (NVL72)
FABRIC_MODE=0                        # 0 = standalone, 1 = shared
FABRIC_NODE_CONFIG_FILE=             # only used in shared mode

# What to do on NVLink errors
FM_STAY_RESIDENT_ON_FAILURES=0       # 0 = exit on init failure, 1 = stay running

# How aggressive about isolating bad GPUs
ACCESS_LINK_FAILURE_MODE=0           # 0 = abort init, 1 = degraded mode (skip bad GPU)
TRUNK_LINK_FAILURE_MODE=0            # similar for switch-to-switch links

ACCESS_LINK_FAILURE_MODE=1 is useful in production: if one GPU's NVLink to the switch fails to train, FM will boot up the other 7 GPUs degraded instead of refusing to start the whole node. Caveat: NCCL on 8 GPUs becomes 7-GPU only, your training script needs to handle that.

How to verify FM is healthy

The fastest end-to-end check:

$ nvidia-smi -q | grep -i fabric -A 3
    Fabric
        State                             : Completed
        Status                            : Success
        CliqueId                          : 0

Good states:

• State: Completed + Status: Success → FM has finished init for this GPU, fabric is up.
• State: In Progress → FM is mid-init, normal during the first ~10 seconds after boot.

Bad states:

• State: Standby → FM not running, GPU is in fabric-isolated mode.
• State: Completed + Status: Error → FM tried and failed; check /var/log/fabricmanager.log.

Cross-check with kernel logs:

$ dmesg -T | grep -i nvswitch | tail -20
[Wed Oct  1 09:11:42 2025] nvidia-nvswitch0: Switch initialized
[Wed Oct  1 09:11:42 2025] nvidia-nvswitch1: Switch initialized
[Wed Oct  1 09:11:42 2025] nvidia-nvswitch2: Switch initialized
[Wed Oct  1 09:11:42 2025] nvidia-nvswitch3: Switch initialized

You should see one line per NVSwitch (4 on HGX H100, more on bigger systems). Anything like "switch X is unhealthy" or "trunk link error" is FM-relevant.

For NVLink itself:

$ nvidia-smi nvlink --status | head -20
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-...)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         ...
         Link 17: 26.562 GB/s

H100 has 18 NVLinks per GPU at 26.562 GB/s each (= 478 GB/s per direction unidirectional bandwidth). If any links show 0 GB/s, FM cannot route around them in standalone mode, and nccl-tests busbw will be capped.

End-to-end fabric proof:

# nccl-tests one-liner — should hit ~370-450 GB/s busbw on healthy 8-GPU H100
$ /opt/nccl-tests/build/all_reduce_perf -b 8M -e 4G -f 2 -g 8
#                                                          out-of-place                       in-place
#       size         count    type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                             (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
     8388608       2097152   float     sum      -1   1075.4    7.80    6.83      0   1063.2    7.89    6.91      0
   ...
  4294967296    1073741824   float     sum      -1   13721.7  313.00  273.87      0  13680.4  313.94  274.69      0

If busbw at 4G size shows ~50-100 GB/s on H100, fabric is unhealthy → check FM, NVSwitch error counters, and nvidia-smi nvlink --status.

Common failures

"FM version mismatch with driver"

Already covered above. The two-line fix is align package versions.

"fabric manager couldn't initialize switches"

[ERROR] failed to initialize switch port for switch index 2 port 8

Indicates one specific NVSwitch port had a training failure. NVLinks are like SerDes — they go through a training process at boot to establish symbol lock. A port can fail to train due to:

• Reseated GPU/baseboard (poor mating).
• Aging cable / connector wear.
• Thermal stress (rare).
• Genuine silicon defect (rare-er, but happens).

Re-seat the affected GPU, bring it up, check nvidia-smi nvlink --status again. If repeats: open RMA. Set ACCESS_LINK_FAILURE_MODE=1 if you need the node usable while waiting for replacement.

nvidia-smi -q | grep -i fabric shows "Standby"

FM never finished initialization. Check:

1. systemctl status nvidia-fabricmanager — is it actually running?
2. tail -100 /var/log/fabricmanager.log — what was the last error?
3. lsmod | grep nvidia_nvswitch — is the NVSwitch kernel driver loaded?
4. Is nvidia-persistenced running first? FM ordering depends on it.

NCCL warns "NVLS not supported"

NVLink SHARP (in-network reduction) needs FM in a specific mode and a recent driver/NCCL combo. If FM logs show "NVLS init failed" but the rest is healthy, NCCL falls back to non-SHARP all-reduce. This is a perf optimization, not a correctness issue — only worth chasing if you're missing the last 10-15% of NCCL throughput on large messages.

Driver/FM/NSCQ pinning matrix

This is what you should bake into your image / DKMS pins / Operator config:

Replace 570.124.06 with whatever PB you've standardized on. The only acceptable variation is host kernel — that's tracked separately by DKMS.

See also

• NVIDIA driver stack — the foundation FM sits on
• nvidia-peermem (GDR) — the other commonly-forgotten piece of the driver
• NVLink / NVSwitch — what FM is actually controlling
• NCCL multi-node — proves end-to-end that FM is doing its job
• DCGM monitoring — how to alert on fabric issues in production