NVLink / NVSwitch: the GPU-to-GPU fabric

NVLink is what lets a multi-GPU server not be a PCIe-bottlenecked sad pile. PCIe gen5 x16 is 64 GB/s per direction. NVLink 4 (Hopper) is 900 GB/s per GPU bidirectional. NVLink 5 (Blackwell) is 1.8 TB/s. The difference shows up the moment you do tensor parallel or any all-to-all between GPUs on the same node.

NVSwitch is what makes NVLink flat. Without NVSwitch, GPUs are connected to each other in a bipartite mesh — a topology where some pairs have direct links and others don't. With NVSwitch, every GPU on the baseboard talks to every other GPU at full NVLink rate through a switch fabric.

This page is what you need to know operationally: speeds across generations, how to debug a degraded link, what it looks like in NCCL, and how it interacts with topology files.

NVLink generations

Each NVLink "link" is multiple SerDes lanes underneath. Per-link rate scales each generation; lane count per GPU also bumps. Per-link NVLink 4 = 50 GB/s bidirectional (26.5 GB/s per direction including overhead). With 18 links per GPU you get 18 × 50 = 900 GB/s.

The number you'll see in nvidia-smi nvlink -s is the per-link, per-direction rate:

GPU 0: NVIDIA H100 80GB HBM3
  Link 0: 26.562 GB/s
  Link 1: 26.562 GB/s
  ...
  Link 17: 26.562 GB/s

26.562 GB/s × 18 = 478 GB/s × 2 (bidirectional) ≈ 900 GB/s aggregate. Some of that is overhead; effective NCCL bandwidth is ~450 GB/s on the busbw scale.

NVSwitch generations

How 8 GPUs become a flat 900 GB/s mesh: the 8-GPU HGX baseboard has 4 NVSwitch chips. Each GPU's 18 NVLinks are split across all 4 NVSwitches (a few per switch). That gives every pair of GPUs 4 paths through the switch fabric. NCCL spreads traffic across all 4 paths and hits ~450 GB/s busbw between any two GPUs — same as adjacent NVLink-direct pairs. That's the "flat all-to-all" property.

NVL72: rack-level NVLink

The Blackwell era introduces NVL72 — a 72-GPU rack where all 72 GPUs share a single NVLink domain via external NVSwitch trays. From the NCCL/CUDA point of view, those 72 GPUs are one giant baseboard. Allreduce between them runs at NVLink 5 speeds (1.8 TB/s).

NVL72 is one rack at 120 kW with direct-to-chip liquid cooling. From an operations standpoint it's a different beast — see GPU generations for the cooling/power realities.

Reading nvidia-smi nvlink

Per-link state

$ nvidia-smi nvlink -s
GPU 0: NVIDIA B200 (UUID: GPU-...)
     Link 0:  53.125 GB/s
     Link 1:  53.125 GB/s
     ...
     Link 17: 53.125 GB/s
GPU 1: NVIDIA B200 ...
     Link 0..17: 53.125 GB/s

Healthy = all links present and at expected per-link rate. Per-link rate for B200 (NVLink 5) is 53.125 GB/s per direction.

If a link is missing or shows <inactive>, that link is dead.

Per-link error counters

$ nvidia-smi nvlink -e
GPU 0: NVIDIA B200
     Link 0:
         Replay Errors: 0
         Recovery Errors: 0
         CRC Errors: 0
     Link 1:
         ...

Steady non-zero on Replay/Recovery/CRC = degrading link. A handful over the lifetime of a node is fine; growing fast under load is not.

NVLink throughput counters

$ nvidia-smi --query-gpu=nvlink.utilization.tx,nvlink.utilization.rx --format=csv -i 0 -l 1

Real-time NVLink utilization. During NCCL allreduce on a single node you should see >70% on most links.

dcgmi for per-link state

$ dcgmi nvlink -s
+----------------------+
|  NvLink Link Status  |
+----------------------+
GPUs:
    gpuId 0:
        U U U U U U U U U U U U U U U U U U
    gpuId 1:
        U U U U U U U U U U U U U U U U U U
    gpuId 2:
        U U U U U U U U U U U U U U U U U U
    gpuId 3:
        U U U U U U U U U U U U U D U U U U
                                  ^ Link 13 is Down
    gpuId 4-7: U×18
NvSwitches:
    No NvSwitches found.
Key: Up=U, Down=D, Disabled=X, Not Supported=_

143 of 144 links Up = one bad link. NCCL intra-node ring assumes ALL 18 links per GPU. A single missing link is enough to break NCCL communicator init with unhandled cuda error on every rank, even though nvidia-smi -L looks fine.

dcgmi health for the human-readable summary

$ dcgmi health -c
+---------------------------+----------------------------------------------------------+
| Health Monitor Report                                                                |
+===========================+==========================================================+
| Overall Health            | Failure                                                  |
| GPU                       |                                                          |
| -> 3                      | Failure                                                  |
|    -> Errors              |                                                          |
|       -> NVLINK system    | Failure                                                  |
|                           | GPU 3's NvLink link 13 is currently down Check DCGM and  |
|                           | system logs for errors. Reset GPU. Restart DCGM.         |
+---------------------------+----------------------------------------------------------+

This is what you want a passive health monitor / Prometheus exporter to scrape. Don't rely on DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL alone — that's 0 on idle GPUs and won't tell you a link is dead until a job lands and dies. Use the per-link state field IDs (DCGM_FI_DEV_NVLINK_LINK_*) directly.

Failure modes

Single link down on one GPU

What you see: dcgmi nvlink -s shows a D somewhere. NCCL workloads landing on this node fail at COMM_INIT with "unhandled cuda error" on all 8 ranks. Other nodes cascade because the comm doesn't form.

Real-world example: B200 node had GPU 3 link 13 in <inactive> state for 11 hours before any human noticed (DCGM health monitor caught it autonomously). Single-node test FAILED at COMM_INIT after 7.7 s of allocation. Multi-node test cascaded to a partner node. Control test (without the bad node) passed at 4153 Gbps busbw — proving the fault was localized to one link on one GPU.

Diagnostics to capture for an RMA:

# 1. Per-link state matrix
dcgmi nvlink -s

# 2. Health
dcgmi health -c

# 3. Per-link error counters
dcgmi nvlink -g 3 -e   # GPU index 3

# 4. Throughput (cross-check)
nvidia-smi nvlink -s -i 3
nvidia-smi nvlink -e -i 3   # if Link 13 missing entirely from output, it never trained

# 5. Full GPU info
nvidia-smi -i 3 -q

Capture chassis serial, GPU UUID, GPU serial number, board PN, VBIOS, driver version. RMA the GPU module — possibly an NVSwitch fabric work item if the link 13 endpoint on the switch side is implicated.

NVSwitch fabric error

$ dmesg | grep -i nvswitch
[12345.678] nvidia-nvswitch0: SXid (PCI:0000:09:00.0): 12028, Hierarchical Switch Error: ...

SXid 12028 etc. are NVSwitch hardware errors. If they're persistent, the NVSwitch chip needs replacement (whole baseboard typically). NVIDIA Fabric Manager logs to /var/log/fabricmanager.log — check there for state changes and reset attempts.

Fabric Manager not running

For H100+ HGX boards, NVIDIA Fabric Manager (nvidia-fabricmanager.service) MUST be running. Without it NVSwitch is in an unconfigured state — GPUs see each other but not at flat 900 GB/s.

$ systemctl status nvidia-fabricmanager
● nvidia-fabricmanager.service - NVIDIA fabric manager service
     Loaded: loaded (/usr/lib/systemd/system/nvidia-fabricmanager.service; enabled)
     Active: active (running) since ...

If you see GPUs at 50 GB/s instead of ~450 GB/s on intra-node nccl-tests, fabric manager is the first thing to check.

NVLink degraded but not down

A link that's flapping or doing heavy CRC retries can cause throughput collapse without showing as "Down." Check nvidia-smi nvlink -e for growing Replay / Recovery / CRC counters. If found, schedule a maintenance window — link is dying.

NVLS errors (CUDA 401)

NCCL WARN Cuda failure 401 'the operation cannot be performed in the present state'

NVLS = NVLink SHARP, the hardware allreduce on NVSwitch. Requires IMEX daemon and a recent enough driver. In containerized environments, IMEX state isn't always propagated. Workaround: NCCL_NVLS_ENABLE=0. You lose the NVSwitch hardware allreduce but keep all RoCE/RDMA + NVLink P2P performance.

How nvidia-smi topo -m translates to NCCL_TOPO

$ nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX

Legend:
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PIX  = Connection traversing at most a single PCIe bridge
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  NV#  = Connection traversing a bonded set of # NVLinks

How to read this for an HGX H100/H200 board:

• GPU↔GPU = NV18: every GPU pair has 18 NVLinks worth of bandwidth via NVSwitch. Flat fabric.
• GPU0..3 ↔ NIC0..3 = PIX/NODE: GPUs in NUMA node 0 are PCIe-near to NICs 0-3.
• GPU0..3 ↔ NIC4..7 = SYS: GPUs in NUMA 0 are far from NICs 4-7 (would have to cross UPI). Don't pin those.

For NCCL to use this correctly, the rail binding pairs up GPU0↔NIC0, GPU1↔NIC1, ..., GPU7↔NIC7. Set NCCL_IB_HCA=mlx5_0,mlx5_1,...,mlx5_7 in that order.

For non-DGX hardware (custom HGX OEM boards) NCCL may infer the topology incorrectly, sending GPU0 traffic through NIC4. Then you need NCCL_TOPO_FILE — see NCCL doc.

Reference: bandwidth ratios across the stack

For a B200 node:

NVLink dominates intra-node. PCIe is the bottleneck for GPU↔NIC traffic. That's why GDR exists — even with PCIe gen5, you can't afford to bounce through host RAM.

See also

• PCIe topology + ACS
• NCCL multi-node tuning
• GPUDirect / GDR
• GPU generations
• DCGM monitoring