PCIe topology + P2P: where ACS, IOMMU, and BAR sizing all collide

For a GPU node, PCIe topology is plumbing. When it works, you don't think about it. When it doesn't, you stare at iperf-equivalents getting half the bandwidth they should and wonder which of {ACS, IOMMU, BAR sizing, root complex P2P} is the broken one. Usually it's some combination, and you only know which after you methodically check.

This page is the operator's view of PCIe on a GPU node: what the topology actually is, what lspci -tv is telling you, and the three knobs (ACS, IOMMU mode, BAR1 size) that matter for GPUDirect.

PCIe basics: speeds and lane counts

H100 / H200 GPUs are PCIe gen5 x16 = 64 GB/s. ConnectX-7 NICs likewise. B100/B200/B300 are PCIe gen5 x16 with PCIe gen6 capability on some configurations.

Verify per-device:

$ sudo lspci -vvv -s 1a:00.0 | grep -E "LnkCap:|LnkSta:"
                LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
                LnkSta: Speed 32GT/s, Width x16

LnkCap = what the device CAN do. LnkSta = what's currently negotiated. If LnkSta is x8 on a card capable of x16, the slot is wired narrow or there's a link training issue. If Speed 16GT/s on a gen5-capable card, the slot is gen4.

A common gotcha: physical x16 slots wired x8 electrically. Confirm both the device thinks it can do x16 and the slot delivered x16 — both numbers must match.

Root complex, switches, endpoints

PCIe is hierarchical:

• Root complex: the CPU's integrated PCIe controller. Top of the tree. Talks to host RAM and to its child slots/switches.
• Root port: each root complex exposes multiple "root ports," each a downstream lane to a slot or PCIe switch.
• PCIe switch: a chip that fans out one root port into multiple downstream slots. Common on GPU baseboards (one switch fans 4 GPU slots out of 1 root port).
• Endpoint: GPU, NIC, NVMe, etc.

Read with lspci -tv:

$ lspci -tv
-+-[0000:c0]-+-00.0  Intel Corporation Device 09a2
 |           +-01.0-[c1-c4]----00.0-[c2-c4]--+-00.0-[c3]----00.0  NVIDIA H100 SXM5 80GB
 |           |                               +-01.0-[c4]----00.0  Mellanox ConnectX-7
 |           ...
 +-[0000:80]-+-...
 +-[0000:40]-+-...
 +-[0000:00]-+-...

Reading top-down:

• [0000:c0] is one root complex (CPU socket 1, NUMA node 1).
• The H100 at c3:00.0 and the ConnectX-7 at c4:00.0 are both downstream of a PCIe switch at c2:00.0.
• They share a parent — that's the topology that allows PCIe peer-to-peer (with ACS off).

If the GPU were under root complex [0000:c0] and the NIC under [0000:00], they'd be in different NUMA domains. P2P traffic between them would have to traverse the inter-socket UPI (Intel) or Infinity Fabric (AMD), which doesn't usually support PCIe P2P at all on most platforms — meaning every transfer bounces through host RAM regardless of GDR config. Cross-socket NIC↔GPU is a hardware design problem, not a software one.

P2P: peer-to-peer DMA

When two PCIe endpoints can directly DMA to each other's memory without going through host RAM, that's P2P. For GPUDirect RDMA, P2P is what lets the NIC read GPU BAR1 directly.

P2P requires:

1. Both endpoints under the same root complex (typically the same CPU socket).
2. No ACS-enabled switches between them (or ACS configured to permit DirectTrans).
3. Root complex / switch hardware that supports P2P (almost all modern Intel / AMD do, but with caveats).

If the path between endpoints crosses the inter-socket interconnect, P2P generally doesn't work. Plan your hardware: GPUs and their paired NICs should be on the same socket.

ACS in detail

Access Control Services — IEEE PCIe spec for forcing PCIe traffic up to the root complex instead of being forwarded peer-to-peer at a switch.

Why it exists: virtualization. A hypervisor assigning VFIO devices to different VMs needs to guarantee that one VM's device can't DMA to another VM's device without going through the IOMMU for translation/filtering. ACS ensures that.

Why it hurts GDR: with ACS forcing traffic upward, "P2P" between GPU and NIC becomes "GPU → switch → root complex → switch → NIC," which (a) adds latency and (b) on most platforms loops through host RAM as part of the root complex's behavior.

Reading ACS state

$ sudo lspci -vvv | grep -A 2 "Access Control Services"
        Capabilities: [220 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

ACSCap = capabilities (what the device CAN do). ACSCtl = control (what's enabled).

• All + in Cap, all - in Ctl = ACS available but disabled. This is what you want for GDR.
• SrcValid+ or UpstreamFwd+ in Ctl = ACS is forcing upward routing. Bad for GDR.

Enumerate every PCIe switch on the box:

$ sudo lspci -vvv | awk '/^[0-9a-f]/ {dev=$1} /Access Control/ {print dev}' | sort -u
00:01.0
00:03.1
40:01.0
40:03.1
80:01.0
...

For each, check ACSCtl. Any switch on the GPU↔NIC path having ACS enabled in Ctl breaks GDR.

Disabling ACS

Three options:

1. BIOS (best, persists across reboots cleanly): "PCIe ACS Control" or "ACS Disable" toggle, usually under "Advanced > PCIe Configuration." Some platforms hide it under "I/O Configuration" or "Intel VT-d / AMD-Vi" submenu.

2. Kernel cmdline override (if BIOS doesn't expose the toggle):

   pcie_acs_override=downstream,multifunction
   

   Disables ACS effects on downstream switch ports and multifunction devices. Logs a warning at boot.

3. Per-device setpci (last resort, fragile):

   $ sudo setpci -s c2:00.0 ECAP_ACS+0x06.w=0x0000
   

   Writes ACS Control Register to all-zeros. Persists until driver reload or reboot. Easy to break.

Production answer: BIOS option if available. Override kernel cmdline if not. setpci only for emergency unblock during incidents.

IOMMU modes

IOMMU translates DMA addresses for security/virtualization. Three modes:

Why GDR cares: pass-through means the IOMMU doesn't have to build per-device translation tables for the kernel's devices. NIC DMA addresses target GPU BAR1 directly. With translated mode, the IOMMU has to know about cross-device mappings, which the IB stack manages but adds overhead and on some platforms doesn't get done correctly.

Verify:

$ cat /proc/cmdline | grep -o 'iommu=[a-z]*'
iommu=pt

If missing, add to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, run update-grub (Ubuntu) or grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL), reboot.

BAR1 sizing

BAR (Base Address Register) is the PCIe-visible aperture into a device's memory. For GPUs, BAR1 is the framebuffer (HBM) aperture.

For GDR the NIC pins memory regions backed by GPU BAR1. If your aggregate pinned MR set exceeds BAR1, registration fails. NCCL's working set is much smaller than the framebuffer, so usually fine. Some workloads (large activation checkpointing, model parallel with full layer pinning) can hit it.

$ nvidia-smi -q | grep -A 3 BAR1
BAR1 Memory Usage
    Total                 : 65536 MiB
    Used                  : 64 MiB
    Free                  : 65472 MiB

Total = the BIOS-exposed aperture. If this is much smaller than expected (e.g., 256 MiB instead of 64 GiB), the BIOS isn't exposing resizable BAR. Two BIOS toggles:

• "Above 4G Decoding" — must be ON for any BAR > 4 GB.
• "Resizable BAR" — must be ON for the BIOS to negotiate the full BAR1 size with the GPU.

Some platforms also need a VBIOS update on the GPU to advertise the larger BAR1 capability. Check with nvidia-smi -q | grep "VBIOS Version" against the latest from NVIDIA's release notes.

Useful lspci commands

setpci for capability dumps

When you need raw register access:

# Dump all PCIe Express config space (cap)
$ sudo setpci -s 1a:00.0 CAP_EXP+0.l
0010d802

# Dump ACS control register specifically
$ sudo setpci -s c2:00.0 ECAP_ACS+0x06.w
000c

ECAP_ACS+0x06 is the ACS Control register. 0x000c would mean some bits are set. Match against the spec to read which.

Common P2P / GDR breaks via PCIe

Cross-socket GPU↔NIC pairing

$ cat /sys/bus/pci/devices/0000:1a:00.0/numa_node     # GPU 0
0
$ cat /sys/bus/pci/devices/0000:c4:00.0/numa_node     # NIC paired with GPU 0?
1

NIC is on NUMA 1, GPU is on NUMA 0. P2P doesn't work cross-socket on most platforms. Either re-cable to a NIC slot on the right socket, or accept the host-RAM bounce.

Hidden ACS on a fabric switch

$ sudo lspci -vvv -s c2:00.0 | grep ACSCtl
        ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-

ACS enabled on the upstream switch. Disable in BIOS or setpci. Without this, GDR is broken even though nvidia-peermem is loaded.

Slot wired narrow

$ sudo lspci -vvv -s 1a:00.0 | grep LnkSta
        LnkSta: Speed 32GT/s, Width x8 (downgraded)

GPU running x8 instead of x16. PCIe bandwidth is halved. Re-seat the card; check the slot in the manual is x16 electrical.

PCIe AER errors masking real issues

$ dmesg | grep -i "pcieport\|aer"
[12345.678] pcieport 0000:c0:01.0: AER: Corrected error received
[12345.678] pcieport 0000:c0:01.0:    [11] CorrIntErr

Corrected errors are usually fine (link survives), but sustained Corrected errors mean the link is degrading. Uncorrectable errors will crash the device.

BAR1 too small after reboot

$ nvidia-smi -q | grep -A 1 BAR1
    Total                 : 256 MiB

BIOS forgot to enable Resizable BAR after a power cycle. Some BIOSes silently revert this. Re-enable, or check if firmware update is needed.

Reference: layout of a typical 8-GPU H100 node

CPU 0 (NUMA 0)                          CPU 1 (NUMA 1)
+--Root Port--+                         +--Root Port--+
| PCIe SW 1   |                         | PCIe SW 3   |
|  +--GPU0    |                         |  +--GPU4    |
|  +--GPU1    |                         |  +--GPU5    |
|  +--NIC0    |                         |  +--NIC4    |
|  +--NIC1    |                         |  +--NIC5    |
| PCIe SW 2   |                         | PCIe SW 4   |
|  +--GPU2    |                         |  +--GPU6    |
|  +--GPU3    |                         |  +--GPU7    |
|  +--NIC2    |                         |  +--NIC6    |
|  +--NIC3    |                         |  +--NIC7    |
+-------------+    UPI / Infinity      +-------------+
                  Fabric (slow path)

Pairings:

• GPU0/1/2/3 ↔ NIC0/1/2/3 (NUMA 0, P2P over their PCIe switches)
• GPU4/5/6/7 ↔ NIC4/5/6/7 (NUMA 1)

Cross-NUMA paths (e.g., GPU0 → NIC4) cross UPI — no PCIe P2P, falls back to host bounce. Hence rail binding matters.

See also

• GPUDirect / GDR
• NVLink / NVSwitch
• NCCL multi-node tuning
• Network performance tuning
• GPU generations