ACS — PCIe Access Control Services on GPU nodes

PCIe Access Control Services is the single most common reason a brand-new multi-GPU box gets half the all-reduce bandwidth you paid for. The mechanism is well-defined in the PCIe spec and well-hidden in the BIOS, and it interacts badly with how NVIDIA GPUs do peer-to-peer over the root complex.

This page explains what ACS is doing, why it kills P2P, and the three operational ways to turn it off (with the security tradeoff each one carries).

What ACS actually is

ACS is a PCIe spec feature (since PCIe 2.0, ECN 2006) that lives in switches and root ports. It gives the platform fine-grained control over how upstream-to-downstream peer traffic is allowed to flow. It exists for two reasons:

1. IOMMU/VT-d isolation guarantees — without ACS-enforced upstream forwarding, two devices behind the same root port can DMA to each other and bypass the IOMMU's translation. A malicious or compromised PCIe device could read another device's memory. This breaks the security model of pass-through virtualization (VFIO/SR-IOV/PF assignment to VMs).
2. Multi-tenant isolation — same problem in cloud hypervisors: one tenant's vGPU/vNIC must not be able to DMA into another tenant's memory.

When ACS is enabled on the upstream port and switches above a GPU, every transaction from GPU0 → GPU1 is forced to traverse all the way up to the root complex, get translated by the IOMMU, then come back down to GPU1. That's a P2P operation turning into a 2× hop on the host fabric. Bandwidth collapses, latency doubles, and nvidia-smi topo -p2p shows the link as NS (not supported) or the bandwidth halves.

This is intended behavior. ACS is doing exactly what the spec says.

The bits that matter

lspci -vvv on an upstream port shows ACSCtl and ACSCap. Six controls matter for the P2P story:

ReqRedir+ and CmpltRedir+ are the killers. If lspci shows ACSCtl: ... RR+ CR+ ... on the switch port above your GPUs, P2P is broken on that path.

When to disable ACS, when to keep it

The tradeoff is concrete: ACS-off means a compromised device on the same root port can DMA to neighbor devices' memory. On a single-tenant GPU node where the only PCIe devices are the GPUs and NICs you trust, this is acceptable. On a multi-tenant host where one tenant has VFIO pass-through of one device, it is not.

Method 1 — pcie_acs_override boot param (preferred when available)

This is a kernel patch originally written by Alex Williamson for VFIO use cases (the irony — it was written to enable pass-through scenarios where the BIOS over-ACSs, but it's exactly what we need here). It is not in mainline. It ships with:

• Ubuntu HWE / Pop!_OS / Proxmox kernels (look for +pcie_acs_override patches)
• Several distro-built kernels for virtualization-heavy users
• Many enterprise GPU-vendor kernel images

If your kernel has it, edit /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="... pcie_acs_override=downstream,multifunction"

Then:

update-grub        # Debian/Ubuntu
# or
grub2-mkconfig -o /boot/grub2/grub.cfg   # RHEL/Rocky/Alma
reboot

Verify it took:

cat /proc/cmdline | tr ' ' '\n' | grep acs
# pcie_acs_override=downstream,multifunction

dmesg | grep -i 'ACS Override'
# [    0.024] PCI: ACS override active: downstream multifunction

downstream strips ACS on downstream ports; multifunction strips it on the function level. The combination covers all known GPU-server PCIe topologies.

If your kernel does not have the patch, the boot param is silently ignored. Confirm with dmesg — if you don't see ACS Override active, you need Method 2 or 3.

Method 2 — runtime setpci per device

This is what works on stock RHEL/Ubuntu kernels without the patch. It writes 0 directly to the ACS Control register on each switch port.

The ACS Control word lives at offset 0x6 of the ACS Extended Capability. Walk the topology and zero it on every PCIe switch port above your GPUs:

# Find every PCIe device that has an ACS capability
for bdf in $(lspci -d "::0604" | awk '{print $1}'); do
  # 0x0604 is "PCI Bridge"; switch ports show up here
  setpci -s "$bdf" ECAP_ACS+0x6.w=0000
  echo "ACS disabled on $bdf"
done

Validate:

sudo lspci -vvv -s 17:00.0 | grep -A2 ACSCtl
#   ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ ...
#   ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- ...

After: every Ctl bit should be -. Any + on ReqRedir, CmpltRedir, or UpstreamFwd means that switch port is still redirecting P2P traffic.

This change is lost on reboot because PCI config space is reinitialized.

Method 3 — the /etc/rc.local pattern (persisted runtime disable)

Because Method 2 doesn't survive reboot and Method 1 requires a patched kernel, the canonical operator workaround on stock kernels is to run the setpci loop at every boot. The classic location is /etc/rc.local, which most distros still honor via the rc-local.service systemd unit.

Create /usr/local/sbin/acs_disable.sh:

#!/usr/bin/env bash
# Disable PCIe ACS on every bridge that supports it.
# Required for GPUDirect P2P on this single-tenant GPU node.
set -euo pipefail

logger -t acs_disable "Disabling PCIe ACS on all bridges"

for bdf in $(lspci -d "::0604" | awk '{print $1}'); do
  # Skip if device has no ACS extended cap
  if ! sudo lspci -s "$bdf" -vvv 2>/dev/null | grep -q "Capabilities: \[.*\] Access Control Services"; then
    continue
  fi
  setpci -s "$bdf" ECAP_ACS+0x6.w=0000 || logger -t acs_disable "FAILED $bdf"
  logger -t acs_disable "ACS disabled $bdf"
done

chmod +x /usr/local/sbin/acs_disable.sh

Then /etc/rc.local:

#!/usr/bin/env bash
/usr/local/sbin/acs_disable.sh
exit 0

chmod +x /etc/rc.local
systemctl enable rc-local
systemctl start rc-local

Why rc.local and not a dedicated unit? Two reasons:

1. It runs after network-online.target and after every PCI device has been enumerated and probed by its driver, which is what you want — patching ACS earlier sometimes races the driver bind on big topologies (32+ GPUs).
2. It's the convention every other acs_disable.sh snippet on the internet uses, so a future operator searching for "acs_disable.sh" finds your script instantly.

There is nothing magical about rc.local; a plain systemd unit with After=multi-user.target works identically. Use whichever fits your config-management story.

Validation that P2P actually works after ACS-off

Three checks, in order:

# 1. ACS bits clear on every switch port
for bdf in $(lspci -d "::0604" | awk '{print $1}'); do
  ctl=$(sudo lspci -vvv -s "$bdf" 2>/dev/null | awk '/ACSCtl/{print; exit}')
  [[ -n "$ctl" ]] && echo "$bdf $ctl"
done | grep -E 'RR\+|CR\+|UF\+' && echo "STILL ENABLED somewhere" || echo "ACS clear"

# 2. nvidia-smi sees P2P
nvidia-smi topo -p2p r
# Look for OK on every off-diagonal entry

# 3. Real bandwidth test
/usr/local/cuda/extras/demo_suite/p2pBandwidthLatencyTest
# Unidirectional P2P=Enabled should show ~270 GB/s within an NVLink island
# and ~10-25 GB/s across PCIe (vs ~12 single-direction without P2P)

If topo -p2p shows NS after disabling ACS, the failure is no longer ACS — it's NVLink topology, IOMMU, or the GPU not being on the same root complex as its peer.

Common operational mistakes

• Disabling ACS but leaving iommu=on, intel_iommu=on without iommu=pt. Pass-through mode (iommu=pt) lets non-VFIO devices skip IOMMU translation entirely, which is what you want for P2P performance. Without it, you fixed ACS but the IOMMU itself still serializes traffic.
• Forgetting BIOS-level ACS. Some server BIOSes have an "ACS Control" or "PCIe ACS" toggle independent of the kernel. The Linux setpci write goes through, but the BIOS re-applies on warm-reset. Disable in BIOS too on those platforms.
• Trying to disable ACS on a virtual machine. Inside a VM, the PCIe topology is virtual; setpci either no-ops or errors. The fix has to be on the hypervisor, not the guest.

See also

• GRUB cmdline reference — where pcie_acs_override and iommu=pt go together
• NUMA pinning — the other reason your P2P bandwidth is bad
• Kernel sysctls for HPC — the broader tuning surface

External:

• Linux kernel Documentation/PCI/pcieaer-howto.rst and drivers/pci/pcie/aer.c for the in-tree ACS enforcement code path
• PCIe Base Specification §6.12 (Access Control Services)
• Alex Williamson's original pcie_acs_override patch series (2013, lkml)