Sizing NICs to GPU count — bandwidth math, rail mapping, oversubscription

Choosing the wrong NIC-to-GPU ratio is one of the most expensive mistakes you can make in cluster design — second only to topology errors, and sometimes caused by the same misunderstanding. Underprovision the fabric and every allreduce in every training job pays a tax for years. Overprovision and you've bought NICs that will never saturate. This page is the sizing reference.

The formula

Start with the per-GPU peak bandwidth requirement:

Required NIC BW per GPU = (per-GPU peak GDR throughput)
                        × (fraction of time the GPU spends on collectives)
                        × (overhead factor for protocol + retransmit headroom)

Aggregate NIC BW for cluster = (per-GPU NIC BW) × (total GPU count)

For pretraining large language models, the dominant collective is allreduce (implemented as reduce-scatter + allgather by NCCL). Typical duty cycle breakdown:

These fractions are operator rules of thumb calibrated on NVIDIA Nsight Systems traces — they vary significantly with model size, sequence length, and batch size. The safe design assumption: size for the worst case (40% comm fraction in pretraining) and accept that inference workloads will be over-provisioned on the NIC.

Peak GDR throughput by NIC

For sizing: use 48 GB/s per ConnectX-7 port as the sustained design number.

Overhead factor

NCCL adds per-message headers, chunk synchronization, and ring-algorithm overhead. A conservative overhead factor of 1.25 (25% overhead) is an operator rule of thumb for sizing. This accounts for:

• Protocol headers and ACKs in RC (Reliable Connected) mode
• NCCL ring algorithm chunk padding
• Occasional retransmit headroom on a well-tuned fabric

Example: 8-GPU H100 server
  Per-GPU target: 48 GB/s (full NDR line rate)
  8 GPUs × 48 GB/s × 1.25 = 480 GB/s aggregate NIC bandwidth
  = 8 × ConnectX-7 NDR single-port NICs (8 × 50 GB/s = 400 GB/s physical)
  → Rail-aligned: 1 CX-7 per GPU, each on its own leaf port
  → Each GPU gets ~48 GB/s sustained = essentially line rate

Per-platform recommendations

H100 SXM5 (8-GPU server)

Recommendation: 8× ConnectX-7 NDR single-port (400 Gb/s each), one per GPU, rail-aligned.

The H100 SXM5 server (DGX H100, HGX H100) has 8 GPUs connected via NVLink 4 / NVSwitch within the node. Intra-node all-reduce uses NVLink at ~3.2 TB/s aggregate — entirely on-board, no NIC involved. Inter-node collectives (allreduce across servers) use the NICs.

PCIe layout on a DGX H100: two Intel/AMD root complexes, 4 GPUs and 4 NICs per root complex. Each NIC is PCIe Gen5 x16 (128 GB/s PCIe BW), comfortably above the NIC's 50 GB/s sustained RDMA BW. NUMA affinity: NICs 0–3 to NUMA node 0, NICs 4–7 to NUMA node 1.

The rule: NIC-0 connects to leaf-rail-0 and is NUMA-local to GPU-0. NIC-7 connects to leaf-rail-7 and is NUMA-local to GPU-7. NCCL's topology file (--topoFile) for DGX H100 encodes this affinity.

H200 SXM5 (8-GPU server)

Recommendation: 8× ConnectX-7 NDR — same as H100 SXM5. Consider CX-8 XDR if available and budget allows.

H200 SXM5 is mechanically similar to H100 SXM5. GPU compute is faster (HBM3e memory); the NIC requirement is the same. XDR NICs would provision 2× the bandwidth per GPU — worthwhile if your allreduce is the bottleneck (i.e., you've profiled and confirmed >35% of step time in comms). For most shops: NDR is sufficient for H200 and gives consistency with an existing NDR fabric.

B200 NVL72

Recommendation: per-rack CX-8 XDR 800G NICs — one per GPU is the target, with NVLink handling all intra-rack traffic.

B200 NVL72 is a rack-scale design: 72 B200 GPUs connected by 5th-generation NVLink within the rack. Intra-rack all-reduce runs over NVLink at multi-TB/s rates. Inter-rack traffic uses the external fabric.

The external NIC layout is OEM-specific (NVIDIA GB200 NVL72 reference design uses 8 CX-8 XDR NICs per rack for external connectivity, roughly 1 NIC per 9 GPUs). This is a notable departure from the H100 1:1 ratio — the higher NVLink aggregate within the rack means fewer external NICs are needed to maintain compute/comms balance for most training configurations.

For B200 NVL72: verify against the actual NVIDIA reference architecture for your system firmware version. The NIC placement and NUMA topology differ from HGX-style servers.

L40S inference servers (8-GPU PCIe)

Recommendation: 1–2× ConnectX-7 NDR 200 Gb/s (dual-port) per server, depending on workload.

L40S is a PCIe Gen4 GPU with no SXM / NVLink option. It is primarily used for inference, not large-scale pretraining. Inference collectives are lightweight:

• Tensor-parallel allgather: small per-token activations
• KV-cache transfer (disaggregated prefill/decode): bursty, not continuous

A single dual-port CX-7 200G (2× 200G) per server gives 400 Gb/s aggregate — sufficient for most inference serving configurations. Add a second NIC only if you observe NIC saturation in production profiling (use ibstatus per-port tx_discards counter or NCCL's bandwidth output during tensor-parallel tests).

Because L40S servers are PCIe-only and don't support GDR via NVLink fabric, ensure the NIC slots are PCIe Gen4 x16 minimum. ACS (Access Control Services) implications apply — see below.

AMD MI300X (8-GPU server)

Recommendation: 8× ConnectX-7 NDR single-port (same physical layout as H100 SXM5).

MI300X uses the ROCm communication library (RCCL, a NCCL-compatible implementation). RCCL supports the same IB transport via the ROCm rdma-core library. The per-GPU NIC recommendation is identical to H100: 1 CX-7 per GPU, rail-aligned.

Key difference: RCCL's ring algorithm and collective scheduling differ from NCCL in some micro-benchmarks. The NIC sizing is the same but you should profile RCCL bandwidth (rccl-bwtest) rather than assuming NCCL results translate directly. Per-GPU sustained RCCL bandwidth on MI300X with CX-7 NDR is typically 85–95% of the NCCL result on H100 (varies by RCCL version and ROCm release).

PCIe lane budget

Each ConnectX-7 requires PCIe Gen5 x16 for full 400G throughput to host memory. The math:

• PCIe Gen5 x16: 128 GB/s (theoretical), ~120 GB/s practical
• CX-7 NDR peak: 50 GB/s in one direction
• Ratio: 120 / 50 = 2.4× headroom — adequate, PCIe is not the bottleneck at NDR

At XDR (CX-8, 100 GB/s per NIC):

• PCIe Gen5 x16 theoretical: 128 GB/s
• CX-8 XDR peak: 100 GB/s
• Ratio: ~1.28× — tight. Ensure no PCIe bandwidth sharing (no switch behind the NIC slot).

Verify the platform's root complex layout before placing NICs. In a DGX-class server this is documented by NVIDIA. In OEM server designs, use lspci -t to map NIC slots to root complexes:

# Show PCIe tree — identify root complex (domain:bus) for each NIC
$ lspci -t | grep -A1 "Root Port"

# For a specific NIC (PCI address 41:00.0):
$ cat /sys/bus/pci/devices/0000:41:00.0/numa_node   # NUMA affinity
$ lspci -vvv -s 41:00.0 | grep -E "LnkSta|Speed|Width"
# Expect: Speed 32GT/s (Gen5), Width x16

A NIC at PCIe Gen4 x16 (64 GB/s) instead of Gen5 x16 (128 GB/s) will not bottleneck NDR traffic (64 GB/s > 50 GB/s NIC peak). But Gen4 x8 (32 GB/s) is a real bottleneck and causes measurable GDR bandwidth loss. Check every slot.

ACS and IOMMU implications

ACS (Access Control Services) is a PCIe feature that forces all peer-to-peer DMA traffic (including GDR) to pass through the CPU root complex. When ACS is enabled on a PCIe switch between a GPU and a NIC, GDR cannot work — every RDMA write has to bounce through host memory.

Without ACS (P2P enabled):
  GPU ──DMA──► NIC ──RDMA──► Remote NIC ──DMA──► Remote GPU
  (direct, zero-copy, full GDR speed)

With ACS enabled on PCIe switch:
  GPU ──DMA──► PCIe switch ──► Root complex ──► PCIe switch ──► NIC ──► ...
  (extra hop, 2× PCIe latency, half the bandwidth in practice)

To enable GDR:

1. Disable ACS on the PCIe switch between the GPU and NIC: setpci -s <pci_dev> ECAP_ACS+0x6.w=0000 (requires careful identification of the correct bridge device).
2. Alternatively, set IOMMU_PLATFORM=false in the IOMMU config (weaker, platform-dependent).
3. On Linux with the NVIDIA driver: ensure nvidia-peermem module is loaded — it registers the GPU memory with the IB stack so the CX-7 can DMA to/from it.

For production: the NVIDIA DOCA or ConnectX documentation specifies which ACS settings to change for each server model. For DGX H100, NVIDIA's BIOS profile disables ACS appropriately by default. On OEM servers, validate explicitly.

When ACS is unavoidable

Some hypervisor environments and security-hardened deployments require ACS. In those cases:

• GDR throughput drops by approximately 30–50% vs. P2P enabled (operator rule of thumb — actual depends on message size and traffic pattern).
• For inference or fine-tuning, this may be acceptable.
• For large-scale pretraining, disable ACS or use bare metal.

Storage NIC placement

A question that comes up in every cluster design: should the storage HCA (for Weka, Lustre, or NFS-over-RDMA) share a compute rail or use a dedicated rail?

Recommendation: dedicated storage rail, do not share with compute.

Why:

• Storage traffic patterns differ from NCCL: large sequential reads (training data) and large sequential writes (checkpoints) that are bursty at job boundaries.
• Sharing a rail means storage bursts at checkpoint time can saturate a leaf switch that NCCL allreduce traffic also depends on.
• During checkpoint (which in large pretraining happens every few hundred steps), all nodes write simultaneously. This is a different bisection stress than allreduce and can degrade allreduce performance if they share leaf uplinks.

A typical layout for a 64-GPU cluster:

• Rail-0 through rail-7: NCCL compute NICs (8 per server)
• Rail-8 (or a separate 9th NIC): storage HCA, connected to a separate storage leaf

This requires a 9th PCIe slot. In DGX H100 servers this is available. In some OEM HGX designs only 8 NIC slots exist — in that case, evaluate whether a shared rail is acceptable by profiling checkpoint-time storage bandwidth vs. allreduce throughput in a test job.

Worked examples

Example 1: 64-GPU H100 cluster (16× DGX H100 nodes, NDR rail-aligned)

Target: 64 GPUs, 8-GPU H100 servers, NDR 400G, non-blocking
NICs needed: 64 GPUs × 1 CX-7 per GPU = 64 NICs
Rails: 8 rails
Leaf switches: 8 (one per rail)
Spine switches: 8 (for non-blocking 1:1)
Ports used per leaf: 16 downlinks (16 servers, one NIC per server per rail)
                   + 8 uplinks to 8 spines = 24 ports → fits 64-port Quantum-2
Total switches: 16 (8 leaf + 8 spine)

Aggregate NIC bandwidth:
  64 GPUs × 48 GB/s per CX-7 = 3,072 GB/s = ~24.6 Tb/s aggregate
  Non-blocking bisection = 24.6 Tb/s (8 rails × 8 leafs × 8 uplinks × 400G)

PCIe check:
  H100 SXM5 DGX: PCIe Gen5 x16 per NIC slot — ✓ no bottleneck
ACS:
  DGX H100: ACS disabled by NVIDIA BIOS profile — ✓ GDR enabled

Example 2: 256-GPU H200 cluster, target XDR

Target: 256 GPUs, 8-GPU H200 servers = 32 servers, XDR 800G
NICs: 256 × CX-8 XDR = 256 NICs
Rails: 8 rails
Servers per leaf: 32 / 8 = 4 servers (wait — need more leafs)

Let's assume 8 leafs per rail × 8 rails = 64 leafs total
Each leaf: 4 servers × 1 CX-8 = 4 downlinks + uplinks
Spine: 8 per rail = 64 spines total (or use fewer with XDR's higher port density)

Revised with Q3200-RA (144-port XDR):
  Leaf: 32 downlinks to servers + 16 uplinks to spines = 48 ports per leaf
  Per rail: 8 leafs × 32 downlinks = 256 server connections per rail
  BUT we have 32 servers, so 4 per leaf is fine
  
  Per rail: 4 leafs × 32 downlinks = 128 server-port downlinks (32 used)
  Spine: 4 per rail needed for 1:1
  Total: 8 rails × (4 leafs + 4 spines) = 64 switches

Aggregate NIC BW:
  256 GPUs × 95 GB/s per CX-8 = 24,320 GB/s ≈ 195 Tb/s aggregate
  Non-blocking bisection: 8 rails × 32 uplink-ports per rail × 800G = 204.8 Tb/s
  → Substantially non-blocking, comfortably above aggregate NIC demand

Example 3: 32-GPU L40S inference fleet, CX-7 200G sufficient

Target: 32 GPU L40S servers (4-GPU or 8-GPU PCIe), inference workloads
Collective pattern: allgather for tensor-parallel, typically TP=4 or TP=8 within server

4-GPU servers: intra-server NVLink for TP-4. Single CX-7 dual-port 200G per server.
  Per-server NIC BW: 2× 200G = 400G = 50 GB/s — sufficient for inference

8-GPU servers: TP-8 intra-server. Same reasoning.

Sizing check:
  Tensor-parallel allgather: 4B param model, FP16, TP-4
  Activation size per token: 4096 hidden × 4 bytes = 16 KB
  Allgather volume per token: 3 × 16 KB = 48 KB (TP-4, ring-allgather)
  At batch_size=64, seq_len=2048: 48 KB × 2048 × 64 = ~6.3 GB per step
  Step duration at 10 TFLOPS/GPU: 200 ms
  NIC BW needed: 6.3 GB / 0.2 s = 31.5 GB/s
  → Well within 2× CX-7 200G = 2× 25 GB/s = 50 GB/s (split across 2 ports)

1-2 CX-7 dual-port 200G per server is the right answer for L40S inference.

Example 4: 1024-GPU H100 pretraining, 3-tier fat-tree, NIC budget per leaf

Target: 1024 GPUs, 8-GPU H100 SXM5 = 128 servers
NICs: 1024 × CX-7 NDR = 1024 NICs
Rails: 8
Topology: 3-tier fat-tree

Per-rail allocation:
  128 servers, 1 NIC per server per rail = 128 NIC ports per rail
  
  Leaf switches per rail: 128 / 32 = 4 leafs (64-port, 32 downlinks)
  Spine switches per rail: 4 (for 1:1, matching 4 leafs × 32 uplinks)
  Super-spine: 4 per rail = 4 × 8 = 32 uplinks from spine group ÷ 8 ports each
             Actually: one set of 16 super-spines shared across all 8 rails

  Per-rail switch count: 4 leafs + 4 spines = 8 switches
  Total across 8 rails: 64 leaf + 64 spine = 128 switches
  Plus: 16 shared super-spines
  Grand total: ~144 Quantum-2 switches for 1024 non-blocking GPU endpoints

NIC budget per leaf:
  32 downlinks × 400G = 12.8 Tb/s edge BW per leaf
  32 uplinks × 400G = 12.8 Tb/s spine BW per leaf
  = 32 CX-7 NICs per leaf → confirmed, each GPU NIC gets a dedicated leaf port

Bisection BW check:
  1024 GPUs × 48 GB/s = 49,152 GB/s ≈ 393 Tb/s aggregate
  Non-blocking bisection at NDR: 
    8 rails × 4 leafs × 32 uplinks × 400G = 4,096 Gb/s × 8 = ~409.6 Tb/s
  → 1:1 non-blocking: ✓

Heuristic: back-solving from training target

To validate NIC sizing from a training throughput target:

Required NIC BW = (target_tokens_per_sec × bytes_per_token × overhead_factor)
                / (comm_fraction_of_step)

Where:
  target_tokens_per_sec  = your desired throughput (e.g., 1M tokens/s for a 70B model)
  bytes_per_token        = model_params × dtype_bytes / seq_len × 2 (for allreduce, 2× BW)
  overhead_factor        = 1.25 (operator rule of thumb, see above)
  comm_fraction          = 0.3 (typical for large pretraining, operator rule of thumb)

Example: 70B parameter LLaMA-scale pretraining
  Model params: 70 × 10^9 × 2 bytes (BF16) = 140 GB gradient per step
  allreduce BW (reduce-scatter + allgather, per link): 2 × 140 GB / step
  
  At 1000 GPU-clusters, per-GPU gradient = 140 GB / 1000 = 140 MB per GPU
  allreduce volume per step per GPU: 2 × 140 MB = 280 MB
  Step time at 30% comm: if comm = 0.3 × step, solve for step time
  
  NIC BW per GPU needed: 280 MB / (0.3 × step_time)
  At step_time = 0.5s: 280 MB / 0.15 s = 1.87 GB/s — NDR is massively overprovisioned
  At step_time = 0.05s (fast optimizer): 280 MB / 0.015 s = 18.7 GB/s — NDR (48 GB/s) still sufficient

The heuristic check: if required NIC BW per GPU < 40 GB/s, CX-7 NDR single-port is sufficient.
If ≥ 40 GB/s (very fast step time + large model), consider CX-8 XDR.

When to over-provision

Research clusters that swap workloads frequently (multiple teams, different model sizes, different collective patterns) benefit from over-provisioned NICs. Sizing for the worst-case workload in advance avoids topology changes mid-lifecycle.

Production training that runs the same pretraining job for months is predictable: profile once, size exactly, and don't over-buy. If a 70B pretraining job profiles at 18 GB/s NIC utilization per GPU, buying CX-8 XDR for 100 GB/s is 5.5× over-provisioning with no measurable job improvement.

The other consideration: future-proofing. If your cluster will run at NDR for 3 years, models will get larger (more parameters, more gradient communication). Build in 20–30% headroom against your current sizing math.

Diagnostics for under-provisioned fabrics

If you suspect NIC under-provisioning is limiting training throughput, these are the signals to look for:

# NCCL all-reduce bandwidth benchmark (run on all nodes before deploying training)
$ mpirun -np 8 --hostfile hostfile all_reduce_perf -b 4G -e 4G -f 2 -g 1
# Expected: >90% of theoretical NDR BW. If <70%, investigate.

# Per-port IB drop counters (check for congestion-induced drops)
$ perfquery -x <port>   # Look for: symbol_error, port_rcv_errors, port_xmit_discards

# Switch-side port counters via UFM or ibqueryerrors
$ ibqueryerrors -p   # Shows ports with non-zero error counters

# NIC Tx/Rx bandwidth utilization in real-time
$ watch -n1 'ethtool -S <nic> | grep -E "tx_bytes|rx_bytes"'
# Or for IB:
$ watch -n1 'cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data'
# Divide successive readings by the interval and by 4 (IB counts in dwords) to get GB/s

# NCCL debug output for per-ring bandwidth
$ NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=NET mpirun ./your_training_job 2>&1 | grep "connected"
# "connected" lines show per-channel BW; compare to expected line rate

If port_xmit_discards is non-zero on leaf uplinks: spine layer is saturated — topology issue, not NIC count. If NCCL all_reduce_perf shows <80% of line rate: may be NCCL algorithm, SHARP config, or ring topology mismatch. If ethtool -S shows Rx drops on the NIC: NIC ring buffer overflow — increase ring size with ethtool -G <nic> rx 8192, or the NIC is genuinely saturated.