Network performance tuning: jumbo frames, NIC queues, and RDMA tunables

When a GPU node is doing 8 × 400 Gb/s worth of RDMA traffic, the defaults on a fresh Ubuntu install aren't going to cut it. NIC ring buffers are too small, IRQs land on the wrong CPU, the kernel sysctls assume a 1 Gb world, and your jumbo frame setting is whatever the previous tenant left behind. This page is the operator's checklist for getting a host into a state where it can actually push line rate.

The rule running through everything below: never make persistent changes without pre-testing. A wrong MTU change can take a node off the network until physical access. Test temporarily first, validate, then persist.

MTU: jumbo frames

Default ethernet MTU is 1500 bytes. With 8x400G RoCE traffic, that means ~3.3 million frames per second per NIC just for payload — the NIC packet rate becomes a bottleneck before raw bandwidth does. Jumbo frames (MTU 9000) cut frame count by ~6x and dramatically reduce CPU/NIC overhead per byte.

For native InfiniBand the equivalent is IB MTU 4096, which is the maximum. Verify with ibv_devinfo | grep mtu.

Pre-test before persistent change

This is the universal rule. Before editing netplan or whatever's permanent, set the MTU temporarily and ping with don't-fragment + the test size:

# On both ends
$ sudo ip link set ens3np0 mtu 9000

# From source, ping with payload sized to fit MTU:
# 9000 (MTU) - 20 (IP header) - 8 (ICMP header) = 8972 bytes payload
$ ping -M do -s 8972 -c 5 10.0.10.2
PING 10.0.10.2 (10.0.10.2) 8972(9000) bytes of data.
8980 bytes from 10.0.10.2: icmp_seq=1 ttl=64 time=0.083 ms
8980 bytes from 10.0.10.2: icmp_seq=2 ttl=64 time=0.045 ms
...

If the ping succeeds, MTU 9000 is end-to-end (host + switch + host). Now persist via netplan / ifcfg / NetworkManager. If it fails — Frag needed and DF set or just timeouts — something on the path doesn't support 9000. Common culprits:

• Switch port MTU not bumped to 9216 (jumbo headroom).
• VLAN sub-interface inheriting old MTU.
• Some intermediate device hard-capped.

Don't persist until the temporary test passes. This rule applies in every environment, no exceptions — production, lab, sandbox. The cost of getting it wrong is taking nodes off the network.

For RoCE, the IP MTU and the RoCE pmtu must agree. RoCE pmtu is negotiated per-QP and typically hits the active port's IB-MTU value (4096 for IB, derived from IP MTU on RoCE).

NIC tuning with ethtool

Ring buffer sizes

Default is usually 1024-2048 descriptors per ring. For high bandwidth, bump to the max the NIC supports:

$ ethtool -g ens3np0
Ring parameters for ens3np0:
Pre-set maximums:
RX:             8192
TX:             8192
Pre-set Current:
RX:             1024
TX:             1024

$ sudo ethtool -G ens3np0 rx 8192 tx 8192

Bigger rings = more in-flight packets, less chance of NIC-side drop under burst. Costs a few MB of pinned memory per NIC.

Interrupt coalescing

Default ethtool -c settings target latency. For throughput on busy NICs you want the NIC to batch interrupts:

$ ethtool -c ens3np0 | grep -E "rx-usecs|tx-usecs"
rx-usecs:       8
tx-usecs:       8

$ sudo ethtool -C ens3np0 adaptive-rx on adaptive-tx on

Adaptive coalescing lets the NIC tune itself based on traffic intensity. For RoCE you generally don't care because the data path doesn't go through the kernel — coalescing only matters for socket fallback / IPoIB / control plane.

Multi-queue and RSS

$ ethtool -l ens3np0
Channel parameters for ens3np0:
Pre-set maximums:
Combined:       63
Current hardware settings:
Combined:       16

$ sudo ethtool -L ens3np0 combined 32

Combined is the number of RX/TX queue pairs. More queues = more CPUs can process traffic in parallel. For a node with 64 cores and 8 NICs, 16-32 queues per NIC is plenty. Don't max it out — each queue costs IRQ + memory.

Verify RSS is hashing flows across queues:

$ ethtool -x ens3np0
RX flow hash indirection table for ens3np0 with 32 RX ring(s):
    0:      0     1     2     3     4     5     6     7
    8:      8     9    10    11    12    13    14    15
   ...
RSS hash key:
9d:33:11:8a:...
RSS hash function:
    toeplitz: on
    xor: off
    crc32: off

For RoCE v2 you want hashing to include UDP src port (which the HCA varies per QP) so that flows distribute across queues. Set:

$ sudo ethtool -K ens3np0 ntuple on
$ sudo ethtool -N ens3np0 rx-flow-hash udp4 sdfn

sdfn = Source IP, Destination IP, source port (F = src), destination port (N = dst).

CPU affinity and IRQ pinning

By default, irqbalance distributes NIC IRQs across CPUs. For RDMA-heavy workloads, that's almost always wrong — you want IRQs pinned to specific CPUs near the NIC's NUMA node, and irqbalance randomly migrating them is bad for cache locality and bad for CPU isolation (cores running training shouldn't take NIC interrupts).

Disable irqbalance

$ sudo systemctl disable --now irqbalance

For RoCE: NCCL data path doesn't actually take CPU interrupts (the verbs poll path is in user space), so this matters less. But for any socket / TCP / IPoIB traffic on the same NIC, IRQ placement matters.

Pin IRQs with the Mellanox helper

Mellanox ships set_irq_affinity.sh:

$ sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity.sh ens3np0
$ sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 0-15 ens3np0

The first form auto-pins to the NIC's NUMA-local CPUs. The second pins to a specific CPU list — useful when you want to reserve cores 16-63 for training.

Verify NUMA-locality

$ cat /sys/class/net/ens3np0/device/numa_node
0

$ cat /proc/interrupts | grep mlx5_async
 192:        ...      mlx5_async8@pci:0000:1b:00.0
                                                   ^ which CPUs got the interrupts

Cross-NUMA NIC traffic costs ~30 ns per cache miss × millions of operations. Worth fixing.

DCQCN tuning on Mellanox

For RoCE, the congestion control loop runs on the NIC itself. Defaults are usually OK, but for tightly-tuned fabrics you can adjust:

# On the NIC, via mlxreg
$ sudo mlxreg -d /dev/mst/mt4129_pciconf0 --reg_name PPCC --get

The interesting fields:

• alpha_update_period (RP_DCE_TIME_RESET): how often the NIC reduces rate when receiving CNPs. Lower = more aggressive backoff.
• rate_increase: hyper-increase additive step when no CNPs arrive. Higher = recovers bandwidth faster after congestion clears.
• min_rate: floor below which rate won't drop. Default ~0.

For most clusters, leave defaults alone — they're tuned for typical RoCE deployments. Touch only if you can characterize "DCQCN is too slow to react" or "DCQCN backs off too aggressively and we never recover bandwidth."

sysctl tuning

# /etc/sysctl.d/99-gpu-network.conf

# Maximum socket buffer sizes (16 MB each)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# Default socket buffer sizes (4 MB)
net.core.rmem_default = 4194304
net.core.wmem_default = 4194304

# TCP min/default/max buffer sizes
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Backlog for incoming packets when CPU can't process fast enough
net.core.netdev_max_backlog = 250000

# Max number of incoming connection requests
net.core.somaxconn = 8192

# UDP buffers (matters for IPoIB UD)
net.ipv4.udp_mem = 102400 873800 16777216

# Disable TCP slow start after idle (helps long-lived bulk connections)
net.ipv4.tcp_slow_start_after_idle = 0

# IB-specific: disable kernel TCP timestamping that breaks IPoIB
# (only if you use IPoIB)
# net.ipv4.tcp_timestamps = 0

These mostly matter for the bootstrap / control plane, NCCL bootstrap, and any non-RDMA TCP traffic. RDMA itself uses NIC hardware queues, not socket buffers.

Apply:

$ sudo sysctl -p /etc/sysctl.d/99-gpu-network.conf

Memory locking limits

Verbs MR registration pins memory. Default RLIMIT_MEMLOCK is 64 KB which is laughable for any real RDMA workload. Set unlimited:

# /etc/security/limits.d/99-rdma.conf
*    soft    memlock    unlimited
*    hard    memlock    unlimited

For systemd services:

# /etc/systemd/system/<your-service>.service
[Service]
LimitMEMLOCK=infinity

For Kubernetes pods, the container runtime needs --ulimit memlock=-1 (Docker) or set per-pod via Pod Security context if you're on a runtime that respects it.

For containerd and the NVIDIA Container Runtime, there's usually no problem — the runtime inherits the host limit if you bumped it.

RDMA-specific module parameters

$ cat /sys/module/rdma_ucm/parameters/mode
shared

$ cat /sys/module/mlx5_core/parameters/prof_sel
2

rdma_ucm mode=shared is required for SR-IOV / multi-tenant RDMA. prof_sel=2 is the high-performance mlx5 profile. Both should be set already by MOFED defaults; verify.

Universal RDMA modules to load at boot

This list is the universally-required set of kernel modules for full RDMA functionality. Several have defaults missing on stock Ubuntu / RHEL where the kernel ships them but doesn't load on boot:

# /etc/modules-load.d/rdma.conf
ib_core
ib_uverbs
ib_umad
ib_cm
rdma_cm
rdma_ucm
mlx5_core
mlx5_ib
nvidia-peermem

Without ib_umad: management datagrams don't work — sminfo, ibstat, etc. fail. Without rdma_ucm: RDMA-CM (used by some apps for connection establishment) fails. Without nvidia-peermem: GDR doesn't engage.

Confirm loaded:

$ lsmod | grep -E 'ib_|rdma|mlx5|peermem'
nvidia_peermem         16384  0
mlx5_ib               524288  0
ib_uverbs             196608  6 mlx5_ib
mlx5_core            2502656  1 mlx5_ib
ib_core               598016  9 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,...
ib_umad                40960  0
rdma_ucm               40960  0

This module set has been validated as the right answer for RDMA functionality on multiple production GPU clusters — adopt as a default hardening baseline.

Verify with iperf3 multi-stream

For non-RDMA paths (control plane, IPoIB), the standard test is iperf3 multi-stream. Single stream maxes out a CPU core long before it maxes the NIC; multi-stream is the right test.

server$ iperf3 -s -p 5201
client$ iperf3 -c server-host -p 5201 -P 16 -t 60 -i 5

-P 16 opens 16 parallel TCP streams. Aggregate bandwidth is what you read. For a 100 GbE NIC you should see ~95 Gbps; for 400 GbE ~380 Gbps with sufficient streams.

If single-NIC TCP only does ~30 Gbps no matter how many streams — your sysctls are too low, or RSS isn't distributing flows across queues.

For RDMA: use ib_write_bw not iperf3. iperf3 measures the kernel TCP stack which is irrelevant to RDMA performance.

Quick reference: the "checklist" for a new GPU node

Common "I tuned everything and still slow" causes

1. Single GPU not paired with single NIC — rail binding broken, traffic crossing NUMA. nvidia-smi topo -m.
2. GDR not engaged — cat /proc/driver/nvidia-peermem/clients empty. See GPUDirect doc.
3. PFC pause storms — ethtool -S | grep -i pause shows constant growth. See RoCE doc.
4. One slow node poisoning the job — find the slow rank with profiling and check it specifically.
5. Mismatched MTU somewhere on the path — ping -M do -s 8972 from every node to every node.
6. Host CPU saturated by something else — DCGM exporter, monitoring agents, log shippers. top should show NCCL using <10% CPU when GDR is working.

See also

• RDMA fundamentals
• RoCE v2 + DCB/PFC/ECN
• NCCL multi-node tuning
• GPUDirect / GDR
• PCIe topology + ACS