Network performance tuning per speed tier: 100G / 200G / 400G / 800G

A 100 Gbps NIC and an 800 Gbps NIC are different machines. The kernel changes you make at 100G are mostly cosmetic — the defaults already saturate the wire. The kernel changes you make at 400G are necessary; without them you cap at 30-40% of line rate. At 800G, the kernel TCP path doesn't get there at all and you have to use RDMA or DPDK or io_uring offload.

This page walks through each tier in order — what the bottleneck is, what you tune, how to verify, and the diagnostic flow when "we paid for X Gbps and we're getting Y". It's the operator's quick reference; for the runtime tooling see networking/perf-tuning, and for the kernel cmdline pieces see kernel-tuning/grub.

The general shape: what you tune at every tier

Before per-tier specifics, the universal levers:

Each tier section below pins concrete numbers to these levers.

100 Gbps tier — ConnectX-5 / ConnectX-6 EDR/HDR

What's the bottleneck

At 100 Gbps, with reasonably modern hardware (Skylake / Cascade Lake / Rome), a single CPU core can mostly saturate the wire on TCP — provided the kernel TCP stack doesn't have to fight the defaults. This tier rarely needs heroic tuning; the most common problem is "we forgot to set jumbo frames".

PCIe gen3 x16 = 128 Gbps theoretical, ~100 Gbps practical. PCIe gen4 x16 = 256 Gbps, plenty.

Concrete tuning

# MTU 9000 — pre-test first per /networking/perf-tuning
sudo ip link set ens3np0 mtu 9000

# Ring buffer at NIC's max (CX-5/6 max is usually 8192)
sudo ethtool -G ens3np0 rx 4096 tx 4096

# Multi-queue — 32 queues is generally plenty at 100G
sudo ethtool -L ens3np0 combined 32

# RSS hashing on flows
sudo ethtool -K ens3np0 ntuple on
sudo ethtool -N ens3np0 rx-flow-hash udp4 sdfn

# Stop irqbalance, pin IRQs to NIC's NUMA cores
sudo systemctl disable --now irqbalance
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity.sh ens3np0

Sysctls — modest bumps over default:

# /etc/sysctl.d/10-network-100g.conf
net.core.rmem_max = 536870912            # 512 MiB max socket buffer
net.core.wmem_max = 536870912
net.core.rmem_default = 4194304          # 4 MiB default
net.core.wmem_default = 4194304
net.core.netdev_max_backlog = 100000
net.ipv4.tcp_rmem = 4096 87380 536870912
net.ipv4.tcp_wmem = 4096 65536 536870912

Apply: sudo sysctl --system.

Hugepages: usually not needed

100G fits in 4 KiB pages without painful TLB pressure. If you happen to have 2 MiB or 1 GiB pages reserved for other reasons (DPDK in some path, GPU memory), that's fine, but hugepages=0 works at 100G TCP.

RSS queues

ethtool -l shows the channels:

ethtool -l ens3np0
# Pre-set maximums:
# Combined:       63
# Current hardware settings:
# Combined:       16

# 32 is plenty
sudo ethtool -L ens3np0 combined 32

More queues isn't free — each queue costs an IRQ vector and a chunk of memory. Don't go to 63 just because you can.

Verification at 100G

# Multi-stream iperf3 — single stream might max a CPU before the NIC
iperf3 -s &
iperf3 -c <peer> -P 8 -t 30 -i 5
# 90+ Gbps aggregate is healthy

# RDMA with perftest (if running RoCE / IB on this NIC)
ib_send_bw -d mlx5_0 -F  &
ib_send_bw -d mlx5_0 -F <peer>
# BW: ~12 GB/s = ~96 Gbps

# GPUDirect RDMA
ib_send_bw -d mlx5_0 -F --use_cuda=0 <peer>
# BW: ~11 GB/s = ~88 Gbps

Healthy 100G numbers:

• iperf3 -P 8: 90+ Gbps
• ib_send_bw: 12 GB/s (96 Gbps)
• ib_send_bw --use_cuda: 11 GB/s (88 Gbps; small GDR overhead)

If you see less, work the diagnostic flow at the bottom of this page.

200 Gbps tier — ConnectX-7 NDR

What's the bottleneck

At 200G, a single core absolutely cannot saturate. The only way to hit line rate is to spread incoming traffic across multiple cores via RSS, multi-queue, and a deeper IRQ map. The kernel TCP stack can do 200G with multiple streams and good tuning; RDMA does it without breaking a sweat.

PCIe gen4 x16 = 256 Gbps; gen5 x16 = 512 Gbps. Either is fine. PCIe gen3 caps at 128 — won't hit 200G.

For RoCE, PFC + ECN are required at this rate. Without them, any congestion event collapses throughput because the kernel-driven TCP retry logic doesn't apply (RDMA's go-back-N hardware retry is much harsher than TCP fast retransmit).

Concrete tuning

sudo ip link set ens3np0 mtu 9000

# Bigger ring buffer
sudo ethtool -G ens3np0 rx 8192 tx 8192

# Multi-queue — half the local NUMA's cores per port
# Example: 64-core CPU, 2 NICs per NUMA = 32 cores per NIC, use 32 queues
sudo ethtool -L ens3np0 combined 32

# RSS hashing
sudo ethtool -K ens3np0 ntuple on
sudo ethtool -N ens3np0 rx-flow-hash udp4 sdfn

# IRQ pinning to NUMA-local cores
sudo systemctl disable --now irqbalance
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 0-31 ens3np0

# Adaptive coalescing
sudo ethtool -C ens3np0 adaptive-rx on adaptive-tx on

# CQE compression — more efficient memory use under high packet rate
sudo mlxconfig -d /dev/mst/mt4129_pciconf0 set CQE_COMPRESSION=1
# (requires reboot)

Sysctls:

# /etc/sysctl.d/10-network-200g.conf
net.core.rmem_max = 1073741824          # 1 GiB
net.core.wmem_max = 1073741824
net.core.rmem_default = 8388608         # 8 MiB
net.core.wmem_default = 8388608
net.core.netdev_max_backlog = 250000
net.core.optmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 1073741824
net.ipv4.tcp_wmem = 4096 65536 1073741824
net.ipv4.tcp_mem = 4096 87380 16777216

# BBR + fq for better fat-pipe long-RTT behavior
net.ipv4.tcp_congestion_control = bbr
net.core.default_qdisc = fq

Hugepages: 64x 1 GiB recommended

GRUB_CMDLINE_LINUX_DEFAULT="... default_hugepagesz=1G hugepagesz=1G hugepages=64 ..."

Why: at 200G, RDMA buffer pools (registered MRs) and DPDK packet pools (if used) benefit from 1 GiB pages. TLB pressure is real — without hugepages, page table walks for 200G of bounce buffers cost measurable cycles.

IRQ map per NUMA

This is where it gets concrete. Two NICs per NUMA, 32 cores per NUMA, 32 queues per NIC = 64 IRQs landing on 32 cores. Pin them so each core takes 2 IRQs from the local NUMA's NICs:

# NUMA 0 NICs: ens3np0, ens4np0; NUMA 0 cores: 0-31
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 0-31 ens3np0
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 0-31 ens4np0

# NUMA 1 NICs: ens5np0, ens6np0; NUMA 1 cores: 32-63
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 32-63 ens5np0
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 32-63 ens6np0

Verify with mpstat -P ALL 1 during a load test — interrupt load should be spread across the local NUMA cores, none on the wrong NUMA.

Real example: with vs without RSS spreading

This is a measurable, reproducible result on a 32-core Skylake CPU, ConnectX-7 200G, single-host TCP test:

The difference between "default" and "tuned" at 200G is more than 4x. At 100G, the same test would show maybe 1.2x — there's just more headroom at 100G to absorb suboptimal tuning.

RoCE specifics at 200G

# Trust DSCP for RoCE priority
sudo mlnx_qos -i ens3np0 --trust dscp

# PFC on TC3 (RoCE traffic class)
sudo mlnx_qos -i ens3np0 --pfc 0,0,0,1,0,0,0,0

# ECN on the fabric — congestion control
# (configured on the switch + via mlxconfig on the NIC)
sudo mlxconfig -d /dev/mst/mt4129_pciconf0 set ROCE_CC_PRIO_MASK_P1=0x10

# Verify
sudo mlnx_qos -i ens3np0

See networking/roce for the full PFC / ECN / DCQCN treatment.

Verification at 200G

# Multi-stream iperf3 — needs more streams to spread load
iperf3 -c <peer> -P 16 -t 30 -i 5
# 180+ Gbps healthy

# RDMA — hits closer to line rate
ib_send_bw -d mlx5_0 -F -s 65536 <peer>
# BW: 22-24 GB/s = 176-192 Gbps

# With CUDA (GDR)
ib_send_bw -d mlx5_0 -F -s 65536 --use_cuda=0 <peer>
# BW: 21-23 GB/s

Healthy 200G numbers:

• iperf3 -P 16: 180+ Gbps
• ib_send_bw: 22-24 GB/s (176-192 Gbps)
• nccl-tests inter-node 2 ranks per node: busbw 22-23 GB/s

400 Gbps tier — ConnectX-7 / ConnectX-8 NDR

What's the bottleneck

At 400G, the kernel TCP stack is the bottleneck. Even with perfect tuning, single-flow TCP caps at ~150 Gbps; aggregate multi-flow caps at ~250-300 Gbps. To get beyond, you need RDMA verbs (which bypass the kernel) or DPDK / AF_XDP / io_uring offload.

PCIe is also tight: gen5 x16 = 512 Gbps theoretical, ~400 Gbps practical. PCIe gen4 x16 = 256 Gbps and won't hit 400G. Gen5 x8 also caps at 256 Gbps. Make sure the NIC slot is gen5 x16 — verify with lspci -vvv | grep LnkSta.

Hugepages are mandatory. The TLB pressure of 4 KiB pages at 400G causes 30%+ throughput loss in measured benchmarks; 1 GiB pages eliminate it.

Concrete tuning

sudo ip link set ens3np0 mtu 9000

# Ring buffer maxed
sudo ethtool -G ens3np0 rx 16384 tx 16384

# Multi-queue — full NUMA-local cores
sudo ethtool -L ens3np0 combined 64

# RSS
sudo ethtool -K ens3np0 ntuple on
sudo ethtool -N ens3np0 rx-flow-hash udp4 sdfn

# IRQ pinning
sudo systemctl disable --now irqbalance
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 0-63 ens3np0

# CQE compression mandatory
sudo mlxconfig -d /dev/mst/mt4129_pciconf0 set CQE_COMPRESSION=1

# Adaptive coalescing
sudo ethtool -C ens3np0 adaptive-rx on adaptive-tx on

Sysctls — large buffers:

# /etc/sysctl.d/10-network-400g.conf
net.core.rmem_max = 2147483647          # 2 GiB - 1 (max int32)
net.core.wmem_max = 2147483647
net.core.rmem_default = 16777216        # 16 MiB
net.core.wmem_default = 16777216
net.core.netdev_max_backlog = 500000
net.core.optmem_max = 134217728
net.ipv4.tcp_rmem = 4096 131072 2147483647
net.ipv4.tcp_wmem = 4096 131072 2147483647
net.ipv4.tcp_mem = 4096 131072 33554432

# BBR + fq mandatory
net.ipv4.tcp_congestion_control = bbr
net.core.default_qdisc = fq

# Bigger conntrack table if firewall is in the path
net.netfilter.nf_conntrack_max = 2097152

GRUB cmdline contributions

... default_hugepagesz=1G hugepagesz=1G hugepages=128 ...
... pci=realloc=on,assign-busses ...
... isolcpus=4-55,60-111 nohz_full=4-55,60-111 rcu_nocbs=4-55,60-111 ...

pci=realloc lets the kernel rebar PCIe windows for the NIC's BAR2; some CX-7/8 cards expose a 64 GiB BAR2 that needs the firmware to allow resizing. Without it, mlx5 logs:

mlx5_core: Failed to map BAR ... insufficient resources

isolcpus/nohz_full matter at 400G because the polling threads can't tolerate jitter — a 100 µs scheduler interruption becomes visible as a hiccup in throughput.

PCIe verification

# Check the NIC's link state
sudo lspci -vvv -s 1b:00.0 | grep -E 'LnkSta|LnkCap'
# LnkCap: ... Speed 32GT/s, Width x16, ASPM L1, ...
# LnkSta: Speed 32GT/s, Width x16   <-- gen5 = 32 GT/s, x16 ✓

Speed 32GT/s, Width x16 = gen5 x16. If you see 16GT/s (gen4) or Width x8, the NIC is link-trained at lower than expected. Causes: bad cable, bad slot, firmware needs update, or motherboard doesn't support gen5 in that slot.

For 400G: must be gen5 x16, period.

RoCE at 400G — PFC must be perfect

At 400G, one bad TC mapping = head-of-line blocking everywhere. PFC config has to agree on:

• NIC: which DSCP value RoCE uses (mlnx_qos --trust dscp)
• NIC: which PFC priority is enabled (--pfc 0,0,0,1,0,0,0,0 for prio 3)
• Switch: PFC enabled on the matching priority on the matching port
• Switch: buffers / Xoff / Xon thresholds set for the priority

A single switch port with PFC mismatched will pause the upstream and HOL-block all RoCE traffic flowing through that port. Symptom: NCCL allreduce times jitter wildly, with one rank consistently slow. Diagnose by checking ethtool -S PFC counters per port.

ethtool -S ens3np0 | grep -i pfc
# rx_prio3_pause_count: 12345
# tx_prio3_pause_count: 678
# rx_prio3_pause_duration: 1234567   <-- microseconds spent paused

Some pause is normal (it's congestion control working). Massive growth = something's wrong.

L3 cache pressure at 400G

At 400G, the per-packet processing budget (even in RDMA, where it's hardware) interacts with L3 cache. Working sets that fit in L3 stay fast; ones that spill to memory eat memory bandwidth. Measure with mlc (Memory Latency Checker) and perf stat -e LLC-loads,LLC-load-misses.

Practical implication: smaller batch sizes for inference; smaller ring buffer per queue if you can afford the burst headroom; carefully sized RDMA MR pools.

Verification at 400G

# iperf3 multi-stream — even with everything tuned, kernel TCP caps somewhere
iperf3 -c <peer> -P 32 -t 60 -i 5
# 350+ Gbps with everything tuned; 250-300 Gbps with default sysctls

# RDMA — closer to line rate
ib_send_bw -d mlx5_0 -F -s 65536 -t 128 <peer>
# BW: 46-48 GB/s = 368-384 Gbps

# With CUDA (GDR)
ib_send_bw -d mlx5_0 -F -s 65536 --use_cuda=0 <peer>
# BW: 45-47 GB/s

# nccl-tests, 2 nodes, 8 GPUs each, 16 ranks total
mpirun -np 16 --hostfile hosts \
  -x NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  -x NCCL_DEBUG=WARN \
  ./build/all_reduce_perf -b 8G -e 8G -f 2 -n 5
# busbw should be 45-47 GB/s per rank pair (the 8 NICs aggregating)

Healthy 400G numbers:

• iperf3 -P 32: 350+ Gbps
• ib_send_bw: 46-48 GB/s (368-384 Gbps)
• nccl-tests busbw: 45-47 GB/s per rank pair

800 Gbps tier — ConnectX-8 XDR

What's the bottleneck

Mostly forward-looking as of 2026. Few production deployments. The realities:

• PCIe gen6 x16 = 1024 Gbps theoretical, ~800 Gbps practical
• Or gen5 x32 (rare slot) or two gen5 x16 cards bonded into one logical interface
• Kernel TCP at 800G is essentially impossible; only RDMA, DPDK with DPU offload, or NVLink-class internal fabric (B200 + NVL72)
• DPU offload becomes normal for non-trivial protocol stacks (TCP termination, encryption, etc.) — the host CPU can't keep up
• Same patterns as 400G but more aggressive: bigger hugepages, deeper isolation, larger ring buffers

Concrete tuning (working hypothesis)

sudo ip link set ens3np0 mtu 9000

# Max ring
sudo ethtool -G ens3np0 rx 32768 tx 32768

# Multi-queue
sudo ethtool -L ens3np0 combined 128

# RSS
sudo ethtool -K ens3np0 ntuple on
sudo ethtool -N ens3np0 rx-flow-hash udp4 sdfn

# IRQ pinning to a much wider housekeeping range
sudo /opt/mellanox/mlnx-ofed/sbin/set_irq_affinity_cpulist.sh 0-127 ens3np0

Sysctls — at 800G, sysctls increasingly become "doesn't matter, you're not using kernel TCP":

# /etc/sysctl.d/10-network-800g.conf
# (mostly only matters for control plane)
net.core.rmem_max = 2147483647
net.core.wmem_max = 2147483647
net.core.netdev_max_backlog = 1000000

GRUB:

... default_hugepagesz=1G hugepagesz=1G hugepages=256 ...
... pci=realloc=on,assign-busses pcie=pcie_bus_perf ...

pcie=pcie_bus_perf maximizes PCIe Max Payload Size and Max Read Request across the topology — at 800G, even MaxPayloadSize=256 (vs 512) costs measurable bandwidth.

Verification at 800G

# RDMA — multi-QP saturation
ib_send_bw -d mlx5_0 -F -s 65536 -q 16 -t 256 <peer>
# BW: 90+ GB/s = 720+ Gbps

# Multi-stream RDMA — multiple flows
for q in mlx5_0 mlx5_1; do
  ib_send_bw -d $q -F -s 65536 -q 16 <peer> &
done
# Aggregate: ~90-95 GB/s per port

Healthy 800G numbers (target — your mileage varies depending on exact firmware and switch):

• ib_send_bw -q 16: 90+ GB/s (720+ Gbps)
• ib_send_bw aggregate across multiple QPs: 95+ GB/s

"Why are we only getting X Gbps when we paid for Y" — diagnostic flow

The most common operator question. The answer is layered; work the layers in order.

Step 1: Single-stream limit (CPU)

# Single-stream iperf3
iperf3 -c <peer> -t 30
# 30 Gbps for example, on a 200G NIC

A single TCP stream is bounded by the receiving CPU's ability to drain the socket buffer. For 200G+, single-stream rarely exceeds 50-100 Gbps. Re-test with multi-stream:

iperf3 -c <peer> -P 16 -t 30
# 180 Gbps     <-- now you're hitting line rate

If multi-stream gets there, you don't have a problem — you have a single-flow workload. Move on.

Step 2: NIC actually negotiated correct rate?

ethtool ens3np0 | grep -E '^\s*Speed:'
# Speed: 200000Mb/s        <-- 200 Gbps ✓

# More detail (Mellanox)
sudo mlxlink -d /dev/mst/mt4129_pciconf0 -p 1
# Speed             : NDR 200G [4x]      <-- correct
# Eye opening info  : ...

# IB devices
ibstat
# Rate: 200       <-- IB rate in Gbps

If the rate is wrong (200G NIC running at 100G), suspect:

• Cable rated for lower speed (DAC vs AOC vs fiber)
• Switch port set to lower speed
• Firmware config wrong (mlxconfig -d /dev/mst/... query)
• HCA actually a different SKU than expected

Step 3: PCIe link width / speed

sudo lspci -vvv -s <bdf> | grep -E 'LnkSta|LnkCap'
# LnkCap: ... Speed 32GT/s, Width x16
# LnkSta: Speed 32GT/s, Width x16        <-- ok for 400G

# Or with shorter pretty output
sudo lspci -s <bdf> -vv | grep -E 'LnkCap|LnkSta'

If LnkSta shows lower than LnkCap — link trained at degraded speed/width:

Reseat the card. Reseat the riser. Check BIOS for slot config. Check dmesg | grep -i 'pci.*degraded\|link.*retrain'.

Step 4: Switch port speed

For ethernet:

# From the NIC, peer's port speed should match
ethtool ens3np0 | grep -E '^\s*Link partner advertised'

# Or check switch CLI directly (vendor-specific)

For InfiniBand:

# State and rate
ibportstate -P 1 query
# State: 4: Active
# Physical State: 5: LinkUp
# Active Speed: 8: NDR
# Active Width: 4

If switch port is forced to a lower rate (autoneg disabled, manually configured to 100G on a 200G port), that's the bottleneck.

Step 5: PFC drops or RoCE collapse

ethtool -S ens3np0 | grep -i 'drop\|pause\|pfc'
# rx_drop_packets: 0          <-- ok
# tx_pause_count: 12345       <-- some pauses, normal
# rx_prio3_pause_count: 12345 <-- PFC RX pauses on RoCE prio
# rx_prio3_pause_duration: 234567

# Growth over time
ethtool -S ens3np0 | grep prio3_pause_duration
sleep 60
ethtool -S ens3np0 | grep prio3_pause_duration
# (subtract: how many us paused over that minute)

A handful of pauses is normal flow control. Continuous, growing pause durations = backpressure into the switch, head-of-line blocking, RoCE thrashing.

Step 6: RSS not spreading flows

# During load test, watch per-CPU interrupt rate
mpstat -P ALL 1
# 04:32:01     all   12.34   ...
# 04:32:01       0   95.45   ... <-- only CPU 0 doing work
# 04:32:01       1    0.12   ... <-- everyone else idle

If interrupts are landing on one core only, RSS isn't spreading. Causes:

• ethtool -L combined 1 (single queue) — fix: combined N
• RSS hash function not including UDP src port for RoCE — fix: ethtool -N ... rx-flow-hash udp4 sdfn
• Single-flow workload with same 5-tuple — RSS can't spread by definition; need multiple flows

# Per-CPU softirq rate
cat /proc/softirqs | head -1; cat /proc/softirqs | grep ^NET_RX
# CPU0 should not be 100x the others

Step 7: Memory bandwidth bottlenecked

# mlc — Intel Memory Latency Checker
sudo /opt/mlc/Linux/mlc --idle_latency
# Local memory access: 80 ns
# Remote memory access (NUMA): 130 ns

sudo /opt/mlc/Linux/mlc --max_bandwidth
# Memory BW (local): 250 GB/s
# Memory BW (remote): 80 GB/s

# uncore PMU counters via perf
sudo perf stat -e uncore_imc_0/cas_count_read/,uncore_imc_0/cas_count_write/ sleep 10

At 400G+ with 8 NICs, aggregate memory bandwidth requirements approach the platform's IMC limit (~400-500 GB/s on a high-end Intel; ~700 GB/s on Genoa/Bergamo). If memory is saturated, CPU is busy waiting on DRAM and packet processing slows.

The fix: NUMA pinning per NIC + per GPU; reduce cross-NUMA traffic; consider CPU model with more memory channels.

Layered diagnostic sequence

1. Multi-stream (verifies CPU isn't single-flow bound)        — cheap
2. Negotiated rate (NIC, IB)                                  — cheap
3. PCIe link state                                            — cheap
4. Switch port state                                          — cheap (if you have switch CLI)
5. PFC counter growth                                         — cheap
6. mpstat during load (RSS spread)                            — moderate
7. mlc + uncore PMU (memory bandwidth)                        — long, requires tooling

In 90% of cases, the answer is in steps 1-5. If you've worked through 1-7 and still don't know, the problem is application-side (e.g., NCCL plugin not matching the topology) or fabric-side (e.g., congestion in the switch core), not host-side.

Per-tier verification expected throughputs (table)

For NCCL multi-node tests, expected busbw scales with number of NICs. For 8 GPUs / 8 NICs at 400G NDR, expect 45-47 GB/s aggregate per rank pair on 2 nodes; 40-44 GB/s typical for full-cluster collectives once tree latency adds up.

See also

• Networking: perf tuning — runtime tooling: ethtool, IRQ pinning, sysctls
• Networking: RoCE — PFC, ECN, DCQCN deep dive
• Networking: PCIe topology — root complex, ACS, NUMA-NIC binding
• Drivers: NVIDIA — GDR enablement
• Drivers: OFED — kernel-version matrix, persistent module loading
• GRUB cmdline — boot params per profile
• Kernel versions — which kernels for which features
• Jitter isolation — isolcpus/nohz_full/rcu_nocbs
• Sysctl tuning — full sysctl reference
• Operations: network validation by speed — runbook for new-node delivery

External:

• NVIDIA Mellanox Performance Tuning Guide for ConnectX-6/7
• Mellanox PRM (Programming Reference Manual) for ConnectX series
• Intel Memory Latency Checker — software.intel.com/mlc
• LWN: "io_uring at high throughput" series
• DPDK programmer's guide — for AF_PACKET / DPDK alternatives