RDMA fundamentals: how zero-copy networking actually works

If you came up through Linux sockets you have a strong mental model for networking: send(2) copies your buffer into the kernel, the kernel hands it to the NIC, on the other side the NIC interrupts the CPU, the CPU copies it into the receiver's buffer. That works fine at 10 Gbps. At 400 Gbps with 8 NICs per node, it doesn't — you'd burn an entire CPU socket just doing memcpy and softirq, and you'd still be slower than the wire.

RDMA (Remote Direct Memory Access) is the alternative. The application registers memory once, hands a "remote address + key" to the peer, and the peer's NIC reads or writes that memory directly over the network with no host CPU involvement on either side. This page is a working operator's primer — what RDMA is, what the verbs API actually exposes, and how NCCL uses it.

Why RDMA exists

Think of a multi-node training job moving 256 MB allreduce buffers between 32 GPUs every iteration. With sockets:

1. GPU DMAs buffer to host RAM (one copy).
2. App calls send(), kernel copies buffer into socket send queue (two copies).
3. NIC DMAs from kernel buffer onto wire.
4. On the receiver, NIC DMAs into kernel buffer.
5. App calls recv(), kernel copies into user buffer (three copies).
6. App copies into pinned host RAM, then DMAs to GPU (four copies).

Per buffer, you've done four memcpys at host RAM bandwidth (~50-100 GB/s), each involving CPU cycles, IRQs, and context switches. At 400 Gb/s wire rate, that's two CPUs at 100% just doing data movement.

RDMA collapses this:

1. GPU memory is registered as a memory region (MR) once.
2. Sender posts a WRITE work request: "put bytes from local VA X to remote VA Y, length N."
3. Local NIC reads GPU memory directly via PCIe peer-to-peer (GPUDirect).
4. Bytes go on the wire.
5. Remote NIC writes directly into GPU memory on the other side — no remote CPU involvement at all.

Zero copies, zero context switches, zero CPU on the receive side. The CPU just polls a completion queue when it cares.

The verbs API in 60 seconds

Everything in RDMA is built from five primitives. Once you understand these, every NCCL log line and every ibv_* tool starts to make sense.

The flow:

1. Open the device: ibv_open_device(...) gets a handle to /dev/infiniband/uverbs0.
2. Allocate a PD so you can group MRs and QPs.
3. Register MRs — call ibv_reg_mr(pd, addr, len, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE). The kernel pins those pages and the NIC's MMU maps them. Pinning is expensive — do this once, reuse forever.
4. Create CQs for completion notification.
5. Create QPs linked to PD and CQs. QP starts in INIT, you bring it through RTR (ready-to-receive) and RTS (ready-to-send) by exchanging info with the peer (LID/GID, QPN, PSN) — usually over RDMA-CM (librdmacm) or a side-channel TCP socket.
6. Post work requests: ibv_post_send(qp, wr, ...). The NIC picks them up by ringing a doorbell (an MMIO write).
7. Poll the CQ with ibv_poll_cq() to know when a WR completed.

That's it. Everything else — RoCE, IB, GPUDirect — is plumbing under that API.

Operations: two-sided vs one-sided

This is the big mental model shift from sockets.

SEND / RECV (two-sided)

Like sockets. Receiver must pre-post a RECV work request describing where to land bytes. Sender posts a SEND. Both sides see a completion. Used for control messages and small transfers.

WRITE (one-sided)

Sender pushes bytes directly into a remote address it already knows (because the receiver told it (va, rkey, len) ahead of time). The receiver's CPU never knows it happened unless you also signal it (e.g., write a flag at the end of the buffer, or use WRITE_WITH_IMM which generates a remote completion). This is the workhorse for bulk data — NCCL allreduce uses RDMA WRITE.

READ (one-sided)

Symmetric: pull bytes from a remote address. Useful when the receiver is the one driving the transfer.

ATOMIC

Compare-and-swap or fetch-and-add at a remote address, atomic from the NIC's perspective. Niche — used for distributed locks, rarely for bulk data.

Transport types: RC vs UD vs RD

Queue pairs come in flavors:

RC is what you debug 95% of the time. It looks like TCP from a behavior standpoint: connection-oriented, reliable, in-order. The HCA does retransmit and ordering in hardware. When you see "vendor err 81 — retry exhausted" in NCCL logs, that's an RC QP that gave up after IBV_QP_RETRY_CNT × IBV_QP_TIMEOUT because the peer never ACKed. The fabric is broken (PFC misconfigured, link flapping, route black-holing), not the application.

User-kernel boundary

+------------------------------------------------+
| Application (NCCL, MPI, custom)                |
+------------------------------------------------+
| libibverbs  +  librdmacm  (user-space verbs)   |
+------------------------------------------------+
| /dev/infiniband/uverbs0  /dev/infiniband/rdma_cm  (uverbs control)
+------------------------------------------------+
| ib_core, mlx5_ib, mlx5_core (kernel modules)   |
+------------------------------------------------+
| HCA hardware (ConnectX-6/7/8)                  |
+------------------------------------------------+

Key insight: the data path never touches the kernel after setup. Posting a WR is a handful of stores into a doorbell-mapped MMIO page; polling a CQ is just reading a host memory ring buffer. That's why RDMA is fast — same reason DPDK is fast.

The kernel only matters for:

• MR registration (pinning + mapping)
• QP state transitions
• Memory-region invalidation on munmap/process death
• Out-of-band events (link state, AER errors)

Walking through what NCCL does at the verbs level

This is the mental model for reading NCCL_DEBUG=INFO logs intelligently. For an 8-node × 8-GPU job over RoCE:

1. Bootstrap over TCP: rank 0 listens, others connect. They exchange "I have these HCAs, here are my GIDs." This is NCCL_SOCKET_IFNAME territory.
2. Per-NIC PD: NCCL allocates one PD per HCA per process (so per GPU: mlx5_0..mlx5_7 mapped to gpu0..gpu7 via rail binding).
3. MR registration: NCCL registers GPU memory using ibv_reg_mr with IBV_ACCESS_RELAXED_ORDERING | IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE. With GPUDirect, addr is a GPU virtual address, not a host one — the nvidia-peermem kernel module makes that work.
4. QP mesh: for ring topology, each rank creates RC QPs to its left and right neighbors on each NIC pair. With NCCL_IB_QPS_PER_CONNECTION=4 you get four parallel QPs per peer pair to spread across HCA hardware queues.
5. QP handshake: NCCL sends QPN/PSN/GID over the bootstrap TCP socket, then transitions both sides INIT → RTR → RTS.
6. Steady state: collective ops translate to sequences of ibv_post_send with WRITE opcodes. Completions are polled in tight loops on a worker thread (NCCL_NSOCKS_PERTHREAD, NCCL_SOCKET_NTHREADS control how many).

In a healthy log:

NCCL INFO NET/IB: Using interface eth0 for sideband communication
NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE ...
NCCL INFO Channel 00 : 0[0] -> 1[0] [send] via NET/IB/0/GDRDMA
NCCL INFO Connected all rings
NCCL INFO comm 0x... rank 0 nranks 16 cudaDev 0 busId 1a000 - Init COMPLETE

GDRDMA in the transport string means GPUDirect RDMA is active (NIC reads GPU memory directly). Without it you'd see via NET/IB/0 with no GDRDMA suffix, meaning data bounces through host RAM — that's the case where RoCE works but you forgot to load nvidia-peermem or ACS is enabled on a PCIe switch.

Tools you'll actually run

ibv_devinfo

Tells you what the HCA reports about itself.

$ ibv_devinfo -d mlx5_0 -v
hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         28.39.2048
        node_guid:                      a088:c203:00ab:cdef
        sys_image_guid:                 a088:c203:00ab:cdef
        vendor_id:                      0x02c9
        vendor_part_id:                 4129
        hw_ver:                         0x0
        board_id:                       MT_0000000838
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               589
                        link_layer:             InfiniBand

Read order: state PORT_ACTIVE, link_layer (InfiniBand or Ethernet for RoCE), active_mtu, fw_ver. vendor_part_id 4129 is ConnectX-7; 4125 is ConnectX-6 Dx.

ibv_rc_pingpong

Tiny verbs ping-pong. Runs in seconds, validates that RC QPs can come up between two hosts. If this fails, RDMA itself is broken — don't bother running NCCL.

host-A$ ibv_rc_pingpong -d mlx5_0 -g 3
host-B$ ibv_rc_pingpong -d mlx5_0 -g 3 host-A
8192000 bytes in 0.01 seconds = 5856.49 Mbit/sec
1000 iters in 0.01 seconds = 11.20 usec/iter

-g 3 picks GID index 3. On RoCE, you must pick the right GID — see RoCE doc for the GID-table layout. On native IB you can usually omit -g.

perftest suite (ib_send_bw, ib_read_bw, ib_write_bw)

The standard bandwidth test. ib_write_bw mirrors what NCCL does (RDMA WRITE on RC).

server$ ib_write_bw -d mlx5_0 -F --report_gbits
client$ ib_write_bw -d mlx5_0 -F --report_gbits server-host
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000           388.50             388.46               0.741
---------------------------------------------------------------------------------------

For a 400 GbE NIC, 388 Gb/s is "good" (line rate after framing). Anything below ~340 Gb/s on a single NIC is suspicious — check PFC, MTU, congestion counters. Add --use_cuda=0 (server) and --use_cuda=0 (client) to drive the buffers from GPU memory and validate the GPUDirect path end to end.

show_gids

Dumps the GID table for all RoCE HCAs — essential for picking NCCL_IB_GID_INDEX.

Common failure modes

WR flush

CQ entries with status IBV_WC_WR_FLUSH_ERR mean a different WR errored before this one and the QP is in ERROR state — all subsequent WRs are flushed. Find the first non-flush error, that's the real one.

Vendor err 81 (retry exhausted)

NCCL WARN NET/IB : Got completion with error 12, opcode 0, vendor err 81

Status 12 (IBV_WC_RETRY_EXC_ERR) means RC's hardware retransmit gave up. Causes, in rough frequency:

1. PFC misconfigured on RoCE — drops happen, RC retries, retries also drop, exhausts. Validate with ethtool -S | grep -i pause on both ends and on the switch.
2. MTU mismatch between hosts (or path MTU through a misconfigured switch).
3. Wrong GID — host is sending RoCE v2 but the GID points at a v1 entry, or DSCP isn't set so packets land in lossy queue.
4. Link flap mid-flight. Check ibstatus, ethtool link counters, switch port state.
5. Subnet manager change on IB causing transient LID re-assignment.

Unreachable

IBV_WC_REM_ACCESS_ERR (status 10) or IBV_WC_REM_OP_ERR (status 11) — the remote side rejected the WR. Wrong rkey, address out of MR bounds, or remote QP in bad state.

MR registration failed

ibv_reg_mr returned NULL, errno=12 (ENOMEM) — usually RLIMIT_MEMLOCK. Set LimitMEMLOCK=infinity in the systemd unit or --ulimit memlock=-1 in Docker. For GPU buffers, also check that nvidia-peermem is loaded:

$ lsmod | grep nvidia_peermem
nvidia_peermem         16384  0

If it's missing, ibv_reg_mr on a CUDA pointer falls back to a host-bounce path (slow) or fails outright depending on the HCA driver.

"No such device" in UCX/verbs

You're enumerating HCAs that don't have GIDs (e.g. NVSwitch management mlx5_4..7 on H200 boards) and trying to bring up QPs on them. Filter to data-plane NICs only with NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_8,mlx5_9,mlx5_10,mlx5_11.

Quick reference: states a QP traverses

Reading NCCL_DEBUG=INFO, you'll often see "QP X moved to RTS" — that's your "TCP connection established" moment.

See also

• RoCE v2 + DCB/PFC/ECN
• InfiniBand fabric primer
• GPUDirect / GDR
• NCCL multi-node tuning
• PCIe topology + ACS