RDMA and Kernel Bypass

Normally, sending data over a network is a chain of hand-offs. Your program copies the data into the operating system, the operating system copies it into the network card, and on the far side the same thing happens in reverse. Each hand-off costs a copy and an interruption, and the receiving program has to be woken up to collect its mail.

RDMA removes the hand-offs. The program tells the network card, once and in advance, which region of memory it is allowed to read and write. After that the card on one machine writes directly into memory on another - no copies through the operating system, and nobody woken up in between. The remote CPU is not merely fast here; it is uninvolved.

The interface is verbsverbsThe de-facto RDMA programming interface: post a work request, poll a completion, never enter the kernel on the data path., and the connection abstraction is the queue pairqueue pairA send queue and a receive queue, together forming one RDMA connection endpoint. - a send queue and a receive queue that together form one endpoint.^{NVIDIA CUDA Docs} Work is not requested through a system call; it is posted as a descriptor onto a queue that the application shares with the adapter, and completions are reaped from a separate completion queue. That shared-memory arrangement is exactly what makes kernel bypass possible.

Three properties fall out of it, and they are usually named together: OS bypass, because the kernel is not on the data path; zero copy, because the adapter reads the application's own buffer; and offload, because packet processing happens on the adapter rather than on a core. Everything above this layer - NCCL, MPI - is ultimately posting work requests to queue pairs.

The cost that matters in practice is not on the data path at all. It is memory registrationmemory registrationPinning a memory region and handing the adapter the keys to it, so it can be read or written without the OS in the loop.: before the adapter can touch a region, that region must be pinned and its translation handed over. Registration is expensive and the pages must stay resident, so real stacks register a small number of large buffers once at startup and reuse them for the lifetime of the job.

This is worth knowing because it is the most common way a benchmark lies. A microbenchmark that registers inside its timing loop measures registration, not transfer, and will report a fraction of what the fabric can do. It is also why buffer pooling shows up in every serious RDMA library - the pool exists to amortise registration, not allocation.

Verbs get the kernel out of the way, but for GPU traffic that is only half the problem. If the adapter can only reach host memory, then a GPU-to-GPU transfer stages through system RAM on both sides - the data crosses PCIe twice more than it needs to, and host memory bandwidth becomes the ceiling on a transfer that never wanted to be there.

GPUDirect RDMA lets the adapter read and write GPU memory directly, "without needing to copy data to host memory".^{NVIDIA CUDA Docs} The nvidia-peermem module registers GPU memory with the InfiniBand subsystem using peer-to-peer APIs from the GPU driver, so a GPU buffer can be the target of an ordinary memory registration.

One detail here matters more than it first appears: GPUDirect RDMA works over RoCERoCERDMA over Converged Ethernet - the same verbs and queue pairs, carried by Ethernet instead of InfiniBand. as well as InfiniBand.^{NVIDIA CUDA Docs} The programming model - queue pairs, verbs, registered regions, direct GPU access - is identical on both fabrics. That is the single most important fact for the comparison at the end of this section, because it means the choice between InfiniBand and Ethernet is not a choice about how software is written. It is a choice about how the wire behaves underneath software that is written the same way either way.

It also explains why this chapter sits before the topology chapters rather than after them. Once transfers are zero-copy and CPU-free, essentially all remaining performance lives in the fabric - in whether the packet gets a clear path, and in how far it has to travel to find one.