NICs — 25GbE to 400GbE, InfiniBand & RDMA
Complete buyer guide & technology reference

RDMA allows one server to directly read from or write to the memory of another server over the network, completely bypassing the OS kernel and CPU on both ends. The receiving CPU is not involved in the data transfer at all. This eliminates kernel processing overhead and reduces latency from ~50µs (kernel TCP) to ~1–3µs (RDMA). RoCEv2 runs RDMA over standard Ethernet, requiring only a RoCE-capable NIC on each end and a lossless Ethernet fabric (PFC/ECN enabled on switches). GPUDirect RDMA extends this to allow direct GPU-to-GPU transfers without copying data through system RAM.

InfiniBand (IB) is a purpose-built high-performance fabric historically used in HPC supercomputers. It provides inherently lossless transport, native RDMA, and very low latency (sub-200ns). NVIDIA ConnectX-7 supports InfiniBand NDR (400 Gb/s) as well as Ethernet. Modern Ethernet with RoCEv2 and PFC/ECN approaches InfiniBand's performance for AI workloads, and offers lower switch costs, easier integration with existing network infrastructure, and more flexible topology options. Most enterprise AI deployments use RoCEv2 over Ethernet; pure HPC supercomputers often use InfiniBand for the lowest possible MPI latency.

SR-IOV allows a single physical NIC to present multiple "virtual functions" (VFs) directly to virtual machines or containers, each with dedicated hardware queues. Without SR-IOV, all VMs share the physical NIC through the hypervisor — the hypervisor copy overhead adds latency. With SR-IOV, a VM with a VF assignment can achieve near-native NIC performance. Essential for latency-sensitive VM networking (NFV, telecom, real-time databases). Supported on NVIDIA ConnectX series, Intel E810, and Broadcom P225P.

GPUDirect RDMA (GDR) allows the NIC to transfer data directly between GPU memory and the network, bypassing system RAM entirely. In a GPU training cluster, GDR means gradient updates can be sent directly from GPU memory to the network buffer without the CPU first copying them into RAM. This significantly reduces latency for distributed training (NCCL all-reduce operations). GDR requires an NVIDIA ConnectX NIC and NVIDIA GPU on the same PCIe segment, plus the GDR kernel module. All NVIDIA DGX systems include GDR-capable ConnectX NICs.

DPDK is a set of libraries that allow network applications to process packets directly in userspace, bypassing the Linux kernel network stack. Instead of the kernel handling each packet interrupt, the application polls the NIC directly. This eliminates kernel overhead and enables millions of packets per second (Mpps) on commodity servers. DPDK is mandatory for telco NFV/vRAN, software-defined networking (Open vSwitch), and high-frequency trading applications. Supported on Intel E810, NVIDIA ConnectX, and Broadcom NICs.

NVMe-oF (NVMe over Fabrics) uses RDMA to extend the NVMe block storage protocol over a network. The server NIC acts as both an NVMe-oF initiator (connecting to remote storage) and optionally a target (serving local NVMe drives to other servers). This enables disaggregated storage — NVMe SSDs in storage servers are accessed by compute servers at latencies of 10–20µs, compared to 1–3ms for iSCSI or FC-SCSI. Pure Storage FlashArray, NetApp AFF, and Dell PowerStore support NVMe-oF. Requires RoCEv2-capable NICs (ConnectX-6 Dx or newer).