Welcome to NVMe over Fabrics (NVMe-oF) for Ultra-Low Latency Storage. For years, decoupling compute nodes from storage nodes meant accepting the heavy latency penalty of iSCSI over TCP. NVMe-oF fundamentally changes this paradigm, allowing network-attached storage to operate at the speed of local PCIe.
1. The Problem with Legacy iSCSI and SATA
Traditional SATA/SAS drives communicate via the AHCI protocol, which was designed for spinning rust and allows only a single command queue with 32 commands. iSCSI adds another layer of TCP/IP overhead. When dealing with millions of IOPS in high-performance computing (HPC) or clustered databases, this overhead chokes the CPU and creates massive I/O wait times.
2. NVMe: Parallelism at the Hardware Level
NVMe (Non-Volatile Memory Express) bypasses the SATA controller entirely, connecting directly to the PCIe bus. It supports 64,000 command queues, each capable of holding 64,000 commands. This massive parallelism unlocks the true speed of NAND flash memory.
3. Extending NVMe across the Network (NVMe-oF)
NVMe-oF takes the NVMe command set and encapsulates it into a network transport protocol. While it can run over TCP (NVMe/TCP), the most performant implementations use Remote Direct Memory Access (RDMA) via RoCE (RDMA over Converged Ethernet) or InfiniBand.
RDMA allows a compute node to read from or write to the memory of the storage node directly, bypassing the OS kernel and CPU network stack on both sides. This drops storage latency from milliseconds down to low microseconds.
4. Use Cases in Cloud Architectures
With NVMe-oF, High Availability architectures can finally achieve local-disk performance while maintaining the benefits of decoupled storage (instant snapshots, volume replication, and live migrations). AI/ML training clusters, high-frequency trading platforms, and massive PostgreSQL instances benefit enormously from this architecture, pulling gigabytes per second with near-zero CPU overhead.
Conclusion
NVMe-oF bridges the gap between Direct Attached Storage (DAS) and Storage Area Networks (SAN). As 100GbE and 400GbE networks become standard in data centers, NVMe-oF is rapidly rendering legacy iSCSI architectures obsolete.