Why Meta Completely Rebuilt RDMA for AI-Scale Ethernet

Reinventing RoCE for AI Scale

Meta open-sourced MetaRoCE last week, and the headline deserves a second look. They didn't upgrade RoCEv2. They designed a replacement from a blank sheet, specifically for AI workloads running over commodity Ethernet at million-GPU scale. To understand why, look at what standard RDMA demands from the network. RoCEv2 expects the fabric to deliver every frame in order. It relies on PFC, priority flow control, to create an artificial lossless environment. It puts the intelligence in the switches. That architecture works fine until you're synchronizing thousands of accelerators across multiple data centers, and a single stalled flow pauses an entire all-reduce. At that scale, the fabric becomes a bottleneck instead of a highway.

Meta's core insight is simple: the fabric sees packets, but the NIC sees intent. Stop asking the network to be a perfect delivery service. Move the smarts to the endpoint.

Designed for Out-of-Order and Multipathing

MetaRoCE treats out-of-order arrival as the normal case. It sprays packets across many ECMP paths deliberately, so they arrive scrambled by design. Every packet carries its own destination memory address. Data writes straight to its final location as it lands. No reorder buffer. No head-of-line blocking. A packet never waits for its predecessor.

Multipathing is native, not an afterthought. Each connection owns multiple first-class paths, and the NIC sprays traffic packet-by-packet. Each path uses a distinct UDP source port for ECMP entropy. The NIC can shift traffic off a bad route instantly because each path maintains its own round-trip estimate and congestion window. The transport distinguishes congestion from failure. A hot link slows one path. The rest of the connection keeps moving.

Embracing a Lossy Fabric

Most importantly, MetaRoCE treats Ethernet as lossy and doesn't try to stop it. No PFC. No pause frames. Because ordering is isolated per path, a gap in a path's selective acknowledgment bitvector means loss, not reorder. The receiver returns a 256-bit SACK. The sender retransmits exactly the missing packet on exactly the path that lost it, immediately. RoCEv2 assumes loss is a fabric failure and collapses. MetaRoCE assumes loss is physics, and it converges gracefully.

At one percent packet loss, MetaRoCE maintains roughly eighty-six percent throughput. At ten percent, it still delivers useful bandwidth. RoCEv2 doesn't.

Bidirectional Congestion Control

Congestion control is bidirectional. The sender uses standard ECN-based AIMD, but the receiver also injects fair-share rate hints into every acknowledgment. Because windows are tracked per path, a congestion mark trims only the affected path and steers packets toward clear ones. The receiver tells each sender exactly what inbound bandwidth share it's allocated. Senders don't hunt for the right speed. They get there directly. Incast—where a storage read fans out to a thousand servers and they all reply at once—resolves in one or two round trips.

Minimal Fabric Requirements

The fabric requirements are minimal. ECN marking and ECMP. That's the entire list. No in-network telemetry. No packet trimming. No credit-based flow control. No switch-side spraying. It works on fat-tree, multiplane, deep-buffer, shallow-buffer, or a vendor cloud you don't control.

An Elegant Structural Change

There's an elegant structural change here too. Traditional RDMA opens dozens of queue pairs per node pair to increase parallelism. Each QP carries its own congestion state, blind to the rest, and each burns NIC resources. MetaRoCE separates message ordering from bandwidth. One connection carries many independent ordered streams above—one per communicator or collective—and many paths below, under a single congestion controller. Connection state stops growing with workload parallelism. The application layer barely changes. Existing RDMA Verbs APIs work without modification.

Hardware Integration and Performance

The numbers come from AMD Pensando programmable NICs on a sixty-four-node GPU cluster. MetaRoCE beat RoCEv2 on all-reduce and all-to-all completion times consistently. Multiplane tests across four-plane and eight-plane topologies with four thousand concurrent connections showed linear scaling. When they simulated plane failures, traffic redistributed autonomously with no operator intervention.

This transport isn't just a paper specification. Meta co-designed it with their MTIA three hundred training chip, which puts the network interface inside the chip package itself. Two network chiplets with twelve custom eight-hundred-gigabit RDMA NICs provide 1.2 terabytes per second of total I/O bandwidth without ever crossing a PCIe bus. The same NICs flexibly partition between scale-up traffic inside a rack—up to one terabyte per second—and scale-out traffic across racks at two hundred gigabytes per second. You reconfigure the split in software, not hardware.

To cut latency further, MTIA three hundred uses express doorbells. The work request write itself serves as the doorbell, eliminating an extra memory read and saving roughly eight hundred nanoseconds per operation. The chip also includes sixteen dedicated message engines that handle all communication independently from the compute grid. Each engine contains a RISC-V core, an NIC interface, and a near-memory compute block that performs reductions at 128 bytes per cycle. Collectively, those blocks deliver over 2.8 terabytes per second of reduction throughput.

Running large matrix multiplications concurrently with collectives introduces less than half a percent degradation to compute throughput. On a traditional GPU, that same contention can degrade performance by over twenty percent because communication kernels steal the same streaming multiprocessors used for training.

The communication library, HCCL, compiles collectives into complete subgraphs of work-queue entries with explicit dependencies, then dispatches them to those message engines for fully autonomous execution. Once the host copies instructions into HBM, it's uninvolved. On a 150-billion-parameter production recommendation model across forty accelerators, MTIA three hundred's total communication time was 3.9 times faster than the equivalent GPU cluster.

The Architectural Philosophy

Most software developers will never choose a transport protocol. You rent instances. The network is someone else's problem. MetaRoCE is infrastructure plumbing for the hyperscaler tier, and that's perfectly fine. But the architectural philosophy isn't invisible. Assume your paths are independent. Assume order is your problem, not the network's. Assume loss is normal and recovery must be local. That's how you build systems that degrade gracefully instead of falling off a cliff when one link flaps.

Your cloud provider will absorb MetaRoCE or something shaped like it, and your distributed systems will run faster because the layer beneath them stopped pretending the network is a single ordered pipe.

Open for Everyone

Meta is releasing the full specification, a DPDK-optimized software reference implementation called libsoftmetaroce, and a production compliance suite through the Open Compute Project. The reference stack runs on commodity Linux over standard UDP sockets without specialized hardware. That matters. It keeps the playing field open to multiple vendors and the behavior verifiable. It also means vendors can prove interoperability before you ever buy their NICs.

If you're running AI workloads on Ethernet—and right now, that's essentially everyone—this is where transport is heading. Simpler assumptions on the fabric. Smarter endpoints. Protocols that perform better in ideal conditions and don't die when reality gets messy.

This is TAKEYOURPILLS.TECH. Go ship something.