What is OCP MRC 1.0? Open AI Supercomputing Networking Explained

Scalable_AI_Networking_with_MRC
Scalable_AI_Networking_with_MRC

Revolutionizing AI Supercomputing Networking: Inside the OCP Multipath Reliable Connection (MRC) 1.0 Specification

When training modern multi-billion parameter AI models, the network isn’t just a conduit for data—it is the backbone of the supercomputer. In large-scale synchronous training workloads, a single delayed packet or localized network glitch can stall thousands of GPUs, idling million-dollar clusters and burning precious energy.

To solve the limitations of traditional single-path transports in hyperscale fabrics, industry leaders—AMD, Broadcom, Intel, Microsoft, OpenAI, and NVIDIA—jointly published the Open Compute Project (OCP) Multipath Reliable Connection (MRC) Specification Revision 1.0.

Here is an accessible, deep-dive breakdown of what MRC is, how it works, and why it changes AI infrastructure networking forever.

What is Multipath Reliable Connection (MRC)?

At its core, MRC is an open transport protocol specification built as an extension to the widely adopted InfiniBand / RoCE Reliable Connection (RC) transport model. While standard RC locks a communication queue pair (QP) to a single network path, MRC introduces explicit per-packet multipath load balancing (packet spraying), allowing a single connection to distribute traffic across hundreds of network paths simultaneously.

Designed to run over standard commodity best-effort Ethernet, MRC overcomes the traditional vulnerabilities of single-path networking—such as link flaps, localized switch congestion, and routing stalls—without requiring specialized or proprietary fabrics.

Key takeaway: MRC preserves the familiar RDMA software programming model while delivering adaptive, fault-tolerant packet delivery under the hood.

Why Legacy RDMA Networking Struggles at AI Scale

Traditional Remote Direct Memory Access over Converged Ethernet (RoCEv2) relies on ECMP (Equal-Cost Multi-Path) hashing at connection setup to pin a Queue Pair (QP) to one fixed network route.

This creates three critical pain points in massive AI clusters:

  1. Hash Polarization & Stragglers: Multiple QPs frequently hash onto the same link, creating “hotspots” and severe tail latency while adjacent links remain underutilized.
  2. Failure Amplification: A single link flap or failed switch drops connection packets, forcing Go-Back-N retransmissions and freezing synchronous AI training steps.
  3. Over-Provisioning Overhead: Operators are forced to build deep, complex switch fabrics to guard against transient congestion.

Key Technical Features & Call Flows of OCP MRC 1.0

The MRC 1.0 specification addresses these bottlenecks through a set of tightly integrated architectural extensions:

1. Per-Packet Multipath Spraying

Instead of routing an entire flow down one path, MRC assigns an Entropy Value (EV) to every packet. By rotating EVs dynamically, the sender “sprays” request packets across all available fabric paths, achieving maximum bandwidth utilization.

MRC supports three primary routing models:

  • ECMP Hash-Based: Uses standard switch hashing via UDP source ports.
  • Structured EV: Encodes deterministic, hop-by-hop switch routing instructions directly into the lower 16 bits of the IPv6 Flow Label and UDP Source Port.
  • SRv6 (Segment Routing over IPv6): Uses micro-segment identifiers (uSID) to explicitly steer packets through target switch paths.

2. End-to-End Operation & Fast Loss Recovery Call Flow

The diagram below illustrates the protocol interaction between a Requestor (Sender), Intermediate Fabric Switches, and a Responder (Receiver) during normal multipath spraying, ECN congestion feedback, and fast recovery via Trimmed Packet NACKs.

+-----------+                +-----------------+               +-----------+
| Requestor |                | Switch Fabric   |               | Responder |
+-----+-----+                +--------+--------+               +-----+-----+
      |                               |                              |
      | --- Write Req (PSN 100, EV1)->|                              |
      | --- Write Req (PSN 101, EV2)->|                              |
      |                               | --- Write Req (PSN 100) ---->|
      |                               |                              | (Out-of-Order:
      |                               |                              |  PSN 100 received)
      |                               |                              |
      |                               | [Fabric Congestion Occurs]   |
      | --- Write Req (PSN 102, EV3)->| --- (Payload Trimmed) ------>| (Receives Header-Only
      |                               |    (DSCP_TRIMMED)            |  "Trimmed Packet")
      |                               |                              |
      |                               |                              | (Generates Fast TRIM NACK
      |                               |<-- TRIM NACK (PSN 102, EV3)--|  with reflected EV)
      |<-- TRIM NACK (PSN 102) -------|                              |
      |                               |                              |
      | (Processes NACK &             |                              |
      |  Marks EV3 as "SKIP")         |                              |
      |                               |                              |
      | --- Retransmit (PSN 102, EV4)->----------------------------->| (Direct Placement
      |     (BTH.rtx=1, new EV)       |                              |  into memory)
      |                               |                              |
      |                               |<-- SACK (cack=102, m=1) ------| (SACK with Bitmaps &
      |<-- SACK (cack=102, ECN=1) ----|                              |  Congestion Signals)
      |                               |                              |
      | (NSCC Congestion Window       |                              |
      |  Adjusted / EV4 Evaluated)    |                              |
      +                               +                              +

3. Selective Acknowledgments (SACKs) & NACKsMRC decouples network packet delivery tracking from semantic RDMA processing. Responders generate Reliability SACKs containing a 64-bit reception bitmap and cumulative ACK tracking.

This allows requestors to immediately identify missing packet sequence numbers (PSNs) caused by reordering versus actual loss, triggering selective retransmissions rather than halting the entire flow.

4. Fast Loss Recovery with Trimmed Packets

In congested switches, instead of silently dropping packets, the network can “trim” payload data and forward lightweight headers (DSCP_TRIMMED) to the destination. Upon receiving a trimmed packet, the responder sends a fast Negative Acknowledgment (TRIM NACK) back to the sender, prompting sub-microsecond retransmissions without waiting for timeout timers to expire.

5. Network Signal Congestion Control (NSCC)

MRC integrates the UltraEthernet Consortium (UEC) NSCC algorithm. Operating per-QP, NSCC monitors two critical signals:

  • Request RTT: Measures queueing delay using reflected timestamps (TSETH headers).
  • ECN (Explicit Congestion Notification): Signals fabric congestion before packet drop thresholds are hit.

6. Multi-Plane NIC Support & Path Health Tracking

In multi-port NIC configurations, MRC allows a single connection to spray across distinct physical network planes. Senders track per-EV health states (GOOD, SKIP, DENIED, ASSUMED_BAD) and issue out-of-band Reliability Probes or EV Probes to automatically bypass unhealthy or degraded links.

Software Ecosystem: libmrc Architecture

To simplify integration into host software and AI frameworks (like NCCL and RCCL), MRC exposes a standardized dual-API software stack:

  • Application API (mrc.h): Modeled directly after standard libibverbs. It allows user-space applications to manage Queue Pairs (QPs), Completion Queues (CQs), and execute high-speed RDMA Write and Write-with-Immediate (WriteIMM) operations.
  • Controller API (mrc_ctl.h): Designed for a privileged system daemon (CAP_NET_ADMIN). It manages background device parameters, configures EV and Congestion Control profiles, monitors network path health, and dispatches path probes.

Business & Infrastructure Impact

DimensionStandard RoCEv2 (RC)OCP MRC 1.0
Path SelectionStatic single-path hashing per QPPer-packet multipath spraying (ECMP, Structured EV, SRv6)
Network Loss HandlingGo-Back-N retransmission on packet lossSelective ACK (SACK) & sub-microsecond Trim NACK retransmit
Fabric UtilizationHotspots due to hash collisionsHighly balanced across 100+ paths
Failure ToleranceLink flap stalls or crashes training jobAutonomous fast rerouting & path denylisting
Supported OperationsWrite, Read, Send, AtomicsOptimized specifically for AI: RDMA Write & WriteIMM

Advancing OCP Core Tenets

  • Openness: Created collaboratively by six industry giants to foster a multi-vendor ecosystem across NICs, switches, and accelerators.
  • Efficiency & Sustainability: Maximizes effective throughput on existing commodity Ethernet infrastructure, preventing idle GPU cycles and lowering energy consumption per job.
  • Hyperscale Scalability: Enables self-healing connectivity that scales smoothly from local compute pods to hyperscale supercomputers.

Conclusion

The release of the OCP Multipath Reliable Connection (MRC) 1.0 Specification marks a major step forward in open AI networking. By combining per-packet multipath spraying, rich SACK loss recovery, and adaptive congestion control over standard Ethernet, MRC provides the predictability and scale required for next-generation AI workloads.

Leave a comment