The Shift from Throughput to Latency

Modern AI workloads, particularly in inference and agentic applications, have shifted the primary networking bottleneck from raw throughput to tail latency. Historically, AI training relied on massive, long-running transfers where throughput was the only metric that mattered. Today, frequent small coordination messages—such as KV cache lookups and barrier synchronizations—are critical. Because these messages often occur between compute phases, high latency in even a single exchange causes GPUs to sit idle, directly degrading overall system throughput.

Why Legacy Protocols Fail

TCP and RDMA (specifically RoCE) are poorly suited for these modern requirements for two primary reasons:

  • Sender-Driven Congestion Control: These protocols rely on the sender to detect congestion reactively, often after packet loss or delayed switch signals. This leads to unstable oscillations and high tail latency because the sender is too far removed from the actual bottleneck (the egress port at the receiver's switch).
  • Byte Stream Model: TCP and RDMA treat data as an opaque stream of bytes. They lack awareness of message boundaries, making it impossible to prioritize short, critical tasks. This results in "head-of-line blocking," where small, time-sensitive messages get stuck behind large, non-urgent data transfers.

The Homa Protocol: A Clean-Slate Redesign

Homa is a transport protocol designed specifically for data center environments to handle the mix of large and small messages. It introduces three core architectural changes:

  • Message-Based Architecture: Homa treats data as discrete remote procedure calls (RPCs) rather than byte streams. By understanding message boundaries, it can predict traffic needs and apply Shortest Remaining Processing Time (SRPT) scheduling to prioritize short messages.
  • Receiver-Driven Flow Control: Instead of the sender guessing when to transmit, the receiver controls the flow. It sends "grant" packets to the sender, pacing the transmission to prevent congestion at the switch before it happens. This allows the receiver to prioritize its most important incoming messages.
  • Priority Queue Utilization: Homa leverages the multiple hardware queues already present in modern data center switches. By dynamically assigning packets to different priority levels, Homa allows short messages to bypass the queues occupied by large, long-running transfers.

Performance Impact

Benchmarks demonstrate that Homa significantly outperforms TCP in mixed-workload environments. For short messages, Homa reduces 99th percentile (tail) latency by over 10x (from >1ms to <100μs). Notably, this prioritization does not penalize large messages; Homa also improves performance for large transfers by nearly 2x compared to TCP, thanks to more efficient "run-to-completion" scheduling.