From Reactive Loops to Global Optimization
Early inference routing at OpenAI relied on proportional controllers that adjusted weights based on real-time engine signals. While this approach was self-balancing, it suffered from two critical flaws: it was difficult to reason about why specific routing decisions were made, and it created harmful oscillations. When traffic was shifted away from a "hot" engine, the engine cooled, causing the controller to immediately send traffic back. This "bouncing" effect destroyed KV cache locality, which is essential for efficient LLM inference.
To solve this, the team moved to a decoupled architecture consisting of a control plane and a data plane:
- Control Plane: Maintains a global view of all CPU clusters and GPU engines. It runs an optimizer that consumes real-time signals (TTFT, throughput, health) and offline regression data to compute globally optimal routing weights.
- Data Plane: Resides on CPU clusters and makes synchronous routing decisions using a locally cached snapshot of the routing weights. This ensures that the request path remains fast and does not wait for the optimizer.
The Optimizer: Beyond Nearest-Neighbor
Routing to the "nearest" engine is insufficient because traffic demand and GPU capacity are rarely geographically balanced. The optimizer treats routing as a minimization problem for expected end-to-end latency, which includes both network distance and engine-side queuing delays.
Inputs to the optimizer include:
- Current request demand per CPU cluster.
- Network latency metrics.
- Engine health and capacity constraints.
- Latency profiles (TTFT/TBOT) based on engine load.
By modeling these variables, the system can intelligently route traffic to a further engine if the nearby engine is nearing capacity, effectively trading off network latency to avoid significantly higher engine-side wait times.
Production Stability and Protection
To maintain reliability under heavy load, the system implements three specific protection layers:
- Outlier Penalties: Automatically reduces routing weights to engines exhibiting degraded performance, allowing them to recover or be replaced without impacting the fleet.
- Dynamic Retry Budgets: To prevent "retry storms" during high utilization, the system enforces strict, dynamic retry caps. These budgets tighten as system utilization increases, ensuring that retries do not exacerbate existing congestion.
- Load Shedding: As a final resort, the system proactively sheds a portion of traffic when demand exceeds total capacity, ensuring the system degrades gracefully rather than suffering a total outage.