The Challenge of Disaggregated LLM Serving
In disaggregated LLM serving architectures, compute and memory resources are decoupled, allowing for flexible scaling. However, this introduces significant complexity in request routing. Traditional load-balancing heuristics often fail because they do not account for the heterogeneous performance characteristics of different model instances or the dynamic, non-deterministic nature of LLM inference latency—where time-to-first-token (TTFT) and inter-token latency vary wildly based on input length and output generation.
The Two-Stage Routing Framework
The authors propose a "Calibrate, Then Route" paradigm to address these inefficiencies.
- Calibration: Raw latency predictions from models are often biased by system-level noise (e.g., network jitter, concurrent request interference). The calibration stage applies a learned correction factor to these raw estimates, transforming them into reliable, real-time performance expectations for specific hardware-model pairings.
- Routing: Once calibrated, the router uses these refined estimates to make informed scheduling decisions. By treating routing as an optimization problem—minimizing the aggregate latency across the entire cluster—the system can effectively balance load even when individual model performance fluctuates.
Performance Impact
The study demonstrates that this approach significantly outperforms standard round-robin or simple least-loaded routing strategies. By accounting for the specific performance profile of the model and the current state of the cluster, the system reduces tail latency and increases overall throughput. The key insight is that routing logic must be aware of the underlying inference dynamics; without calibration, the router is essentially making decisions based on 'stale' or 'noisy' data, which leads to suboptimal resource utilization and increased queuing delays for end-users.