The Shift to Agentic Inference

Agentic workloads differ fundamentally from classic LLM serving. They are characterized by multi-turn interactions (up to 3,000 turns), high cache hit rates (>90%), and massive input-to-output token ratios (often >100:1). Public benchmarks typically ignore these dynamics, focusing on sanitized, steady-state performance. In production, the client-driven nature of these sessions makes KV cache management volatile, leading to frequent evictions and high costs—cached tokens can be 10x cheaper than uncached ones, making cache locality a primary economic and performance lever.

KV Cache-Aware Routing

To optimize for these workloads, the inference stack must coordinate routing with cache availability. The LLM-D framework uses an "Endpoint Picker" (EP) plugin to score pods based on load and KV cache locality. By routing requests to pods where the system prompt or context is already cached, systems can reduce Time-to-First-Token (TTFT) significantly—for example, dropping from 3 seconds to 1 second for cached turns. This approach treats cache locality as a first-class scheduling metric rather than an afterthought.

Prefill-Decode (P/D) Disaggregation

Aggregated serving forces a single GPU to handle both prefill (compute-intensive, bursty) and decode (memory-bandwidth-intensive, latency-sensitive). This leads to "phase interference," where a long incoming prompt stalls ongoing token generation, causing jitter.

Disaggregation separates these tasks into independent pools:

  • Prefill Workers: Optimized for high throughput and batch parallelism.
  • Decode Workers: Optimized for low-latency token generation.

This architecture allows for independent scaling. However, it is not a universal solution. Disaggregation requires a high-speed network fabric (RDMA or RoCE) to transfer KV caches between workers. Without this fabric, the overhead of moving cache data outweighs the benefits, and aggregated serving remains the better choice. P/D disaggregation shines most in middle-concurrency regimes where the workload is prefill-heavy, providing smoother P99 inter-token latency (e.g., improving from 900ms to 100ms).

Practical Implementation

For real-world deployments on hardware like H200s, the authors recommend a modular architecture. In a recent case study using GLM 5.2, they utilized a 3:1 ratio of prefill to decode workers. This configuration yielded 4x faster TTFT and 60% higher request capacity compared to standard aggregated setups. Additionally, they noted that using BF16 for KV cache storage can outperform FP8 for long prefill tasks, highlighting the importance of continuous experimentation with hardware-specific knobs.