The Memory Bottleneck in Long-Lived Sessions
As LLM applications move toward longer context windows and persistent, multi-turn interactions, the Key-Value (KV) cache has become a primary memory bottleneck. Storing the entire KV cache in GPU VRAM is unsustainable for high-concurrency or long-context scenarios, as it leads to rapid memory exhaustion and limits the number of active sessions a single node can support.
Tiered Placement Policies
The research proposes a hierarchical approach to KV cache management, treating memory as a tiered system rather than a single pool. By distributing the cache across GPU VRAM, CPU RAM, and NVMe SSDs, systems can maintain significantly more concurrent sessions than VRAM-only architectures allow.
- GPU VRAM (Hot Tier): Reserved for the most active tokens or the immediate working set to ensure low-latency generation.
- CPU RAM (Warm Tier): Acts as a buffer for sessions that are active but not currently generating, providing a faster retrieval path than disk while freeing up precious GPU space.
- SSD (Cold Tier): Used for long-lived, idle sessions. While retrieval latency is higher, it allows for virtually infinite context persistence, enabling applications to resume long conversations without re-processing the entire prompt from scratch.
Balancing Latency and Throughput
The effectiveness of this tiered strategy depends on predictive prefetching and eviction policies. The system must anticipate when a session is about to become active to move its KV cache from the SSD or CPU RAM back into the GPU before the next token generation request arrives. This hides the latency of data movement, ensuring that the user experience remains responsive despite the physical distance of the data from the compute unit. The core trade-off is between the increased memory capacity provided by the lower tiers and the potential latency penalty if the prefetching logic fails to predict the next active session accurately.