Overcoming Prefix-Caching Limitations

Traditional prefix-caching systems for LLMs are restricted to contiguous, leading prompts, which limits their effectiveness in scenarios where shared content appears at arbitrary positions. KVBoost addresses this by implementing a chunk-level KV cache reuse system that decouples content from its absolute position in the prompt. This allows the system to identify and reuse previously computed KV tensors even when the shared data is not at the start of the input.

Dual-Hash Keying and Accuracy Repair

To manage the complexity of non-contiguous reuse, KVBoost utilizes a dual-hash keying scheme that separates the prefix hash (positional identity) from the content hash (content identity). This enables both exact and approximate matching of cached chunks. Because independent chunk caching can introduce attention boundary errors, the system employs two specific repair strategies:

• SelectiveRecompute: Re-encodes only the boundary regions of the cached chunks to ensure coherence.
• CacheBlendRecompute: Performs a probe pass to identify tokens with high deviation, then selectively recomputes those specific tokens to maintain model output quality.

Performance and Efficiency

KVBoost is designed for practical deployment with memory-bounded constraints. It incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction to manage memory usage effectively. In evaluations using Qwen/Qwen2.5-3B, the system achieved a 4.49x reduction in time-to-first-token (142.4 ms vs. 639.1 ms) and outperformed standard prefix caching by 16%, all while maintaining 99.2% accuracy compared to the 99.1% baseline.