The Economics of Inference: Prefill and KV Cache

Inference stacks are primarily constrained by the cost of prefill—the compute-heavy process of processing input tokens. Because agentic workflows often feature 80-90% overlap in input sequences between consecutive requests, the platform's primary optimization lever is KV cache management.

To minimize recomputation, the system prioritizes KV cache locality during routing. Rather than evicting cache between chat turns, the platform offloads it to high-bandwidth storage, allowing the system to reload the context into HBM (High Bandwidth Memory) immediately when a follow-up request arrives. This avoids the expensive prefill stage entirely, directly improving latency and reducing costs.

Managing Workload Shapes via 'Tetris' Scheduling

Platforms must support diverse workload shapes—agentic, chat, voice, video, and batch—without forking the underlying stack. The strategy involves:

  • Heterogeneous Routing: The router balances requests across different GPU generations and zones, prioritizing cache locality first and least-loaded status second.
  • Temporal Load Balancing: Real-time workloads (agentic/chat/voice) dominate daytime capacity, while batch workloads—which have loose SLAs and can tolerate hours of latency—are scheduled to fill idle capacity overnight.
  • Consumption Models: The platform serves two primary models:
    • Serverless: Pay-per-token with optional "provisioned throughput" to avoid noisy neighbor issues.
    • Dedicated: Billed per GPU hour with private gateways, allowing customers to own the deployment, select the inference engine (e.g., vLLM, SGLang, TensorRT-LLM), and decide whether to disaggregate prefill and decode.

Performance Levers for Throughput

To maximize price-performance, the platform employs several specific technical optimizations:

  • Quantization: Utilizing 4-bit quantization (NVFP4) to reduce memory footprint.
  • Asynchronous Speculators: The platform trains speculators on a customer's specific dataset asynchronously. These speculators are then deployed to increase acceptance lengths, significantly boosting output throughput.
  • Disaggregated Architectures: Providing the flexibility to separate prefill and decode engines, which allows for more granular scaling depending on the specific latency requirements of the workload.