The Shift from Chat to Agentic Inference

Traditional inference stacks are optimized for chat, where the unit of work is a single request-response pair. Agentic workflows, however, operate on a task-based loop: planning, acting via tools, observing results, and appending data to a growing context. Because agents run for minutes or hours and frequently share massive prefixes across consecutive steps, recomputing these prefixes for every call is computationally wasteful. The primary metric for success in agentic systems is not the latency of a single token, but the total time to complete the end-to-end task.

Four Pillars of Agentic Inference

To optimize for task latency, FriendliAI has developed an inference stack built on four technical pillars:

  • Prefix Caching: By computing the key-value (KV) cache for shared prefixes once and reusing it for subsequent steps, the system significantly improves time-to-first-token and reduces redundant compute.
  • Hierarchical KV Cache Management: To handle growing contexts, the system uses memory-efficient packing and distributes the cache across GPU memory, host memory, and disk, allowing for context lengths that exceed individual GPU limits.
  • Cache-Aware Routing: Rather than using naive load balancers that distribute requests evenly, the system routes requests to replicas that already hold the relevant prefix in cache, turning expensive prefill operations into simple cache hits.
  • Agent-Aware Scheduling: By understanding that individual LLM calls are part of a larger program, the scheduler can make smarter decisions regarding speculative prefilling, preemption, and cache eviction.

Economic and Operational Impact

Open-weight models have reached a capability threshold where they rival closed frontier models for agentic tasks. When combined with an agent-optimized inference stack, the economic benefits are significant. In a split test conducted by Kilo Code, using FriendliAI’s stack with open-weight models resulted in performance that was seven times faster with a lower error rate compared to other providers. This demonstrates that the combination of frontier-quality open models and specialized inference infrastructure is the key to making complex agents economically viable for production environments.