The Inference Trade-off Triangle

Building production-ready LLM services requires managing a constant tension between three competing metrics: Quality, Latency, and Throughput.

  • Quality: Often tied to context length. Reducing context to fit more users on a GPU degrades model performance.
  • Latency: Measured by Time to First Token (TTFT) and inter-token latency (decode speed).
  • Throughput: The number of concurrent users a single GPU can support.

Every architectural decision involves sacrificing one of these to optimize the others. For example, a premium chat application prioritizes low latency and high quality, while an asynchronous agentic workload might prioritize throughput and quality, accepting higher latency.

Understanding the Bottlenecks

Inference performance is dictated by the interaction between model weights and the KV (Key-Value) cache.

  • Memory Growth: The KV cache grows linearly with context length. For a Mistral 7B model, each token requires 131 KB of KV cache. At 16k context length and 80 concurrent users, the cache alone consumes 42 GB of VRAM, exceeding the capacity of standard 24 GB consumer cards.
  • Pre-fill vs. Decode:
    • Pre-fill (Compute-bound): Processing input tokens to generate the initial KV cache. High arithmetic intensity; latency scales with input size.
    • Decode (Memory-bound): Generating tokens one by one. The bottleneck is the memory bandwidth required to pull the KV cache for all previous tokens from High Bandwidth Memory (HBM) to the GPU's shared memory.

Model-Side Optimizations

To reduce the footprint of the model itself, researchers use several techniques:

  • Quantization: Reducing the precision of model weights (e.g., from 16-bit to 4-bit) to free up VRAM for the KV cache.
  • Attention Variants: Moving from standard Multi-Head Attention to Multi-Query Attention (MQA) or Grouped-Query Attention (GQA) to drastically reduce the size of the KV cache per token.
  • Flash Attention & Tiling: Optimizing the attention mechanism to minimize memory reads/writes by keeping data in faster, on-chip memory (SRAM) as long as possible.

Serving-Side Optimizations

Serving engines like vLLM and SGLang implement specific strategies to maximize GPU utilization:

  • Paged Attention: Borrowed from OS memory management, this allows non-contiguous memory allocation for the KV cache, eliminating memory fragmentation.
  • Continuous Batching: Instead of waiting for a batch of requests to finish, the server injects new requests as soon as individual sequences complete, keeping the GPU saturated.
  • Prefix Caching: Reusing the KV cache for common prompt prefixes (e.g., system prompts) across different requests, saving significant compute.

Engine Selection

For standard workloads, the choice between vLLM and SGLang is often negligible. However, when moving to complex agentic workflows—where branching logic and dynamic prompt generation occur—SGLang often outperforms vLLM by 3x to 4x due to its specialized handling of agentic state.