The Shift Toward Training-Integrated Inference

Modern inference engineering is moving away from treating model weights as a static, finished product. Instead, the industry is adopting a cycle where faster inference enables more data collection, which in turn fuels better training, leading to further inference optimizations. This blurring of lines means that many of the most effective production optimizations now require a dedicated training phase rather than simple post-hoc adjustments.

Advancements in the "Big Three" Techniques

  • Quantization: While techniques like TurboQuant (using polar coordinates to quantize KV cache to 4-bits) gained viral attention for memory savings, they often fail in data center environments due to the high computational overhead during decode. Production systems currently favor NVFP4 quantization on model weights, combined with KV-aware routing and offloading to manage memory bottlenecks.
  • KV Compaction: To handle million-token sequences, systems are moving beyond simple caching. The "Still" approach from Baseten uses a learned bottleneck to cross-attend fixed query vectors against the full KV cache, emitting compact keys and values in a single forward pass. This creates a differentiable, compressed memory that LLMs can attend to with near-lossless retention.
  • Speculative Decoding: The field has evolved from using small draft models to using diffusion-based models like DFlash. DFlash generates sequences of 8–16 tokens simultaneously, significantly outperforming auto-regressive draft models. Furthermore, continuous speculator retraining—where the draft model is updated on live production prompts—can improve token acceptance rates by 20% to 2x, provided the infrastructure exists to handle the compute and data privacy requirements.

Local vs. Data Center Priorities

Inference engineering strategies diverge based on the environment:

  • Local Inference: The primary goal is to "get it working" on constrained hardware. Techniques like TurboQuant are highly valuable here because memory is the primary bottleneck, and the extra computation cost is often acceptable.
  • Data Center Inference: The goal is to "get it less slow." Optimization focuses on throughput, batching, and system-wide communication (e.g., KV cache disaggregation), where the latency cost of extra computation is often prohibitive.