From Model Serving to Distributed Orchestration

AI inference is evolving rapidly, mirroring the 2008 transition of cloud computing from simple virtual machines to complex, orchestrated ecosystems. While early AI efforts focused on model kernels and serving frameworks (like vLLM), the current bottleneck is the orchestration layer. Unlike traditional microservices, which are largely stateless and uniform, modern agentic AI workloads involve complex, stateful distributed transactions where a single request may trigger thousands of model calls.

The Seven Axes of Inference Scheduling

Scheduling for inference requires a multi-dimensional approach that traditional microservice schedulers cannot handle. A robust scheduler must be aware of seven critical axes:

  • GPU Generation: Managing heterogeneous hardware (e.g., H100 vs. A100).
  • Memory Headroom: Tracking HBM availability.
  • KV Cache State: Managing expensive, long-lived per-request state.
  • Weight Warmth: Distinguishing between cold-start models and ready-to-serve weights.
  • Tenant Priority: Balancing multi-tenant SLO profiles.
  • Latency Budget: Aligning with specific agentic application requirements.
  • Workflow Context: Understanding the request's position in a multi-step chain to prevent wasting compute on doomed workflows.

The Optimization Framework: Avoid, Share, Move, Delay

Every optimization technique fits into one of four quadrants. The goal is to optimize for cost per successful task rather than cost per token:

  • Avoid: Use prefix, response, or semantic caching to skip redundant work.
  • Share: Utilize continuous batching, chunked prefill, and speculative decoding to maximize GPU utilization.
  • Move: Route requests to smaller models or cheaper regions based on complexity.
  • Delay: Implement admission control and deadline-aware queuing to manage traffic spikes.

Reliability as a Control Plane Property

Inference systems are prone to cascading failures, particularly due to the "KV cache twist": if a GPU host fails, you cannot simply retry the request elsewhere because the state (KV cache) is tied to the original host. Reliability must be built into a centralized control plane that acts as a feedback loop. This control plane must implement:

  • Circuit Breakers: At the routing layer to prevent traffic from hitting degraded nodes.
  • Load Shedding: Tied to queue depth rather than simple CPU/memory metrics.
  • Retry Budgets: To prevent runaway costs during regional failures.

The Latency-Cost-Throughput Triangle

Every serving decision involves trade-offs analogous to the CAP theorem. Increasing batch size improves throughput but hurts tail latency; speculative decoding improves latency but increases compute costs. The platform's role is to act as a control loop that constantly adjusts these variables based on real-time telemetry, treating models, GPUs, and KV cache as schedulable resources rather than static endpoints.