The Optimization Triangle

Building production-ready AI requires balancing three competing factors: accuracy, performance (latency), and cost. You can typically optimize for two, but the third will inevitably suffer. Leaderboard scores are insufficient because they measure isolated model performance, whereas production environments test the entire system under specific, often unpredictable, traffic patterns.

Model vs. System Evaluation

Effective evaluation requires a two-pronged approach:

  • Model Evaluation: Measures reasoning and accuracy. Standardized benchmarks (e.g., MMLU) work for general knowledge, while execution-based benchmarks (e.g., SWE-bench) test coding capabilities by running actual tests. For open-ended tasks without a single "correct" answer, use LLM-as-a-judge patterns. This involves using a powerful model to critique outputs against a rubric, scaled by human-in-the-loop annotations to ensure the judge aligns with domain-expert standards.
  • System Evaluation: Measures infrastructure performance. Key metrics include Time to First Token (TTFT), inter-token latency, request latency, and throughput. Inference performance is split into two phases: pre-fill (compute-heavy, processes the prompt) and decode (memory-heavy, generates tokens). Because these phases have different bottlenecks, you must benchmark using token distributions that mirror your actual application (e.g., chat vs. RAG vs. coding agents).

Evaluating Agentic Workflows

Agents introduce complexity because they are chains of decisions rather than single model calls. Each step—intent recognition, tool selection, retrieval, and code execution—is a potential failure point.

To evaluate agents, use a layered pyramid approach:

  1. Base System: Can the infrastructure scale?
  2. Formatting: Is the output structured correctly for downstream consumption?
  3. Safety & Bias: Are guardrails preventing PII leaks or harmful content?
  4. Factual Accuracy: Is the model grounded in real-world data?
  5. Domain-Specific Logic: Does the agent solve the specific business problem?

Teams often fail by starting at the top of the pyramid. Instead, establish Service Level Objectives (SLOs) for your specific traffic patterns and identify the "inflection point" in your latency-throughput curve to determine your system's true capacity before it hits production.