Shift from Benchmarks to Production Testing
Most public discourse on AI evaluation is stuck in the 'chatbot era,' focusing on static datasets that break whenever a model or tool harness changes. Instead of chasing benchmark scores, treat agent evaluations like standard software tests (unit and end-to-end). Use tools like vitest or similar harnesses to run tests locally as code. This approach is more resilient to the rapid pace of change in the AI ecosystem and avoids the 'eval theater' that delays product shipping.
The 'Floor Raising' Framework
When building agents, distinguish between the 'ceiling' (the agent's peak capability) and the 'floor' (the worst thing it can do, such as leaking data or hallucinating harmful actions). User trust is built by raising the floor, not just chasing capability. To effectively manage this, you must track two specific metrics for every issue:
- Temporal Origin: When did this issue start? Knowing if an issue is new helps you correlate it with recent code or model changes.
- User Impact: What percentage of users are affected? This allows you to prioritize fixes based on scale rather than reacting to every single trace.
Tactical Lessons for Issue Discovery
- Avoid Naive Clustering: While clustering traces is useful for one-off analysis, it fails at scale. It lacks temporal context and often groups issues with different root causes together, making it difficult to track regressions over time.
- Use Code-Based Classifiers: Apply the 'code mode' pattern to your traces. Write deterministic classifiers and run them in sandboxes at production scale. This provides more control than automated clustering.
- Don't Ask Agents to Find Anomalies: Agents are poor at detecting anomalies. Instead, use deterministic methods (like tracking keyword frequency spikes) to identify potential issues, then use the agent to investigate those specific, tractable anomalies.