The Evaluation Bottleneck

Building production-grade AI agents—especially for high-stakes environments like banking—requires rigorous evaluation. Traditional methods for gathering eval data are either manual (time-consuming, difficult to maintain state) or rely on production traces (slow, risky, and sparse). For multi-turn agents, this creates a massive bottleneck that prevents rapid experimentation and iteration.

Simulation as a Force Multiplier

Nubank and Snowglobe advocate for "SimulationMaxxing": using synthetic, grounded data to stand in for real-world interactions. By pointing a simulation SDK at an agent, developers can generate thousands of multi-turn conversations featuring specific personas (e.g., a 34-year-old designer ordering a credit card).

Key benefits of this approach include:

  • Grounded Context: Simulations provide consistent, mocked account data and tool outputs, ensuring the agent is tested against realistic scenarios.
  • Short-Circuited Cycles: The time to test changes drops from days (waiting for human curation or production feedback) to hours or minutes.
  • Risk Mitigation: Simulations allow teams to catch regressions and performance issues before they ever reach a live customer.

Closing the Sim-to-Real Gap

To make simulations reliable, teams must validate them against production data. Nubank found that 80% of simulated conversations were comparable to real ones, providing high-quality signal for both greenfield development and improving mature agents.

This creates a self-improving loop:

  1. Ship/Observe: Deploy the agent and gather traces.
  2. Simulate: Generate synthetic data to test variants and edge cases.
  3. Eval: Run automated metrics (LLM-as-a-judge) against both real and simulated data.
  4. Optimize: Use the feedback to tune prompts and agent harnesses.

This framework allows teams to test multiple open-source models and agent variants in parallel, launching only the most successful versions to production. This approach has enabled Nubank to double the TNPS (customer satisfaction) of their agents while simultaneously improving self-service rates.