The 'Replay Agent' Exploit

Many current computer use benchmarks are deterministic, allowing them to be gamed by "replay agents." These are scripts—often under one megabyte—that simply record a successful trajectory once and replay it blindly. Because these benchmarks lack variation, these scripts frequently match or outperform the frontier models they were derived from. The metric pass@k is particularly problematic in these environments, as it effectively functions as a formal measure of this replay exploit rather than a measure of agent intelligence.

The PRISM Principles for Robust Environments

To move beyond static, hackable benchmarks, environments must be designed with the PRISM principles:

  • Privileged Verification: Use ground-truth data to verify outcomes.
  • Realism: Ensure the environment is a faithful reproduction of real-world systems.
  • Integrity-Checked Configurations: Validate that every generated variation is functional.
  • Sandboxed Execution: Isolate the agent's actions.
  • Multifactorial Variation: Introduce stochasticity across data, themes, and starting states.

Pierluca D'Oro introduces DIGIWORLD as an implementation of these principles, featuring 15 mobile apps and 3.2 million verified configurations. The key to scaling this is a compiler-like system that generates combinations and automatically rejects broken configurations, ensuring that the benchmark remains both diverse and valid.

Honest Uncertainty and Financial Impact

Even with a robust environment, evaluation metrics often fail due to "overconfident" statistics. Naive rollouts on a single base case typically yield confidence intervals that cover the true performance only 20% of the time, rather than the expected 95%.

This lack of rigor has direct financial consequences. In a scenario with one million tasks, a 4% performance gap between two models—hidden by tight but inaccurate confidence intervals—can result in hundreds of thousands of dollars in losses per month. A rigorous evaluation methodology must account for two sources of variance: the model's stochastic actions and the environment's inherent variability. When uncertainty is measured honestly, the data may reveal that a model's performance is statistically indistinguishable from another, preventing costly, premature deployment decisions.