Redefining Long-Horizon Performance

Measuring "long-horizon" work as a scalar metric (e.g., 16-hour human-equivalent tasks) is inherently noisy and prone to misinterpretation. Relying on human-time benchmarks or token counts creates a false sense of progress because these metrics fail to account for task complexity, model-specific efficiencies, and the divergence between human and agent capabilities. A task that is tedious for a human (like manual Excel formatting) may be trivial for an agent, while a task that is simple for a human might be impossible for an agent due to context window limitations or poor state coherence.

Environment Complexity and State Dependency

Theta Software argues that the true measure of an agent's capability lies in its ability to navigate environment complexity. They distinguish between two types of complexity:

  • Paralyzable Complexity: Tasks where sub-agents can work independently (e.g., analyzing multiple files in a codebase). These are easier to scale but don't test the agent's ability to handle cascading dependencies.
  • Sequential Complexity: Tasks where early decisions influence later outcomes (e.g., debugging a deployment failure). Here, a single misread of a log file cascades through the entire trajectory.

Effective evaluation requires environments that mirror real-world ambiguity, forcing agents to explore artifacts rather than following a rigid path. When environments are open-ended, standardized evaluation becomes significantly harder, necessitating a shift from deterministic tests to nuanced, agentic verification.

The Role of Agentic Judges

As tasks move into "software-fellow" domains, deterministic verifiers (like unit tests) become brittle or impossible to write. The solution is to use judge models that evaluate the final state of the environment rather than just the agent's trajectory.

Key principles for designing these judges include:

  • Judges as Agents: The judge must have the same observability as the agent, accessing logs and databases to verify the actual outcome of an action (e.g., checking if a PR actually fixed a deployment).
  • Queryable Trajectories: Because long-horizon trajectories are too large for a single LLM context window, they must be stored in databases and enriched with metadata, allowing the judge to query specific phases or failure points.
  • Avoiding Reward Hacking: Judges must be designed to prevent agents from accessing privileged information or hidden test suites. This requires strict read-only permissions for the judge to ensure it observes the environment without mutating it.
  • Iterative QA: Rubrics must be carefully calibrated. If a rubric is too dense or complex for the current frontier of models, the judge will fail to apply it consistently, leading to wasted compute and poor learning signals.

Key Takeaways

  • Stop measuring by time: Human-time estimates are noisy; focus on state-based outcomes and the agent's ability to handle sequential dependencies.
  • Design for state changes: A task is only "long-horizon" if early actions meaningfully impact the state of the environment for later steps.
  • Make trajectories queryable: Don't stuff entire logs into a context window. Use sub-agents to parse and enrich trajectory data so judges can find critical failure points.
  • Judges need environment access: A judge cannot verify a task by looking at the agent's output alone; it must be able to inspect the environment (logs, DBs, CI/CD) to confirm success.
  • Prioritize learnability: If your reward signal is too sparse or your rubric is too complex for the model, you are wasting compute. Use partial credit or dynamic rubrics to guide learning.