Evaluating Agentic Systems Without Ground Truth

The core challenge addressed in this research is the difficulty of evaluating agentic systems—specifically those operating in complex environments like information security—where traditional labeled datasets are often unavailable or insufficient. The authors propose a methodology that shifts the focus from static label-based evaluation to a dynamic assessment of learning harness capabilities. By applying the scaling hypothesis, the researchers demonstrate that agentic performance can be predicted and measured based on the relationship between compute, model size, and task complexity, even in the absence of explicit ground-truth labels.

The Scaling Hypothesis as an Evaluation Framework

The paper argues that agentic learning follows predictable scaling laws. Instead of relying on human-annotated datasets, the authors utilize the internal consistency and task-completion efficiency of the agent as a proxy for capability. By observing how an agent's performance improves as it is allocated more compute or as the underlying model scales, researchers can infer the agent's learning potential and robustness. This approach is particularly relevant for security-focused AI, where agents must adapt to novel, adversarial environments that cannot be fully captured in a static training set. The findings suggest that scaling behavior provides a reliable signal for assessing whether an agent is effectively 'learning' to navigate a task or merely memorizing patterns, offering a path toward more scalable and automated evaluation pipelines for autonomous systems.