The Measurement Problem: Tokens vs. Outcomes
Agents are currently suffering from a "measurement problem" where early adopters are biased and token usage is often untethered from business value. The current cycle involves "token maxing" into austerity, followed by dropping out, and eventually returning due to FOMO.
To break this cycle, builders must stop treating tokens as the primary metric. Tokens are an output, not an outcome. Value should be measured by tangible results—such as bugs closed or support tickets resolved—rather than the volume of compute consumed. The current bottleneck in AI-assisted workflows is often human verification; while agents can generate code or designs rapidly, the human review process has not scaled, making it difficult to judge the quality of the output at the speed of production.
The Entropy Matrix: A Framework for Task Selection
To determine which tasks are suitable for agents, builders should use an "entropy matrix" based on two axes of uncertainty:
- Step Uncertainty: How predictable is the path to completing the task?
- Criteria Uncertainty: How clear is the rubric for verifying that the task was done correctly?
- Low Step Uncertainty: If the steps are highly predictable, do not use an agent. Write a script instead.
- High Step Uncertainty: If the task is too unpredictable, it likely falls out of the model's distribution and will result in sparse rewards.
- High Criteria Uncertainty: If verifying the work is as difficult as doing the work, the agent provides no net efficiency gain.
The Sweet Spot: Verification-First Design
The ideal tasks for agents are those that resemble NP-style problems: they are computationally expensive to solve but relatively easy to verify. When a task has a repeatable verification pattern, the most effective strategy is to build a secondary agent specifically tasked with verifying the output of the primary agent. This approach allows for "measurement at the speed of compute," effectively creating a modern equivalent of James Watt’s "horsepower"—a legible metric that helps users justify the ROI of adopting agentic workflows.