The Shift to Agentic Development

Traditional software engineering relies on deterministic code where outcomes are predictable. Agentic systems, however, function like machine learning models: they are stochastic, black-box artifacts. To build reliable agents, developers must move away from traditional unit testing and adopt an ML-centric workflow. This means treating agent performance as a metric to be optimized through empirical evaluation rather than manual code review.

The 'Rollout' Framework

A 'rollout' is the atomic unit of agent evaluation. It consists of three components:

  1. Environment: A sandboxed virtual computer where the agent executes tasks.
  2. Instruction: A clear definition of the goal.
  3. Verifier: A programmatic test, rubric, or secondary agent that grades the outcome.

By running these rollouts in parallel, developers can generate large datasets of agent trajectories. This data becomes the foundation for optimization, allowing teams to perform 'agentic map-reduce'—running thousands of agents across distributed sandboxes to aggregate results, detect reward hacking, or process complex workflows like PR analysis or receipt reconciliation.

Practical Application: The Harbor Approach

Harbor provides a standardized format for these environments, enabling interoperability across the industry. The workflow for optimizing an agent follows a loop similar to ML training:

  • Evaluation: Run agents against specific tasks (e.g., internal codebases, product usage scenarios) to establish a baseline.
  • Feedback: Use the verifier to produce rewards or text-based feedback.
  • Optimization: Use the feedback to inform prompt engineering, model selection, or fine-tuning (SFT/RL).

This approach allows companies to move beyond 'brand trust' in models and instead make data-driven decisions about the cost-performance trade-offs for their specific internal use cases. Whether evaluating how well an agent builds your product or how well it uses your product, the key is to start with the evaluation (the 'eval') that matters most to your business outcomes.