The Distillation Spectrum

To implement continual learning, Applied Compute categorizes distillation into a 2x2 grid based on two axes: the source of the production traces (offline vs. online) and the source of the hints (offline vs. online).

  • Offline Traces: Uses static batches of historical production data. This is the entry point for enterprises to improve agents without needing a live, replayable environment.
  • Online Traces: Integrates inference and training into a unified engine, where the model learns from its own live rollouts. This represents the "holy grail" of continual learning.
  • Offline Hints: Uses static rubrics or general priors (e.g., "don't give refunds so easily") to guide the model.
  • Online Hints: Dynamically constructs hints based on the specific behavior observed in the current online rollout.

Implementation Strategies

Applied Compute focuses on two primary quadrants to drive value:

Quadrant 1: Offline Traces + Offline Hints

This approach allows for immediate value by taking existing production logs and nudging the model toward specific behaviors. For example, in SWEBench, the team successfully trained an agent to call a 'submit' tool before turn 40 by using a hint that warned the model about its turn limit. Crucially, this improved the task completion rate (from 22% to 60%) without degrading the base test pass rate, even though the original traces lacked the specific reasoning path for the tool call.

Quadrant 4: Online Traces + Online Hints

This is the most scalable solution for raising performance ceilings. By generating hints dynamically based on the model's live performance, the system can adapt to specific edge cases. For instance, when teaching an agent to use a specific, out-of-distribution hyperlink format, online hinting improved accuracy from 15% to 80%, significantly outperforming offline hinting methods which struggled to adapt to the nuance of the harness.

Tips for Effective Distillation

  • Per-step Hinting: Rather than applying a hint to an entire rollout, use an LLM judge to identify the specific moment in time where the teacher should intervene. Distillation is most effective on the immediate next steps following the hint.
  • Relevance Masking: Use an LLM judge to sample and select which tokens to learn from. This prevents the student model from picking up irrelevant stylistic preferences (like connector words) from the teacher model, which helps avoid catastrophic forgetting or degradation of base capabilities.
  • Avoid 'Golden' Rubrics: The most effective continual learning systems do not rely on having a perfect, pre-existing answer key for every task. Instead, they focus on nudging behavior through targeted hints.