The Limitations of Current RL Paradigms

Modern AI training relies heavily on benchmarks that are becoming increasingly expensive and time-consuming to scale. Current methods like GRPO (Group Relative Policy Optimization) require massive parallel infrastructure to generate on-policy rollouts and rely on sparse, sequence-level rewards. This creates a "trillion-token problem" where models fail to learn from the vast amounts of real-world inference data generated daily. The core challenge is moving toward a system that features online task distributions, on-policy sampling, minimal infrastructure requirements, and dense, per-token feedback.

On-Policy Self-Distillation (OPSD)

OPSD addresses these bottlenecks by replacing fixed datasets with student-generated rollouts. The algorithm functions by providing a "hint" (privileged information) to a teacher model, then training the student model to match the teacher's log probabilities.

Key advantages include:

  • Infrastructure Efficiency: Unlike GRPO, which requires parallel rollouts, OPSD works with single-example trajectories, removing the environment bottleneck.
  • Dense Feedback: By matching log probabilities at the token level rather than the sequence level, the model receives rich, granular feedback across the entire vocabulary.
  • Distribution Shifting: OPSD doesn't just sharpen existing distributions; it shifts them, allowing models to explore new solution spaces and improve token efficiency.

Scaling Challenges and Solutions

As models scale (e.g., 120B+ parameters) and tasks become more complex (e.g., 100+ tool calls), OPSD faces two primary failure modes: the "but-wait" problem and hint leakage.

  • The "But-Wait" Problem: In long-horizon tasks, the student and teacher distributions diverge, causing the model to collapse into suboptimal, repetitive hedging (e.g., "but," "maybe," "wait"). This is mitigated by step-level divergence weighting, where the KL divergence between student and teacher is used to dynamically weight tokens. This allows the model to ignore off-track segments while focusing on productive reasoning steps.
  • Hint Leakage: If the hint provides the answer directly, the model "shortcuts" its reasoning process. This is solved via residual guidance, where the model is trained on a linear combination of partial and full hints. This prevents the model from shifting into unknown territory by ensuring the guidance remains close to the model's current distribution.

By combining these techniques, OPSD can surpass traditional RL performance in agentic tasks, providing a path toward systems that improve automatically with every real-world interaction.