The Challenge of Online Knowledge Injection

Integrating new knowledge into Large Language Models (LLMs) via online distillation often triggers catastrophic forgetting, where the model loses previously acquired capabilities while learning new information. Traditional methods struggle to balance the stability of existing knowledge with the plasticity required to incorporate new data streams. RoCo-ACE (Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection) addresses this by treating the distillation process as a conditional optimization problem.

Rollout-Conditioned Distillation Mechanics

The core innovation of RoCo-ACE is the use of rollout-conditioned feedback to regulate the distillation process. Instead of applying a uniform update across all incoming data, the framework evaluates the model's performance on a "rollout"—a simulated or sampled sequence of tasks—before finalizing weight updates. This allows the system to identify which specific knowledge injections are likely to cause performance degradation on historical tasks.

By conditioning the distillation on these rollouts, the model dynamically adjusts its learning rate and objective function. If a proposed update threatens to degrade performance on high-retention-priority tasks, the framework attenuates the update or redirects the optimization path. This ensures that the model maintains a stable baseline of general knowledge while selectively absorbing new, task-specific information.

Impact on Model Stability

RoCo-ACE demonstrates that retention-aware mechanisms are significantly more effective than static regularization techniques. By incorporating performance feedback into the online loop, the model achieves:

  • Higher Retention Rates: Reduced performance decay on benchmark tasks compared to standard online distillation.
  • Adaptive Learning: The ability to prioritize critical information without requiring a full replay buffer of historical data, which is computationally expensive.
  • Improved Convergence: More stable training trajectories as the model avoids "catastrophic" weight shifts during the injection of noisy or conflicting new data.