The Problem of Skill Overfitting in Self-Evolution

Self-evolution—where AI models iteratively improve their own performance through self-generated data or feedback—is a powerful mechanism for scaling capabilities. However, a critical failure mode is 'skill overfitting.' As models focus intensely on optimizing specific task-based objectives, they often lose the breadth of their original training distribution. This leads to a degradation in general reasoning or adaptability, effectively 'narrowing' the model's intelligence to satisfy the immediate optimization loop.

A Constrained Exploration-Exploitation Framework

The authors introduce a framework designed to manage the trade-off between refining existing skills and maintaining general performance. Instead of allowing unconstrained optimization, the process imposes structural constraints on the exploration phase. By treating self-evolution as a constrained optimization problem, the model is forced to explore new task variations or data distributions while remaining within a 'trust region' of its original, generalized capabilities. This prevents the model from drifting too far into specialized niches that compromise its foundational knowledge.

Balancing Refinement and Robustness

The core insight is that exploitation (refining known successful strategies) must be strictly coupled with exploration (testing the boundaries of the model's current knowledge). By implementing a constrained feedback loop, the system ensures that performance gains on specific benchmarks are only accepted if they do not result in statistically significant regressions across a broader set of control tasks. This approach effectively treats general capability as a regularization constraint, ensuring that as the model evolves, it retains the versatility required for diverse, real-world applications.