The Problem with Unchecked Self-Evolution

Many LLM agent frameworks improve performance by iteratively refining natural-language artifacts—such as playbooks, strategies, or prompts—without updating model weights. While effective in specific benchmarks, these methods often suffer from high variance and safety issues. Without a mechanism to validate these refinements, agents can easily "drift" into poor performance, as seen in methods like Dynamic Cheatsheet, which performs well on some tasks but collapses on others like WebShop (scoring 0.14 vs 0.43 for the base ReAct agent).

RSEA: A Monotone-Safe Architecture

Recursive Self-Evolving Agents (RSEA) address this instability by introducing a strict "keep-better" gate. RSEA maintains a three-layer natural-language state:

  1. Imperative Strategy: High-level guidance for the agent.
  2. Reusable Skills: Modular task-specific knowledge.
  3. Procedural Playbook: Step-by-step execution logic.

Across generations, the agent rewrites these layers based on its own performance trajectories. Crucially, a candidate artifact is only committed if it demonstrates non-regression on a disjoint held-out dataset. This ensures that the agent never performs worse than the base model; if an evolution attempt fails the validation gate, the agent simply reverts to its previous state or the vanilla ReAct baseline.

Performance and Trade-offs

Evaluation across four benchmarks (ALFWorld, GAIA, τ-bench, and WebShop) against six baselines (including ReAct, Reflexion, and AWM) reveals three key insights:

  • No Universal Winner: While RSEA is the strongest single-pass method on ALFWorld (69.3% vs 64.6% for ReAct), it is not a silver bullet. For specific tool-use tasks, concrete-workflow induction methods like AWM remain superior.
  • The Necessity of Validation: The study confirms that "unguarded" context evolution is inherently unsafe. The performance gap between RSEA and methods lacking a held-out gate highlights that validation is the primary factor in achieving stable, recursive improvement.
  • Safety First: RSEA’s primary contribution is not just peak performance, but reliability. By enforcing strict held-out selection, it guarantees a performance floor, effectively neutralizing the risk of catastrophic forgetting or context degradation during self-evolution.