The Evolution of Reasoning and RL

Ross Taylor argues that the transition from base models to useful AI products was driven by Reinforcement Learning from Human Feedback (RLHF). Drawing on his experience with Galactica, he notes that while Galactica outperformed larger models like Chinchilla and GPT-3.5 on scientific benchmarks, it lacked the post-training polish that made ChatGPT a product. The core lesson is that a strong base model is a prerequisite, but RL is the mechanism that unlocks reasoning.

He highlights that "thinking tokens"—internalizing the reasoning process within special tags—was a key, early insight for enabling models to perform inference-time computation. The recent success of models like OpenAI's o1 is attributed to the "bitter lesson": the combination of superior base models, massive RL compute, and larger context windows creates emergent reasoning capabilities.

The Long-Horizon Mindset

Chengxi Taylor defines long-horizon tasks not as a benchmark, but as a mindset. Solving complex, multi-year problems (like scientific breakthroughs) requires AI to operate over sequences far longer than current context windows allow.

To manage these horizons, the team proposes:

  • Compaction: Summarizing long trajectories to fit within context limits, which can be optimized via RL.
  • Value Models (Critics): These are essential for reducing gradient variance and solving credit assignment problems in sparse-reward environments. By bootstrapping—generating expectations before an episode ends—models can learn without waiting for a final reward.
  • Infrastructure: Using tools like scratch pads, self-search, and file systems allows agents to manage state externally, though this introduces the risk of the model "cheating" by retrieving answers rather than reasoning.

Trade-offs in Compute and Simulation

Scaling to long horizons creates a conflict between GPU utilization and off-policy staleness. In traditional pipeline RL, waiting for long sequences to finish leads to GPU idle time. While off-policy training (up to ~8 steps) is generally acceptable, longer horizons force a choice: either leave GPUs idle or accept the bias introduced by bootstrapping with a value model.

Furthermore, current benchmarks are criticized for being too focused on procedural, coding-heavy tasks. The authors argue that frontier models struggle with real-world complexity because current environments lack true open-endedness and multi-agent simulation. Their "Kelly Bench" experiment, where models failed to trade football matches profitably, demonstrated that models lack the ability to handle the uncertainty and competitive dynamics of real-world environments.