The Hidden Impact of Evaluation Harnesses

Benchmarks like ARC-AGI-3 are often treated as objective measures of model intelligence, but they are heavily influenced by the 'harness'—the code that manages the model's interaction with the task. OpenAI found that GPT-5.6 Sol's initial poor performance (7.8%) on ARC-AGI-3 was not due to a lack of reasoning capability, but rather a result of a generic, suboptimal harness that discarded critical context.

The Two Settings That Triple Performance

To improve performance, OpenAI shifted from a legacy, stateless approach to their production-grade Responses API, enabling two specific settings:

  • Retained Reasoning: By default, many harnesses discard the model's 'private reasoning' messages after each action. This forces the model to re-interpret the game state from scratch every turn. Retaining these reasoning messages allows the model to maintain a continuous thread of logic, strategy, and insight, which significantly improves its ability to learn over time.
  • Compaction: Many benchmarks use 'rolling truncation,' which simply deletes the oldest messages when the context window is full. This causes the model to lose historical observations and actions. Compaction provides a more intelligent way to manage context, allowing the model to preserve essential information about the game across longer runs while using fewer output tokens.

Practical Recommendations for Developers

When building agentic systems, developers should avoid generic harnesses that treat each turn as a blank slate. To maximize performance and efficiency, the following stack is recommended:

  1. Use the Responses API: Move away from legacy Chat Completions APIs to gain better control over context management.
  2. Retain Reasoning: Ensure that the model's internal chain-of-thought is passed forward in the conversation history.
  3. Implement Compaction: Use native compaction features to manage long-running tasks efficiently, ensuring the model retains the most relevant information without hitting context limits or wasting tokens.