The Shift in Context Management

For years, developers have treated context window limitations as a problem to be solved through aggressive compaction—summarizing history, truncating tool outputs, and pruning logs. However, the rise of prompt caching in 2026 has inverted the economics of context. Because cached tokens are significantly cheaper (up to 50x) and faster to process, any compaction technique that rewrites the context invalidates the cache, forcing the model to recompute the entire prompt at full price.

The "Do Nothing" Strategy

In a series of experiments conducted on an open-source AI tutor, the team at Towards AI compared 11 different context management presets. The surprising result was that doing nothing—simply passing the raw, uncompacted history—consistently outperformed all other methods.

  • Recall: Keeping full history maintained 95% recall on specific details, compared to 32% when using summarization.
  • Cost & Latency: Because the raw history remained in the prompt cache, the system avoided the overhead of generating summaries and the cost of re-processing tokens.
  • Rot: Distinctive facts remained retrievable even at 800,000 tokens, provided the model was not forced to "compress" the information.

When Compaction Actually Fails

Compaction is often a "trap" because it assumes the model's primary constraint is the context window size. In reality, the constraints are often cost and latency.

  • Dense Retrieval vs. BM25: While dense retrieval is popular, the team found that in long-context scenarios (400k+ tokens), dense retrieval recall dropped to 0% for buried facts, whereas traditional BM25 keyword search remained reliable.
  • The Cache Penalty: For compaction to be economically viable, it must shrink the context by more than 50x to offset the loss of the prompt cache. Most summarization techniques fail to achieve this level of compression without significant information loss.

Architectural Recommendations

Instead of blanket compaction, the team recommends a strategy of progressive disclosure and modular skills:

  1. Offload, Don't Compact: Use RAG or an "LLM Wiki" to offload data to external files. This keeps the active context small while ensuring information is fully reversible and inspectable.
  2. Optimize for Cache Hits: Structure your agent prompts to keep static information (system instructions, tool definitions) at the beginning of the prompt to maximize cache reuse.
  3. Log Everything: Use observability tools to track cache hit rates and user frustration. If a user is unhappy, the system should be able to identify if it was due to context rot or poor retrieval.
  4. Name the Constraint: Before applying a technique, define the specific bottleneck. If the bottleneck is cost, prioritize caching. If the bottleneck is recall, prioritize better retrieval (like BM25) over summarization.

Key Takeaways

  • Prompt caching is the primary optimization: Avoid any technique that invalidates your cache unless the compression ratio is massive (50x+).
  • Summarization is often a net negative: It destroys recall and increases costs by forcing re-computation of tokens.
  • Hybrid search is essential: Do not rely solely on dense embeddings; combine them with BM25 for better fact retrieval in long-context documents.
  • Use progressive disclosure: Only load the specific "skills" or context chunks necessary for the current turn rather than keeping a massive, bloated state.
  • Measure before optimizing: The team's experiments showed that their production defaults were actually worse than the simplest possible implementation.