Enhancing Cache Efficiency for Persistent Agents

GPT-6 introduces a refined prompt caching system designed to support persistent agents that maintain context over long sessions. By reusing shared prefixes—such as system instructions, tool definitions, and reference material—developers can significantly reduce latency and costs, with discounts reaching up to 90% for cached input tokens. The system now supports a 30-minute window for eligible shared prefixes, providing more flexibility for complex, multi-turn workflows.

Monitoring and Diagnostics

To help developers maintain high cache hit rates, OpenAI has released two primary tools:

• Prompt Caching Dashboard: Provides visibility into cache performance, allowing developers to track hit rates over time and analyze the composition of cached versus uncached tokens.
• Diagnostics Tool: Enables developers to investigate specific cache misses. By comparing a request against recent history, the tool identifies the exact cause of a miss—such as changes in model settings, tool definitions, or input structure—and quantifies the impact in terms of missed tokens.

Strategies for Cache Optimization

Developers can maximize cache efficiency through several architectural patterns:

• Append-Only Updates: To preserve cache, keep tool definitions and schemas stable. Instead of removing tools, use allowed_tools to restrict access or set tool_choice to none. For new instructions, append them to the end of the context rather than modifying earlier segments.
• Reasoning Flexibility: GPT-6 allows developers to adjust reasoning effort mid-conversation using configuration_update without invalidating the cache, enabling dynamic performance tuning for routine versus complex tasks.
• Prewarming: Applications can "prewarm" the cache during startup by sending known context (like system prompts or reference data) to the API before the user initiates a request, effectively eliminating cold-start latency for the end user.