The Reasoning Model Trap

When migrating a voice-AI pipeline from Claude Sonnet to OpenAI's gpt-5-nano, the author encountered a silent failure: the API consumed 512 output tokens but returned an empty string. This occurred because reasoning models use max_completion_tokens to cover both internal "thinking" and final output. If the budget is set too low, the model exhausts its tokens during the reasoning phase, leaving nothing for the actual answer. For retrieval-based Q&A tasks, reasoning chains are unnecessary overhead that increase latency and cost.

Optimizing for Task-Specific Performance

The author successfully migrated to gpt-4.1-nano, a non-reasoning model. This change reduced LLM generation time from 7.0s to 3.1s and total pipeline latency (STT → RAG → LLM → TTS) from 10s to 6s. The key insight is that "newest" does not mean "best"; reasoning models are designed for complex planning and multi-step problem solving, whereas retrieval-style tasks benefit from the speed and efficiency of smaller, non-reasoning models.

Migration Playbook and Gotchas

Migrating between LLM providers is rarely a drop-in replacement. The author suggests building a mapping table for API differences before writing code, specifically focusing on:

• Streaming: Anthropic uses an async context manager (messages.stream()), while OpenAI uses a flag-based pattern (stream=True).
• Usage Retrieval: OpenAI requires stream_options to track tokens during streaming.
• System Prompts: Anthropic uses a top-level parameter, while OpenAI embeds system prompts within the message list.
• Dependency Management: The author warns that pydantic-settings 2.7.1 introduced a breaking change by defaulting extra to forbid, which caused silent failures in test suites. Pinning minor versions and reading changelogs is essential to avoid production-breaking surprises during minor dependency updates.