The Shift from Turn-Based to Continuous Streaming
Traditional voice AI systems relied on turn detectors—small models that guessed when a user finished speaking before triggering the LLM. This introduced inherent latency and "sluggish" responses. GPT-Live replaces this with a full-duplex, streaming architecture where audio flows continuously in and out of the model. By removing the turn detector from the critical path, the system sustains an uninterrupted media loop, allowing for natural, human-like conversation.
Decoupling Media from Logic
To maintain low latency, the architecture enforces a strict separation between the "live" media path and application/business logic:
- Media Frontend: Built in Go (replacing Python
asyncio), this layer handles audio transport via WebRTC. It prioritizes frame delivery, using WebRTC's ability to stretch or accelerate audio to handle packet loss and clock drift without audible gaps. - Asynchronous Delegation: Deeper reasoning, tool use, and frontier model calls (e.g., GPT-5.5) occur on an asynchronous RPC path. This ensures that slow backend processes cannot stall the real-time audio stream.
- Stateful Inference Management: To handle long-running conversations, the system uses a handoff mechanism. It warms up replacement model instances, prefills them with current context, and performs a seamless cutover, ensuring the conversation never misses a beat during context compaction or model transitions.
Optimizing the Startup and Delegation Path
Responsiveness begins at the first user interaction. OpenAI developed the WebRTC Abridged Roundtrip Protocol (WARP) to reduce the handshake process from six network round trips to one. Combined with Instant Connect—which pre-negotiates session parameters without reserving server capacity—the system can initiate a live session with a single UDP packet.
For delegation, the system pre-warms frontier model sessions and uses stable session affinity. This minimizes the time until a frontier model can contribute to the conversation. Meanwhile, the application server derives discrete turns from the continuous audio stream for UI and analytics purposes, maintaining two views: a speculative view for the UI and an authoritative, finalized record for logging.
Production Validation
OpenAI validated the system using a "silent test" (shadow mode), routing production traffic to both the existing system and the new architecture. This revealed that capacity planning must account for CPU-side stream handlers and network paths, not just GPU throughput. The testing process highlighted the importance of geographic proximity for inference and forced the development of more granular telemetry and staged rollout controls to manage failures in real-time.