The Limitation of Turn-Taking
Modern voice agents, including many speech-to-speech models, operate on a 'half-duplex' basis. They function like walkie-talkies: the system is either listening or speaking, never both. This architecture forces an artificial turn-taking structure that breaks human conversational norms, such as 'back-channeling' (using verbal cues like 'mm-hmm' or 'yeah' to show engagement). In human-to-human phone calls, participants speak over each other up to 20% of the time. Current models that segment conversation into strict 'listen' or 'speak' phases fail to handle these overlaps, leading to unnatural, stuttering interactions where the model stops dead if interrupted.
The Technical Challenge: Audio vs. Text
Raw audio is computationally expensive to process within an LLM due to its high sampling rate. A 3-second audio clip at 24 kHz contains 72,000 timesteps. Because attention mechanisms in transformers have quadratic complexity relative to sequence length, processing raw audio is orders of magnitude more expensive than text. To solve this, developers use neural codecs (audio tokenizers) to compress audio into dense, token-like representations that an LLM can process. However, even with these codecs, most models remain trapped in a sequential 'user-then-system' token stream, which enforces the half-duplex limitation.
The Tension Between Naturalness and Intelligence
There is a fundamental trade-off between naturalness and reasoning capability. A model has a fixed 'weight budget' (capacity). When a model is tasked with both high-fidelity audio processing and complex reasoning, it must divide its capacity between the two. Consequently, current speech-to-speech models often exhibit lower reasoning intelligence than cascaded systems (which route audio through separate speech-to-text, LLM, and text-to-speech modules).
Two Paths Forward
To evolve voice agents, the industry is pursuing two distinct strategies:
- Scaling Speech-to-Speech: This involves training larger, more robust frontier models that handle both audio and reasoning natively. While this creates a single, easy-to-deploy system, it is computationally expensive and ties the user to a specific model provider, limiting the ability to swap back-ends as new, smarter models emerge.
- Splitting Interface from Intelligence: This approach uses a small, full-duplex, on-device model to handle the natural conversation interface (the 'voice'). This interface asynchronously delegates complex reasoning, tool-calling, and planning to a separate, powerful background text LLM. This architecture is more economically viable, allows for model swapping, and ensures the 'voice' remains responsive and natural regardless of the complexity of the task being performed in the background.