The Misconception of the Conversational Agent

Most builders assume a voice agent's primary purpose is to talk back. This is a limiting design pattern. Instead of viewing voice as a chatbot, developers should view it as an interface layer. We have spent decades perfecting non-verbal UI patterns—notifications, state changes, button highlights, and cursor movements—that voice agents can trigger without needing to verbalize every action.

Three Modes of Voice Interaction

Charlie Guo categorizes voice interaction into three distinct, remixable modes:

• Speech-to-Speech: The traditional conversational model. Best suited for coaching, language learning, and concierge-style support where emotional nuance and tone are critical.
• Speech-to-Action: The most neglected and high-potential area. Here, the model uses tools to perform tasks. This is a "capability overhang" where voice can replace tedious manual work like filling out government forms or operating complex creative software (e.g., music or design tools) for users who lack technical skills.
• Event-to-Speech: The model monitors system events and proactively alerts the user. This is ideal for hands-free or screen-free contexts, such as cooking or driving, where the user's attention is diverted.

Implementation Strategy for Developers

Developers do not need to build new infrastructure to support these modes. Most modern web applications already expose their functionality through API endpoints and React hooks. These serve as the "verbs" that a voice model can call as tools.

To build effective voice-native applications, developers should move away from chained models (transcribe -> LLM -> text-to-speech) which suffer from high latency and loss of emotional context (tone, cadence, interruptions). Native audio models, such as OpenAI's Realtime family, process audio as tokens, allowing for lower latency and better reasoning. When using tool-calling, developers should implement "preambles"—short verbal cues that inform the user the agent is thinking or performing a background task—to maintain a natural, human-like interaction flow.

The Core Design Question

Instead of asking "What kind of voice agent should I build?", developers should ask: "What role does voice play in this specific interaction?" This shift in perspective helps determine whether the agent should be a conversational partner, a silent executor of tasks, or a proactive notifier, leading to more functional and accessible product design.