The Case for Voice-In, Visuals-Out

While voice-in/voice-out interfaces often suffer from high latency and awkward interactions, "voice-in, visuals-out" offers a more forgiving and effective UX. Humans process visual information rapidly, and modern LLMs are now capable of generating rich HTML, interactive controls, and illustrations that communicate complex responses faster than audio can. This approach leverages the high-bandwidth nature of human speech for input while using the visual cortex for output, effectively bypassing the extreme 200ms latency requirement needed for natural conversational audio.

Achieving the 1-Second Latency Envelope

To make an AI agent feel responsive, the system must react within a 1-second window. Exceeding this causes users to lose their train of thought. To achieve this, developers should adopt three core technical strategies:

  • Prioritize Model Speed over Parameter Count: Use "Haiku-class" models that prioritize low P95 latency. If a complex task is required, use a fast, small model for the immediate interaction and offload heavy reasoning to a larger model asynchronously, interleaving the results once they are ready.
  • Eager Inference: Avoid waiting for natural pauses or silence in speech. Instead, trigger inference every 1–2 seconds while the user is still speaking. This creates a sense of seamless, real-time engagement where the agent begins acting on intent before the user has finished their sentence.
  • Aggressive Prefix Caching: Leverage platform-specific prefix caching to reuse the first 90% of your context window. By keeping the majority of the prompt consistent across requests, you significantly reduce inference time and cost, allowing the model to focus only on the final 10% of the input.

Architectural Trade-offs

Building for low latency requires moving away from traditional "listen-then-process" architectures. While novel architectures like time-sliced 200ms inference chunks exist for voice-only systems, the "voice-in, visuals-out" pattern is more practical for most product teams. It allows for a higher-quality experience by utilizing the visual interface to provide feedback, which feels natural even if the underlying model takes nearly a full second to process the request.