The Latency-Intelligence Trade-off

Voice agents operate under a strict "thinking budget" that renders most recent LLM advancements—specifically those relying on extended chain-of-thought or reasoning—useless. While frontier models (GPT-4, Claude) are powerful, their P90 latency often exceeds 1.2 seconds, causing users to hang up. To achieve a production-grade experience, the target must be under 300ms.

Venky B recommends hosting smaller, open-source models like Qwen 3.5 or Gemma 4. The choice of model should be driven by "token fertility"—the number of tokens required to generate a single word in a specific language. Gemma 4, for instance, is significantly more efficient for multilingual use cases. For most applications, a 3B-4B parameter Mixture of Experts (MoE) model provides sufficient intelligence out of the box without the complexity of fine-tuning.

Rethinking Data Collection as Structured Input

Treating voice input as a raw transcript to be parsed by an LLM is a primary failure point. Instead, treat data collection as a form-filling exercise. By defining the "shape" of the expected data (e.g., a phone number, a date, or a specific proper noun) before the agent asks for it, you can apply validation logic similar to TypeScript interfaces or Python data classes.

This shift allows for field-level unit testing. If a transcription error occurs (e.g., an 'E' appearing in a phone number), the system can immediately identify it as a validation failure rather than attempting to interpret it as part of a larger conversation. This approach can boost accuracy from 30% to over 95% without requiring model fine-tuning.

Managing Transcription Brittleness

Transcription engines are inherently brittle, especially with proper nouns, code-switched languages, and noisy environments. Relying on a single, static transcription configuration is a mistake.

  • Dynamic Keyword Boosting: Instead of loading a massive list of keywords, inject context-specific keywords only when the agent is in a state where that information is expected.
  • LLM Post-Processing: Use an LLM to clean transcripts before they hit your core logic. An LLM can easily correct common transcription errors (like interpreting 'E' as '3' in a phone number) because it understands the domain context that the transcription engine lacks.
  • Normalization: Never feed raw LLM output directly into a Text-to-Speech (TTS) engine. Implement a normalization layer to strip emojis, markdown, and non-verbal artifacts that cause TTS engines to hallucinate or mispronounce text.

Key Takeaways

  • Target 300ms: Aim for sub-300ms latency to prevent user abandonment; frontier models are often too slow for production voice.
  • Use MoE Models: Small, open-source Mixture of Experts models (3B-8B parameters) are currently the best balance of speed and instruction-following capability.
  • Define Data Shapes: Treat every piece of information you need to collect as a typed field with specific validation rules.
  • Unit Test Fields: Evaluate your agent's performance at the field level rather than relying on end-to-end conversation tests.
  • Normalize TTS Input: Always strip markdown and emojis from LLM output before sending it to a TTS engine to ensure clean, professional audio.

Notable Quotes

  • "The irony with voice agents is like almost always your LLM or the agent that's talking has to have thinking turned off... so all the advancements we've had in the LLM layer in the last one year like none of that even apply here."
  • "Decide the shape before you ask. A phone number is a typed field with a length and a validator, so a stray letter in the middle is either corrected with confidence or sent back to the caller."
  • "If you just pollute your context of the transcription engine with tons of keywords it'll start hallucinating again."
  • "We've seen accuracy grow from 30% to like 95% from a data collection standpoint when you start thinking in that manner as typed fields."