The Cascaded Voice Stack

Bohan Li frames the architecture of a voice agent using a self-driving car analogy:

  • Perception (Transcription): Converting real-world audio signals into structured data.
  • Planning (LLM): The "brain" that processes inputs and decides on actions.
  • Control (Speech Synthesis): Converting text back into audio for the user.

By decoupling these layers, developers can optimize each for speed and accuracy without forcing the entire system to wait for the slowest component.

Perception: Speculative Transcription

To balance speed and accuracy, the system runs two transcription engines in parallel. A fast, streaming engine (like Flux) provides immediate, low-latency text, while a slower, more accurate engine (like Scribe v2) processes the audio with more context. The system uses a "corrective" logic: if the slower engine produces a more accurate result, it overrides the initial streaming output. If new audio arrives before the slow engine finishes, the correction is discarded, ensuring the agent remains responsive to the most recent user input.

Planning: Background Tool Execution

To minimize round-trip latency, the system offloads tool calls to background agents. These background processes perform the heavy lifting—such as phonetic matching or database lookups—and inject the results into the main LLM's context. This creates a "sleight of hand" where the main model acts as if it performed the tool call itself, preventing the need for the user to wait for the model to initiate and complete a tool-calling cycle.

Control: Prefix Caching and Seamless Synthesis

To hide the latency of text-to-speech (TTS) generation, the system employs a prefix cache.

  • Caching: The system monitors the agent's output stream for common phrases or sequences already generated in previous turns.
  • Streaming: When a cache hit occurs, the agent begins playing the audio immediately while the LLM is still generating the remainder of the sentence.
  • Seaming: Once the cached audio finishes, the system seamlessly transitions to the live stream from the TTS provider (e.g., Cartisia). The system suppresses the overlapping audio from the TTS provider so the transition between the cached "opener" and the unique, generated content is imperceptible to the user.