The Architecture of Real-Time Voice

Traditional voice pipelines (Speech-to-Text → LLM → Text-to-Speech) are unsuitable for real-time interaction because they introduce latency that makes conversation feel broken. Instead, treat voice AI like a phone call rather than a walkie-talkie. This requires a persistent, full-duplex WebSocket connection where audio streams in both directions simultaneously. The browser handles the microphone and speaker, while the backend manages the ADK components to facilitate communication with the Gemini Live API.

ADK Building Blocks

The Agent Development Kit (ADK) simplifies the management of live sessions through three primary abstractions:

  • Agent: A configuration-based definition consisting of a model, a personality, and a set of tools (Python functions).
  • Runner: The engine that manages the lifecycle of the live call from start to finish.
  • Session: The state of the conversation. For real-time voice, it is critical to keep the session in memory. Relying on network-based session storage introduces round-trip latency for every audio chunk, causing the agent's voice to lag significantly behind the conversation.

The Live Loop and Decoupling

To prevent audio streams from blocking each other, use a LiveRequestQueue to decouple the upstream (microphone) and downstream (agent response) traffic. This acts like a sushi conveyor belt: the browser pushes audio chunks onto the queue, and the model consumes them independently.

When sending data to the model, distinguish between two methods:

  • send_realtime(): Used for continuous streams like microphone input. Crucially, you must send audio data constantly, even during silence, so the model can detect natural sentence boundaries.
  • send_content(): Used for discrete, complete inputs (e.g., a typed message or a photo) where the user has finished their input.

The runner processes the live call as a stream of events. Your application must be reactive to these events—such as audio chunks, captions, or tool requests—to ensure the agent feels responsive. If an 'interrupted' event occurs, the system must immediately stop the current audio playback to allow for natural human-like turn-taking.