Moving Beyond Text-to-Speech

Traditional text-to-speech (TTS) systems are unidirectional: they receive text and output audio, but they cannot "hear" the user. Gemini Live shifts this to an audio-to-audio paradigm. It processes audio input directly, allowing the model to interpret tone, pauses, and energy. Crucially, it supports streaming, meaning the model begins generating and outputting audio before the full response is finalized, creating a more natural, low-latency conversational experience.

Architecture and the Core Loop

To build a live voice agent, you must maintain a persistent connection between the browser (client), the Gemini Live API, and a backend. A standard HTTP request-response cycle is insufficient; instead, use a WebSocket to keep the channel open for continuous, bidirectional audio flow. The backend manages two concurrent processes: one streaming microphone audio to the model and another receiving the model's audio response. The core development loop consists of four steps:

  1. Open: Establish the WebSocket session.
  2. Send: Stream microphone audio to the model.
  3. Receive: Capture the model's audio output.
  4. Play: Stream the audio back to the browser.

Making Conversations Feel Natural

To move from a functional loop to a conversational agent, three specific mechanisms are required:

  • Voice Activity Detection (VAD): The model must distinguish between user speech and silence to identify turn boundaries. Gemini Live includes built-in VAD, which monitors the audio stream to detect when a user begins and ends their turn.
  • Barge-in: This allows users to interrupt the agent mid-sentence. When the VAD detects the user speaking while the agent is outputting audio, the model stops its current generation. To make this feel instantaneous, the client-side application should immediately stop local audio playback upon detecting user input, rather than waiting for a signal from the server.
  • Tool Execution: Since models cannot directly interact with UI or external hardware, you must provide "tools"—functions defined in your code with specific names and descriptions. When the model determines an action is needed (e.g., skipping a track), it requests the tool execution. A critical rule for voice agents is to keep tool execution fast; because the model pauses while waiting for a tool to return, slow execution results in awkward silence in the conversation.