The Tool-Calling Loop

In a live voice environment, the model cannot directly manipulate application state (e.g., playing music or skipping tracks). Instead, it relies on a four-step loop:

  1. Declare: Provide the model with a menu of available functions, including names and descriptions.
  2. Decide: The model emits a tool call based on user input.
  3. Execute: Your backend code runs the corresponding function.
  4. Return: The result is sent back to the model, allowing the conversation to continue seamlessly.

Crucially, the model selects tools based on their descriptions. If a model consistently calls the wrong tool, the primary fix is to refine the function's docstring or description rather than adjusting the model itself.

Implementation Approaches

There are two primary ways to implement these abilities:

  • Raw SDK: You manually manage the dictionary-based tool definitions and wire the four-step loop yourself. This offers maximum control.
  • Agent Development Kit (ADK): You define abilities as standard Python functions with docstrings. The framework automatically parses the schema and handles the execution loop, significantly reducing boilerplate code.

Rules for Smooth Interactions

To prevent the agent from feeling "broken" during a live conversation, follow these two rules:

  1. Tools Must Be Synchronous and Instant: In a live session, the agent's voice pauses while waiting for a tool to return. If a tool is slow, the silence feels like a system failure rather than a loading state. If an action is inherently slow, return an immediate acknowledgment to the model and perform the heavy lifting asynchronously.
  2. Use a 'Before Tool' Checkpoint: Implement a before_tool_callback to intercept every tool call before execution. This allows you to:
    • Observe: Log all tool activity in one central location.
    • Block: Prevent specific actions based on policy (e.g., enforcing "quiet hours" by blocking loud music requests).
    • Rewrite: Modify arguments before the function runs.

This checkpoint is essential for building robust agents, especially when moving toward more complex tasks like browser automation, where you must validate actions before they interact with the real world.