The Two Knobs of AI Performance

Most AI development focuses exclusively on accuracy—the first knob. However, for voice interfaces and embodied AI, accuracy is only half the battle. The second, often ignored knob is system behavior under uncertainty. When a model is unsure of its output, it has a choice: act, stop, or confirm. By treating these behaviors as an optimization problem rather than a gut-feeling threshold, developers can drastically reduce user frustration without changing the underlying model's accuracy.

The OUCH Heuristic

To optimize these behaviors, Amit Desai proposes the Outcome User Cost Heuristic (OUCH). This framework acknowledges that not all errors are equally painful. For example, playing the wrong song on a smart speaker is significantly more disruptive than the system saying, "I didn't understand that."

To implement OUCH:

  1. Quantify the Cost: Assign a numerical "pain" value to every possible outcome (e.g., correct act = 0, stop/reject = 4 seconds of lost time, wrong act = 10 seconds of lost time).
  2. Define the Thresholds: Instead of picking an arbitrary confidence threshold (e.g., 65%), use the cost function to find the mathematical optimum. In the speaker example, the optimal threshold was 43%, which minimized the total "OUCH" points per turn.
  3. Expand Behavioral Options: Introduce a "confirm" behavior. This splits the confidence distribution into three regions (act, confirm, stop) and requires two thresholds. By assigning a lower cost to a confirmation (e.g., 2 seconds) compared to a wrong act (10 seconds), the system can steer users toward a more efficient resolution path.

Scaling Across Modalities

This approach is not limited to audio-only smart speakers. When applied to multimodal interfaces like televisions, the cost variables shift. Because a TV can display visual choices, the "cost" of a confirmation decreases, as users can select an option via remote control rather than through a lengthy voice correction. The core principle remains constant: by mapping the specific user journey and the relative pain of each failure mode, developers can build systems that feel significantly more intelligent and reliable, even when the underlying model remains at a constant accuracy level.