The Challenge of Multimodal Agentic Evals

Uber's food enhancement agent aims to improve the quality of photography for independent merchants. The core challenge is balancing creative enhancement with strict faithfulness to the original dish. The system must avoid "AI-looking" results, preserve merchant branding, and prevent marketplace homogenization. The team treats the agent as a spectrum: balancing the creativity of LLMs with rigid, rule-based safety guardrails.

The Multi-Stage Orchestration Flow

The system operates through a series of specialized agents in a flat, JSON-logged structure, which allows for easy diagnosis by both technical and non-technical teams. The workflow consists of:

  1. Image Understanding & Routing: A multimodal agent describes the image and decides whether to enhance or skip it based on a rubric. This is evaluated using a confusion matrix (precision/recall) to ensure no poor-quality images are enhanced (wasting compute) and no high-quality images are degraded.
  2. Iterative Enhancement: If routed for enhancement, the agent generates a specific prompt and edits the image. A QA agent then checks for plating, faithfulness, and color. If it fails, the feedback is fed back into the generation loop for up to K iterations.
  3. Post-Processing QA: A final "Swiss cheese" gate provides holistic policy and quality checks before publishing, acting as a redundant safety layer.

Closed-Loop Self-Correction & Autotuning

To prevent model drift and handle edge cases, the team implemented a self-learning system that operates without human intervention:

  • Diagnosis Agent: An "umbrella" agent analyzes production mismatches (comparing agent output against human-labeled golden datasets) to localize failures.
  • Autotuning Pipeline: Once a failure is localized, a two-part sub-agent process triggers: a Reflect Agent identifies systemic issues and noise, and a Synthesize Agent updates the specific agent's configuration.
  • Continuous Benchmarking: New configurations are benchmarked against the golden dataset. If they pass, they are automatically pushed to the production config store.

Managing Reward Hacking and Faithfulness

Reward hacking occurs when agents over-optimize for specific metrics at the expense of reality (e.g., replacing a unique bowl with a generic one to satisfy a "cleanliness" metric). The team mitigates this by:

  • Pairwise Comparison: Using human-defined policies to assess if an edit is actually "better" rather than just "different."
  • Internal Dogfooding: Incorporating thumbs-up/down feedback from internal teams and merchants into the diagnosis loop.
  • Segmented Analysis: Tracking conversion metrics across different geographies, device types, and dish types to identify where the agent is succeeding or failing in specific marketplace segments.