The Reliability Gap in Agentic Workflows

Most current AI agents are capable of executing individual tasks—clicking, typing, or calling APIs—but fail when stringing these tasks into end-to-end workflows. The author argues that this failure occurs because the "real work" exists in the seams between applications, where context is lost. While coding agents have achieved high reliability due to the verifiable nature of code (via unit tests and execution), general knowledge work is messy and lacks built-in verification mechanisms. Without a way to verify outcomes, agents cannot be trusted with complex, multi-step processes.

Perception Agents: Closing the Loop

To bridge this gap, the author proposes "Perception Agents" that mirror human collaboration patterns. Instead of relying solely on text-based prompts or backend APIs, these agents operate on the rendered interface, allowing them to see what the user sees. This creates a shared context that reduces the need for lengthy, lossy explanations. The perception loop consists of three stages:

  1. Perceive: The agent observes the rendered screen or listens to meeting transcripts to understand the current state and user intent.
  2. Plan: The agent reasons about the necessary actions based on visual or auditory input.
  3. Act: The agent executes tasks and, crucially, monitors the results to confirm success.

Practical Implementation and Verification

The author introduces an open-source harness for building these agents, focusing on two primary capabilities:

  • Annotation: A Chrome extension that allows users to visually select and mark elements on a screen (e.g., highlighting a heading to change its font or color). This provides the agent with precise, non-textual instructions, reducing ambiguity.
  • Verification: An automated check that compares the agent's output against predefined design specifications or user flows. By walking through the application like a human user, the agent can flag violations (e.g., off-brand colors or broken layouts) and generate a report, removing the need for manual review.

By integrating these tools, agents can move from "fire and forget" execution to a continuous, self-correcting loop that adheres to design guidelines and functional requirements.