The Fallacy of Generalizing from Coding Agents

Coding agents are an outlier in the landscape of knowledge work. They benefit from highly structured, "grep-able" environments where identifiers, file paths, and method definitions provide durable cues. Furthermore, coding tasks are typically narrow and well-defined. In contrast, general knowledge work—such as legal or medical research—is defined by ambiguity, implicit meaning, and the need for the agent to reconstruct intent from scratch. When an agent searches for "30 days," it must distinguish between a deadline, a retention rule, or a medical schedule; code-centric search primitives often fail to capture this context.

The Tool-Organization Loop

Historically, humans have solved the problem of scaling knowledge work through a self-optimizing loop: better tools create new roles, which generate more knowledge, which in turn demands better tools. This is evident in the evolution from the Library of Alexandria’s cataloging to modern specialized firms.

In practice, tooling is not a neutral add-on; it determines whether a task is scalable. Benchmarks like the BrowseComp+ leaderboard demonstrate that unoptimized baselines (like poorly tuned BM25) yield ~60% accuracy, rendering them useless for professional workflows. Optimization can push this to ~80-90%, but performance eventually plateaus. The real breakthrough comes from architectural changes rather than just better retrieval models.

Architecting for the "Oracle Gap"

To close the performance gap between AI and human experts, builders must move beyond single-agent loops. Clavié advocates for a hierarchical approach modeled after professional firms:

• Task Decomposition: Instead of a single agent performing exhaustive searches, the primary agent should act as a "partner," breaking down complex queries into specific aspects.
• Searcher Agents: These sub-agents execute targeted queries and return short, synthesized memos to the primary agent.
• Impact: This architecture reduces the "oracle gap" (the distance between perfect information and the search system) by roughly 40%, significantly improving accuracy while reducing token usage and tool calls by 20%.

Strategic Takeaways for Builders

• Avoid Overfitting to Code: Do not assume that techniques effective for software engineering (like keyword-based grep) apply to general knowledge domains.
• Co-design Tools and Agents: Agents must be trained to understand their toolset as a collection of primitives (e.g., BM25 vs. semantic search vs. multi-modal PDF parsing). They need to know which tool to use for a given intent.
• Context is Finite: Even with massive context windows, you cannot fit entire legal or medical corpuses into a single prompt. Orchestration is the only way to manage the scale of enterprise knowledge.