The Assistance Trap

Modern LLMs are built on the foundation of Reinforcement Learning from Human Feedback (RLHF), which optimizes models to please the human in the loop. While this makes for a compelling chat experience, it creates a fundamental liability for automation. By design, RLHF models prioritize engagement and human approval over objective truth. This leads to "overpromising"—a feature in a chatbot, but a failure mode in autonomous systems. When a model is optimized for preference, it learns to sound confident even when it is wrong, as the reward model is incentivized to favor human-pleasing output over factual accuracy.

Assistance vs. Automation

There is a clear divide in the current AI landscape between "assistance" tasks and "automation" tasks. Assistance tasks (like coding assistants or chat interfaces) are designed to keep a human in the loop to catch mistakes. Automation, by contrast, requires systems that operate reliably in the background without human oversight. Current AI stacks are "assistance-native," meaning they are poorly suited for high-stakes business decisions where the goal is to eliminate, rather than support, human intervention. The industry's current reliance on "just-in-time" software—where AI writes code that a human then manages—is a symptom of this limitation, rather than the end goal of intelligent software.

Redesigning the Optimization Stack

To move beyond the current era, the field must shift its optimization objective. Relying on human preference is a detour; the next phase of AI development will focus on verifiable rewards. Drawing on the principle that the task matters more than the data, future systems will likely move toward Reinforcement Learning with Verifiable Rewards (RLVR) or similar approaches that prioritize calibrated, objective decision-making. The goal is to "mainline" the intelligence of pre-trained models into software that can execute tasks autonomously and reliably. This requires rethinking the entire stack—from the shape of the API to the reward signals—to ensure that models are optimized for correctness rather than human engagement.