The Problem: Over-Crediting in LLM-as-a-Judge

LLM-based evaluation (LLM-as-a-judge) has become the standard for assessing agentic workflows, but it is prone to significant bias. A primary issue is 'over-crediting,' where judge models tend to assign high scores to agent outputs that appear fluent or confident, even when they fail to meet specific task requirements or contain subtle errors. This behavior stems from the judge's tendency to prioritize stylistic coherence over functional correctness, leading to inflated performance metrics that do not reflect real-world utility.

Inducing Reward-Free Rubrics for Objective Assessment

To mitigate this bias, the authors propose a method for inducing reward-free judging rubrics. Instead of relying on a single scalar reward signal—which often conflates multiple dimensions of performance—this approach forces the judge to decompose the evaluation into specific, observable criteria.

By inducing these rubrics, the system shifts the evaluation paradigm from holistic 'gut-feeling' scoring to a structured, evidence-based assessment. The key mechanism involves:

• Decomposition: Breaking down complex agent tasks into discrete, verifiable sub-tasks or constraints.
• Reward-Free Scoring: Eliminating the reliance on a singular reward score, which often encourages the judge to 'guess' what the user wants to hear, and instead requiring the judge to justify its assessment against the induced rubric.
• Constraint-Based Verification: The rubric acts as a set of guardrails, forcing the judge to explicitly check for the presence or absence of specific functional elements in the agent's output.

Impact on Evaluation Accuracy

This approach significantly reduces the 'halo effect' where a well-written but incorrect response receives high marks. By forcing the judge to adhere to an induced rubric, the evaluation becomes more granular and reproducible. The research demonstrates that this method provides a more accurate reflection of agent capabilities, as it forces the judge to penalize failures in logic or constraint satisfaction that would otherwise be masked by the agent's persuasive tone. This leads to a more reliable feedback loop for developers, allowing for better identification of specific failure modes in agentic systems.