The Illusion of Consensus in LLM Evaluation

When using LLMs as judges to evaluate other models, a common practice is to employ multiple judges and rely on their consensus to determine the 'correct' answer. The underlying assumption is that if multiple models agree, the probability of error decreases significantly. This research challenges that assumption, demonstrating that LLM judges exhibit strong error dependence. Because these models are often trained on similar datasets and architectures, they tend to make the same mistakes, meaning their consensus is not an independent verification but rather a reflection of shared systematic biases.

Why Consensus Fails to Mitigate Bias

In traditional statistical evaluation, independent observers reduce variance and error. However, LLM judges are not independent observers. The study highlights that when one LLM judge fails on a specific type of prompt or reasoning task, other LLM judges are statistically more likely to fail in the exact same way. This 'error dependence' means that consensus-based evaluation metrics—like majority voting—often provide a false sense of security. Instead of filtering out noise, the consensus mechanism frequently reinforces the shared blind spots of the models involved.

Implications for AI Benchmarking

This finding suggests that current evaluation pipelines relying on LLM-as-a-judge are likely overstating the reliability of their results. To improve evaluation accuracy, the paper implies a need for more diverse judge ensembles or the development of evaluation methods that account for the correlation between model errors. Relying on consensus without acknowledging these dependencies leads to inflated performance metrics and masks the true limitations of the models being tested.