The Persistence of High-Confidence Errors

Standard approaches to measuring LLM uncertainty often rely on the assumption that if a model is wrong, it will at least be 'uncertain' (i.e., have low probability scores). This research identifies a critical failure mode: 'stable miscalibration.' In this state, models do not merely hallucinate; they express high confidence in their incorrect outputs, and this behavior is consistent across repeated sampling or slight variations in prompts. Because the error is stable, simple techniques like temperature scaling or basic self-consistency checks often fail to flag these incorrect responses as unreliable.

Why Standard Calibration Fails

Calibration techniques typically attempt to align a model's predicted probability with its actual accuracy. However, stable miscalibration suggests that the model's internal representation of 'certainty' is fundamentally decoupled from its factual accuracy in specific domains. The authors argue that this is not a random noise issue but a systematic bias in how LLMs represent knowledge. When a model is confidently wrong, it is effectively 'locked in' to a specific, incorrect reasoning path. Consequently, standard calibration methods—which assume that uncertainty is a function of the model's internal probability distribution—cannot detect these errors because the model's confidence scores remain artificially high even when the output is factually incorrect.