The Knowing-Saying Gap

Research into LLM reliability reveals a critical disconnect between a model's internal representation of truth and its final output. While models often output high-confidence responses, their internal hidden states frequently contain information that contradicts the generated text. This phenomenon, termed the 'knowing-saying gap,' suggests that models possess latent knowledge of their own errors that is not reflected in their surface-level confidence scores.

Probing as an Error Detection Mechanism

Standard confidence metrics—such as log-probabilities or self-reported certainty—are often insufficient for identifying hallucinations or factual inaccuracies. The authors demonstrate that internal state probes (linear classifiers trained on the model's intermediate activations) can outperform these traditional metrics. By analyzing the model's internal processing during the generation phase, these probes can identify when the model is 'uncertain' or 'incorrect' even when the model's output layer suggests high confidence.

Implications for AI Safety and Reliability

This research highlights that relying on a model's own output confidence is a flawed strategy for safety-critical applications. Instead, developers should look toward 'white-box' monitoring techniques. By implementing probes that monitor internal activations, systems can trigger interventions or human-in-the-loop reviews when the model's internal state indicates a high probability of error, effectively bridging the gap between what the model knows and what it says.