The Challenge of Factuality Evaluation
Evaluating the factual accuracy of Large Language Models (LLMs) has historically suffered from a fundamental trade-off: granular verification (checking individual claims) often loses the broader context of the document, while holistic evaluation (checking the whole output) lacks the precision to pinpoint specific hallucinations. TriQua proposes a solution to this by reconciling these two approaches.
The TriQua Framework
TriQua introduces a multi-dimensional evaluation strategy that breaks down the verification process into three specific components:
- Atomic Fact Extraction: Instead of evaluating long-form text, the framework decomposes responses into atomic claims. This ensures that every individual assertion can be verified against a source document.
- Contextual Alignment: Rather than treating claims in isolation, TriQua maps these atomic units back to their original context. This prevents "factually correct but contextually misleading" errors, where an LLM might state a true fact that is irrelevant or distorted by the surrounding narrative.
- Granular Verification: By applying a structured scoring mechanism to these mapped claims, the system provides a more nuanced view of model performance. This allows developers to distinguish between minor errors (e.g., date inaccuracies) and major hallucinations (e.g., fabricated events).
Impact on Model Development
By using this three-pronged approach, TriQua enables more precise debugging of LLM pipelines. Instead of receiving a single "accuracy" score, developers can identify whether their model struggles with specific types of information, such as entity extraction or logical reasoning within a context window. This framework moves the industry toward more robust evaluation benchmarks that better reflect how humans actually verify information: by checking the facts while keeping the full story in mind.