Yin-Yang LLM Pipeline Cuts Noise in Code Scanning
Build reliable AI code scanners by pitting a recall-focused hypothesis agent against a precision-focused evidence agent, stripping reasoning to avoid bias, and enforcing a deterministic policy gate—treating LLMs as stochastic machines, not oracles.
Designed Distrust: Yin-Yang Agents Balance Recall and Precision
Treat LLMs as stochastic machines with expected errors, not infallible oracles. Veritas pipeline uses two opposing agents: the hypothesis agent (Yang) maximizes recall by scanning architecture entry points, trust boundaries, and threat models to flag every plausible vulnerability candidate cheaply—prioritizing low cost of refuted hypotheses over missing real issues. The evidence agent (Yin) maximizes precision as a skeptical auditor, refuting claims unless backed by cited source-code evidence like sanitizers or validation logic. This tug-of-war decouples generation from judgment, letting hypothesis over-generate (reducing Type 2 false negatives) while evidence strictly verifies (reducing Type 1 false positives), avoiding single-stage scanners' internal conflicts on the precision-recall curve.
Information Bottleneck Defeats Anchoring Bias
Strip reasoning from hypotheses via slim_hypotheses_for_evidence() before evidence review, forcing independent verification against raw code context. Deleting the "why" prevents anchoring bias, where prior explanations cause overfitting and hallucinated confirmations. Evidence agent must rediscover exploit paths or discard findings, making the system smarter by intentionally "dumbing down" the second stage—trading tokens for quality.
Deterministic Policy Gate Overrides AI Verdicts
AI never decides alone: a mechanical policy gate applies checklists post-pipeline. Findings below 0.3 confidence score marked "Inconclusive"; critical/high-severity inconclusive ones flagged NEEDS_HUMAN instead of dropped. Every pre-scan finding gets confirmed, disproven, or escalated with full audit trail. This handles variation like assembly-line quality control, producing reviewable outputs for developers during manual reviews—not replacing SAST tools like CodeQL, but assisting Security Champions.
Pipeline Outcomes: Process Over Prompting
Statistical process control trumps prompt engineering: accuracy emerges from architectural tension (expansion then contraction), not model size or perfect prompts. Hypothesis runs high-recall/low-precision; evidence shifts to high-precision. Extra compute yields hallucination-free reports tied to evidence, with next steps in empirical calibration against known vuln/non-vuln cases. Veritas (GitHub POC) proves treating LLM outputs as intermediates—with inspection, disposition, escalation—yields trustworthy code review aids.