Architectural Foundations for Health AI

Most healthcare AI failures are not model hallucinations, but architectural flaws. To build safe, member-facing health AI, you must move beyond reactive safety measures and implement three non-negotiable foundations:

• Architectural Constraints over Policies: Do not rely on policies to redact PHI. Strip PHI at the pipeline ingestion boundary before it ever reaches your data lake. If the data isn't there, it cannot be leaked or misused. Maintain strict separation between production and non-production environments to prevent data leakage.
• Deterministic Code over Prompts: High-stakes behaviors—such as emergency routing (911/988), identity verification, and intent routing—must live in a code layer above the model. LLMs are probabilistic and unreliable for irreversible decisions; they should never be given a vote on safety-critical paths.
• Continuous Monitoring over Gating: Safety is not a pre-launch checklist. It is a continuous evaluation layer. Use a combination of automated judges (scoring for clinical accuracy, safety, and drift), member feedback (thumbs up/down), and manual sampling of high-stakes traces. When a failure occurs in production, it indicates a need for a new judge or monitoring rule, not just a prompt tweak.

Decision Frameworks for High-Stakes Shipping

When stakeholders disagree on whether to ship a feature, use a structured decision-making framework to remove politics and prioritize safety:

• Worst-Case Severity: Severity is defined by the worst plausible outcome, not the average outcome. A bug that causes serious harm to 0.1% of users is a higher priority than a minor annoyance affecting 100%.
• Asymmetric Defaults: When unsure, always choose the safer mistake. For safety bugs, delay the launch. For polish bugs, ship and iterate.
• Revealed vs. Stated Tolerance: Your true risk tolerance is defined by what is currently running in production, not what your team claims is acceptable. Calibrate your launch bar to your existing production reality.
• Human-in-the-Loop: The bottleneck is not compute; it is the human capacity to interpret signals. Design systems that allow humans to act on the data provided by your monitoring layer.

The Discipline of Judging the Judges

In a non-deterministic system, your automated judges are also non-deterministic. Before reacting to a drop in quality scores, you must verify the judge. If a judge flags a correct clinical response as a hallucination, you must fix the judge's prompt, not the agent's. Fixing a judge is not cheating; it is necessary maintenance for a production-grade safety system.