The Need for Clinical AI Peer Review

Clinical AI systems often fail in ways that are opaque, non-deterministic, or context-dependent. The authors argue that current software debugging practices are insufficient for healthcare environments where AI outputs directly influence patient outcomes. By adapting the traditional medical 'Morbidity and Mortality' (M&M) conference—a standard practice where clinicians review adverse events to identify systemic failures rather than individual blame—the authors provide a structured approach for AI developers and clinicians to audit failures.

The M&M Framework for AI

The proposed framework shifts the focus from 'why did the model get this wrong' to 'how did the sociotechnical system fail.' The review process involves three core components:

  1. Event Identification & Reporting: Establishing a low-friction mechanism for clinicians to flag AI outputs that led to, or could have led to, patient harm. This requires a culture of psychological safety where reporting is seen as a tool for system improvement rather than a punitive measure.
  2. Multidisciplinary Root Cause Analysis (RCA): AI failures are rarely purely technical. The framework mandates a review by a panel comprising data scientists, clinicians, and human-factors engineers. They categorize failures into three domains:
    • Technical/Algorithmic: Data drift, training bias, or model hallucination.
    • Human-AI Interaction: Alert fatigue, over-reliance (automation bias), or poor UI/UX design.
    • Systemic/Workflow: Integration issues, lack of training, or misaligned clinical incentives.
  3. Actionable Remediation: The goal of the M&M process is to produce concrete changes. This might include updating training datasets, adjusting confidence thresholds, modifying clinical workflows, or implementing new guardrails for the AI agent.

Moving Beyond Debugging

The authors emphasize that clinical AI safety is an ongoing process of continuous improvement. Unlike traditional software, where a 'fix' is often a code patch, clinical AI failures often require a combination of model retraining and changes to clinical policy. By formalizing this review process, organizations can move from reactive troubleshooting to proactive risk management, ensuring that AI tools become safer and more reliable over time.