The Failure of Reactive Safety

In high-stakes environments like healthcare, the standard software engineering playbook—shipping to 5% of users, monitoring dashboards, and rolling back if errors occur—is unacceptable. When the user is a patient, a red dashboard means someone has already been harmed. Because medical AI agents (like Ufonia's 'Dora') provide clinical advice, they are regulated medical devices. This necessitates a shift from reactive monitoring to a proactive, simulation-first safety stack.

The Simulation Flywheel: Matrix

To replace the reactive loop, Ufonia developed 'Matrix,' a framework that simulates clinical conversations to identify hazards before they reach real patients. This framework relies on two core components:

  • PatBot (Simulated Patient): An LLM conditioned on specific clinical scenarios. By using simulated patients rather than human actors, the team can iterate rapidly and test diverse personas, from verbose to concise speakers. Validation via Patient and Public Involvement (PPI) studies confirmed that real patients often found these simulated interactions more realistic than human-to-human ones.
  • Bev Judge (Automated Evaluator): An LLM judge that reviews dialogues against a list of clinician-defined hazards. Validated against 10 clinicians across 10 specialties using 240 test cases, the judge achieved an F1 score of 0.96 and near-perfect sensitivity. This allows for scalable, expert-level evaluation of thousands of simulated calls.

From Hand-Tuning to Optimization

Manual prompt engineering is brittle and subjective, with minor formatting changes often causing massive performance swings. Instead, the team uses automated prompt optimizers (such as JPE/DSPY) to refine system instructions.

  • Cost-Matrix Optimization: Rather than optimizing for generic accuracy, the system uses a cost matrix that assigns higher penalties to missed red-flag symptoms than to false alarms. This aligns the model's behavior with clinical priorities.
  • Evidence-Based Shipping: The final deliverable is not just the model, but the comprehensive evidence trail. Every prompt version, judge verdict, and simulation run is traced back to specific clinical hazards. This creates a reproducible, auditable safety case that satisfies regulatory requirements while allowing the system to improve through a continuous data flywheel.