Safety & Risk in
Healthcare AI

Categories of AI-related
patient safety events

AI-related patient safety events do not fit neatly into existing incident taxonomies developed for traditional clinical errors. They require a distinct classification framework that reflects the novel failure modes of algorithmic systems. The primary categories include: model malfunction — where the AI system produces an error, crash, or unexpected output; unexpected or incorrect output — where the system functions as designed but produces a clinically inappropriate recommendation; bias-related differential harm — where the system produces systematically worse outcomes for specific patient groups; and human-AI teaming failures — where the interaction between clinician and AI system breaks down in ways that cause harm.

Alert fatigue events — where clinicians dismiss genuinely important AI-generated alerts due to habituation — represent one of the most prevalent and most preventable AI safety event categories. Automation complacency events — where clinicians reduce their own clinical vigilance because of trust in an AI system — represent the complementary risk. Data drift events occur when the patient population, clinical workflows, or data inputs to an AI system change in ways that degrade model performance without triggering any system error.

The reporting of AI-related safety events is critically underdeveloped. Most existing safety reporting systems have no specific category for AI-related events. Events are frequently attributed to individual clinical error rather than to the AI system that contributed to the conditions for that error. This attribution gap makes it impossible to learn systematically from AI-related harm events or to identify patterns that should trigger governance responses.

Model drift
the silent performance degradation

Model drift describes the gradual degradation of an AI system's performance over time as the real-world data environment diverges from the training data environment. It occurs because healthcare is not static — patient populations change, clinical practices evolve, documentation patterns shift, new medications are introduced, and disease prevalence fluctuates seasonally and epidemiologically.

A sepsis prediction model trained before the COVID-19 pandemic may perform poorly on patients with COVID-19-associated sepsis, whose presentation differs from the sepsis phenotypes in the training data. A documentation AI trained on one version of ICD-10 coding guidelines may produce incorrect suggestions when coding guidelines are updated. A predictive readmission model trained in a pre-telehealth era may systematically underperform for patients who have access to remote monitoring and virtual follow-up.

Model drift is insidious because it does not trigger any system error. The AI continues to function and generate outputs. The outputs are simply less accurate than they were — and in many organizations, nobody is monitoring performance closely enough to detect the degradation before it causes harm. Ongoing performance monitoring — comparing current model predictions to actual clinical outcomes in the local patient population — is the only reliable mechanism for detecting model drift.

Governing AI safety
the organizational infrastructure required

Effective AI safety governance requires organizational infrastructure that most healthcare institutions have not yet developed. The minimum governance requirements include: a mechanism for reporting AI-related safety events and near misses with AI-specific data fields; regular performance monitoring comparing model predictions to actual outcomes in the local patient population; a defined process for investigating AI-related safety events that can distinguish between model failure, deployment failure, and human-AI teaming failure; and clear escalation pathways when AI safety concerns are identified.

The AI Oversight & Governance Committee — covered in detail in Lesson 09 — is the governance body responsible for maintaining this infrastructure. But governance committee oversight alone is not sufficient. Safety monitoring must be embedded in clinical operations, with front-line staff empowered and supported to report AI-related concerns without fear of attribution to individual error.

Post-market surveillance — the ongoing monitoring of AI performance after regulatory clearance and clinical deployment — is an emerging regulatory requirement in multiple jurisdictions. Organizations that build safety monitoring infrastructure proactively will be better positioned to meet these requirements and, more importantly, to protect their patients from AI-related harm.

Key concepts
from this lesson