Bias, Equity &
Fairness in Clinical AI

How bias enters
healthcare AI systems

Algorithmic bias in healthcare AI arises through multiple mechanisms, often simultaneously. Training data bias is the most fundamental — when the population represented in training data is not representative of the population in whom the system will be deployed, the model systematically underperforms for underrepresented groups. Historical healthcare data frequently underrepresents racial and ethnic minorities, patients with disabilities, patients from lower socioeconomic backgrounds, and women — because access to care, documentation practices, and research participation have all historically been unequal.

Proxy variable bias occurs when AI systems use variables that appear clinically neutral but function as proxies for protected characteristics. The most documented example in healthcare AI: a widely used commercial algorithm used healthcare spending as a proxy for health need, on the assumption that sicker patients incur higher costs. For Black patients, systemic barriers to care access meant lower spending despite equivalent or greater health need — causing the algorithm to systematically underestimate the health needs of Black patients at similar health status, resulting in fewer referrals for high-need care programs.

Label bias occurs when the clinical outcomes used to train AI models reflect historical inequities in diagnosis and treatment rather than true disease burden. If a diagnostic condition has historically been underdiagnosed in women — as has been documented for cardiovascular disease, autism, and autoimmune conditions — an AI trained to predict that diagnosis will learn the biased historical pattern and perpetuate the underdiagnosis.

Documented examples
of AI bias causing clinical harm

The 2019 study in Science documenting bias in a commercial risk-stratification algorithm affecting millions of patients is the most cited example — and the most instructive. The algorithm used healthcare cost as a proxy for health need, producing systematic underestimation of Black patients' health needs. The investigators estimated that approximately 46,000 additional high-risk Black patients would have been enrolled in care management programs if the algorithm had not been biased.

Pulse oximetry — a non-AI technology but algorithmically determined — has been documented to systematically overestimate arterial oxygen saturation in patients with darker skin tones, causing delayed recognition of hypoxemia. This example illustrates that algorithmic bias is not a new phenomenon unique to deep learning — it is a design and validation failure that can occur in any algorithmic system when diverse populations are not adequately represented in development and testing.

Dermatology AI systems trained predominantly on images of lighter skin tones have consistently demonstrated degraded performance on patients with darker skin tones — a documented concern that has not prevented widespread deployment of tools with known performance differentials.

Evaluating and mitigating
algorithmic bias in practice

Bias evaluation requires analyzing model performance across demographic subgroups — not just overall accuracy metrics. An AI system that achieves 90% overall accuracy may achieve only 74% accuracy for a specific demographic group. Overall performance metrics mask subgroup disparities, which is why disaggregated performance analysis is a governance requirement for any AI system deployed in a diverse population.

The clinical governance framework for bias evaluation should examine: training data composition — is the training population representative of the deployment population across relevant demographic dimensions? Proxy variable review — do predictor variables function as demographic proxies? Label validation — do training labels reflect true disease burden or historical diagnostic inequity? And subgroup performance analysis — does the model perform differently across demographic groups, and is the differential clinically meaningful?

Bias mitigation strategies include: modifying training data composition to improve representativeness; removing or replacing proxy variables; applying algorithmic fairness constraints during model training; and calibrating model outputs separately for demographic subgroups. No mitigation strategy eliminates bias entirely — the governance obligation is ongoing monitoring, transparent reporting of known performance differentials, and a commitment to continuous improvement.

Key concepts
from this lesson