Artificial Intelligence and Health: What the Clinical Evidence Actually Shows

The phrase "artificial intelligence and health" covers an enormous and uneven terrain. At one end: FDA-authorized software that flags pulmonary nodules on CT scans, backed by prospective validation studies and integrated into radiology workflows at major health systems. At the other: large language models answering patient questions in hospital portals, operating without regulatory authorization and with documented tendencies to generate plausible-sounding but factually incorrect clinical information.

Treating these as a single phenomenon — "AI in healthcare" — produces confusion that affects clinical adoption decisions, procurement, and policy. The more useful frame is to ask, for any specific tool or claim: what task does it perform, what evidence supports that performance, and what regulatory status governs its use?

The Regulatory Divide: Cleared vs. Not Cleared

The FDA's framework for AI-enabled medical devices distinguishes software that meets the definition of a medical device — Software as a Medical Device (SaMD) — from software that supports clinical workflows without making diagnostic or treatment decisions. This distinction matters practically.

As of mid-2026, the FDA has authorized several hundred AI/ML-enabled devices, the large majority through the 510(k) pathway. Radiology accounts for the densest concentration — tools for detecting pulmonary nodules, flagging intracranial hemorrhage, and assisting in mammography reading have accumulated the most clearances and the most post-market evidence. Cardiology and pathology follow at some distance.

Generative AI tools — including large language models used for clinical documentation, patient communication, differential diagnosis support, and discharge summary generation — occupy a different regulatory position. As of Q2 2026, no LLM-based generative AI product has received FDA authorization as a medical device for diagnostic or treatment decision support. Tools in active clinical deployment in this category are operating under a combination of clinical workflow software exemptions, institutional governance policies, and in some cases ambiguous regulatory status.

Where AI Has Demonstrated Clinical Utility

The strongest evidence base for AI in health sits in narrow, well-defined visual classification tasks where ground truth is established by pathology or structured clinical outcomes. Three domains stand out on evidence quality.

Diabetic Retinopathy Screening

AI-based grading of fundus photographs for diabetic retinopathy is one of the most extensively studied clinical AI applications. Multiple prospective studies and at least one large RCT have demonstrated non-inferiority to human graders in controlled settings. The FDA has authorized several devices in this category. Real-world deployments at community health centers and safety-net hospitals have shown operational feasibility, though performance gaps have been documented in images from lower-quality cameras and in patients with media opacity.

Equity concerns are documented here: multiple studies have found that models trained predominantly on images from academic medical centers show reduced sensitivity in darker-pigmented fundi. This is not a minor footnote — diabetic retinopathy disproportionately affects Black and Hispanic patients, the same populations underrepresented in many training datasets.

Pulmonary Nodule Detection

AI-assisted detection of pulmonary nodules on chest CT has moved from research to routine radiology workflow at a number of large health systems. The evidence base is primarily retrospective, with some prospective validation. Sensitivity figures for nodules above 6mm are generally high in published studies, though specificity varies considerably by model and scanner type. Workflow integration with PACS systems remains a practical friction point — interoperability with existing radiology infrastructure is not automatic and requires institutional implementation effort.

Sepsis Prediction

Early warning systems for sepsis have been widely deployed in hospital EHR environments, with Epic's Sepsis Prediction Model being the most studied example in real-world settings. The evidence picture here is more complicated than in imaging. A large external validation study published in JAMA Internal Medicine found that the Epic model's performance in independent hospital systems was substantially lower than originally reported — AUC dropped from the vendor-reported 0.76 to around 0.63 in some external settings. Alert fatigue from high false-positive rates has been documented in multiple implementation studies.

Generative AI in Clinical Settings: The Current State

LLMs have entered healthcare settings faster than the evidence base or regulatory framework has adapted. The gap between deployment velocity and evidence quality is wider here than in any other segment of clinical AI.

Ambient Documentation

Ambient AI scribes — tools that listen to clinical encounters and generate structured notes — are the most commercially mature generative AI application in healthcare as of mid-2026. Products from companies including Nuance (Microsoft), Abridge, Suki, and Ambience Healthcare are deployed across hundreds of health systems. Physician surveys and vendor-disclosed metrics consistently report reductions in documentation time, with some studies showing 30–50% reduction in after-hours chart work.

The evidence quality for ambient AI scribes is weaker than the deployment scale suggests. Most published outcome data comes from vendor-disclosed studies or single-site implementation reports without independent validation. Peer-reviewed RCTs examining patient safety outcomes — medication errors, missed diagnoses attributable to note inaccuracies — are largely absent from the literature as of this writing. The FDA has not classified most ambient documentation tools as medical devices, placing them outside the formal pre-market review process.

Clinical Reasoning and Diagnosis Support

Published evaluations of GPT-4, Gemini, and similar models on clinical reasoning benchmarks — USMLE-style questions, case vignettes, diagnostic challenges — have shown performance at or above passing thresholds on standardized tests. This is a meaningful finding and a limited one simultaneously.

Benchmark performance on multiple-choice clinical questions does not translate directly to safe performance on open-ended clinical reasoning tasks with real patients. Several published evaluations have documented hallucination rates — confident, fluent generation of incorrect drug dosages, contraindication statements, or fabricated clinical guidelines — that would be clinically dangerous if acted upon without verification. The hallucination problem is not solved by larger models; it is reduced but not eliminated.

Patient Communication and Health Information

Several health systems have piloted LLM-generated responses to patient portal messages, with human clinician review before sending. Early results from published pilots show high patient satisfaction and reduced physician message burden. The safety record for this specific, human-reviewed application is better than for autonomous clinical reasoning — the human-in-the-loop structure contains the hallucination risk.

Autonomous chatbots providing health information directly to patients — without clinician review — present a different risk profile. Studies examining consumer-facing health chatbots have found inconsistent accuracy, with particular problems around medication interactions, emergency triage guidance, and mental health crisis responses.

Evidence Quality Across AI Health Applications

The evidence base for AI in health is not uniformly weak — it is unevenly distributed. Understanding where the stronger evidence sits helps practitioners and procurement staff make more calibrated decisions.

Algorithmic Bias and Health Equity

Algorithmic bias in healthcare AI is not a theoretical concern — it has been documented in deployed systems across multiple specialties. The mechanisms are well understood: models trained on data from academic medical centers reflect the demographics of those institutions' patient populations, and performance degrades when deployed in settings with different patient mix, equipment, or documentation practices.

• Skin lesion classification models have shown lower accuracy on darker skin tones, reflecting underrepresentation in training datasets from predominantly white patient populations.
• Pulse oximetry AI correction algorithms have been studied in the context of known SpO2 overestimation in patients with darker skin pigmentation — a hardware bias that AI post-processing may or may not adequately correct.
• Chest X-ray pathology detection models trained on datasets with demographic imbalances have shown differential sensitivity by race and sex in external validation studies.
• Commercial risk stratification algorithms used in health system population management have been documented to systematically underestimate illness severity in Black patients relative to white patients at the same actual health status.

The FDA's 2021 action plan for AI/ML-based SaMD identified algorithmic bias as a post-market surveillance priority, but mandatory demographic subgroup performance reporting is not yet a standard requirement for 510(k) submissions. Researchers and procurement staff evaluating AI tools should request disaggregated performance data by race, sex, age, and relevant comorbidities — and treat the absence of such data as a gap in the evidence, not a neutral finding.

Deployment Realities: What Goes Wrong

Clinical AI tools that perform well in controlled validation studies frequently encounter problems when deployed in real health system environments. The failure modes are predictable enough that they can be anticipated during procurement.

Alert Fatigue

High-sensitivity AI tools with moderate specificity generate frequent alerts, many of which are false positives. Clinicians adapt by dismissing alerts without review — a behavior that can paradoxically reduce the clinical utility of the tool below the baseline it was meant to improve. Sepsis alerts, deterioration alerts, and radiology triage flags have all been documented to produce alert fatigue in real-world deployment.

Distribution Shift

Models trained on data from one time period, institution, or patient population can degrade when the distribution of incoming data shifts — due to changes in patient demographics, documentation practices, equipment upgrades, or disease prevalence. This is called model drift, and it is a post-market surveillance problem that most health systems are not currently equipped to detect systematically.

Workflow Integration Friction

Even well-validated tools can fail to achieve adoption if they require clinicians to leave their existing workflow to access them, add steps without removing others, or present outputs in formats that don't fit how clinicians think during a clinical encounter. Implementation science for AI in health is an underinvested area — most vendor submissions focus on model performance, not workflow fit.

What Practitioners and Researchers Should Verify

The questions below are not exhaustive, but they cover the verification tasks that most commonly separate useful AI health tools from ones that generate liability without clinical benefit.

1. FDA authorization status: Is the tool cleared, De Novo authorized, or PMA-approved? If not, what regulatory basis governs its clinical use?
2. Intended use scope: What is the tool authorized to do, exactly? Many radiology AI tools are cleared for "detection support" — not for replacing radiologist review.
3. External validation: Was the model tested on data from institutions other than where it was developed? Single-site retrospective validation is a thin evidence base for deployment decisions.
4. Demographic subgroup performance: Does the published evidence include disaggregated performance by race, sex, age, and relevant comorbidities? If not, you cannot assess equity risk.
5. Post-market surveillance plan: How will performance be monitored after deployment? Who is responsible for detecting and responding to model drift?
6. For generative AI specifically: What is the human oversight structure? What happens when the model generates a hallucinated drug dose or fabricated guideline reference?

The Regulatory Horizon for Generative AI

FDA's posture toward generative AI in clinical settings has been cautious and, as of mid-2026, has not produced a final guidance document specifically addressing LLM-based medical software. The agency's 2023 discussion paper on AI/ML-based SaMD and subsequent workshop proceedings indicate awareness of the regulatory gap, but formal rulemaking has not yet closed it.

The practical implication: health systems deploying LLMs for any task that could be characterized as clinical decision support are making a regulatory bet that those tools will not be reclassified as SaMD requiring pre-market authorization. That bet may prove correct, or it may not. Institutions with legal and compliance exposure should be tracking FDA guidance development in this area.

How This Site Tracks AI and Health

Clinical AI Record organizes information about AI in health around stable, structured objects rather than a chronological article feed. A device record does not expire when a newer article is written. A clinical application brief is updated as evidence accumulates. Regulatory entries are timestamped and linked to primary government documents.

Readers tracing a specific question — whether a given tool is FDA-cleared, what the external validation evidence shows, what regulatory changes affected a device category this quarter — can follow those threads across content groups without losing the connection between regulatory status, clinical evidence, and deployment context.

• The Generative AI in Medicine Watch category tracks LLM evaluations, hallucination documentation, institutional policies, and the evolving regulatory status of generative AI tools in clinical settings.
• The FDA-Cleared AI Device Registry covers authorized devices with links to official submission records, intended use in plain language, and notes on available real-world evidence.
• Clinical Application Briefs synthesize evidence across multiple studies for specific clinical tasks — one brief per application, updated as new evidence is published.
• The Regulatory & Policy Tracker records formal government actions affecting AI in healthcare, scoped to primary source documents rather than vendor or advocacy interpretations.