Artificial Intelligence in Medical Diagnosis: What the Evidence Actually Shows

Artificial intelligence in medical diagnosis has moved well past proof-of-concept. Hundreds of AI-enabled devices have received FDA authorization, the majority in imaging specialties, and a growing number have been evaluated in prospective clinical settings. The picture that emerges from that evidence is more complicated than either the optimistic vendor narrative or the skeptical academic counternarrative suggests.

Performance on benchmark datasets is often impressive. Performance in deployed clinical environments — across different scanner vendors, patient populations, and workflow contexts — is more variable. Understanding that gap is the practical challenge for clinicians, procurement staff, and researchers evaluating these tools.

Where AI Diagnostic Tools Are Concentrated

The distribution of FDA-cleared AI devices is not uniform across medicine. Radiology accounts for the largest share by a significant margin, driven by the availability of large labeled imaging datasets, the relatively well-defined nature of detection tasks, and established reimbursement pathways for imaging interpretation.

Ophthalmology stands apart from this list. The FDA's 2018 De Novo authorization of IDx-DR for autonomous diabetic retinopathy screening established a regulatory precedent for AI operating without a clinician in the immediate decision loop — a meaningfully different clearance model than most imaging AI, which is cleared as a decision-support adjunct rather than a standalone diagnostic.

How Diagnostic AI Actually Works in Imaging

Most cleared imaging AI tools perform one of a small set of tasks: detection (flagging the presence of a finding), segmentation (delineating its boundaries), classification (assigning a category or severity score), or triage (prioritizing cases for radiologist review). These are not interchangeable. A tool cleared for triage — routing suspected intracranial hemorrhage cases to the top of the worklist — is not cleared to diagnose hemorrhage, and conflating the two is a common source of misunderstanding in procurement discussions.

Pulmonary Nodule Detection

Lung nodule detection on chest CT is one of the most studied AI diagnostic tasks. Multiple cleared tools exist, and the published evidence base includes both retrospective validation studies and prospective implementation data. The consistent finding across studies is that AI detection sensitivity for nodules above a certain size threshold (typically 6mm and above) is high — often above 90% — but specificity varies considerably, and false positive rates can generate meaningful additional workload if not managed through careful threshold calibration.

The clinical question is not whether AI can find nodules, but whether AI-assisted reading changes patient outcomes compared to unassisted radiology. That evidence is thinner. Studies showing improved detection rates do not automatically translate to improved management decisions or reduced lung cancer mortality, and prospective RCT data on clinical endpoints remains limited relative to the number of cleared devices.

Mammography CAD and Density Assessment

Mammography has the longest history of AI-assisted reading in clinical practice, dating to first-generation CAD systems in the late 1990s. The early CAD literature was not encouraging — several large studies found that traditional CAD increased recall rates without improving cancer detection. The newer generation of deep learning-based mammography tools has performed better in retrospective studies, with some showing non-inferior or superior detection compared to single-reader interpretation.

The Lancet Oncology published a prospective randomized trial (Lång et al., 2023) in which AI-supported screening detected more cancers than standard double reading while reducing radiologist workload by approximately 44%. That study was conducted in Sweden, using a specific vendor tool, in a population-based screening program. Whether those results replicate across different screening program structures, patient populations, and mammography equipment configurations is an open question — and the study authors were explicit about this.

Whole-Slide Imaging and Computational Pathology

Computational pathology — AI applied to digitized whole-slide images — has seen rapid development over the past several years. FDA clearances in pathology have accelerated, particularly for prostate cancer grading and breast cancer biomarker assessment. The task structure differs from radiology: pathology AI is often asked to quantify findings (mitosis count, tumor proportion score, Ki-67 index) rather than simply detect presence, and the ground truth for training requires substantial expert annotation effort.

Inter-pathologist variability in grading — particularly for intermediate-grade prostate cancer — is well-documented, which creates both a rationale for AI assistance and a challenge for validation. If pathologists disagree on the correct label, what does it mean for a model to be accurate? Studies using AI for Gleason grading have shown AUC values in the 0.90–0.96 range on held-out test sets, but external validation across different tissue preparation protocols and scanner types remains inconsistent in the published literature.

The Regulatory Landscape: What FDA Clearance Does and Does Not Mean

FDA clearance via the 510(k) pathway — the most common route for imaging AI tools — establishes that a device is substantially equivalent to a legally marketed predicate device. It does not require the manufacturer to demonstrate clinical superiority, improved patient outcomes, or performance across diverse populations. This is a structural feature of the 510(k) pathway, not a failure of individual submissions.

The practical implication: a cleared imaging AI tool may have been validated primarily on a dataset from a single health system, using a specific scanner manufacturer's equipment, in a population that does not reflect the demographic mix of the intended deployment site. The 510(k) submission will disclose this, but the disclosure is often not surfaced in vendor marketing materials.

The De Novo pathway, used when no predicate exists, requires more extensive evidence and results in a new device classification. PMA, required for the highest-risk devices, demands clinical trial data. For diagnostic AI, PMA authorization is rare — most imaging AI tools are cleared via 510(k) or De Novo.

Performance Metrics: Reading the Numbers Correctly

Sensitivity and specificity are the most commonly reported metrics for diagnostic AI, but they are not sufficient on their own to evaluate clinical utility. Sensitivity tells you how often the AI correctly flags a positive case; specificity tells you how often it correctly dismisses a negative one. Both depend on the threshold the manufacturer has set, and both change when the prevalence of the condition in your patient population differs from the study population.

• AUC (area under the ROC curve) summarizes performance across all possible thresholds. An AUC of 0.95 is strong, but a model with AUC 0.95 can still have poor specificity at the operating threshold chosen for clinical deployment.
• Positive predictive value (PPV) is what clinicians actually care about in triage contexts: given that the AI flagged a case, how often is the finding real? PPV drops sharply in low-prevalence populations even when sensitivity and specificity are high.
• False positive rate per scan matters for nodule detection tools. A tool with 95% sensitivity and 2 false positives per scan in a high-volume chest CT program generates a substantial downstream workload.
• Reader study design affects comparability. Studies comparing AI-assisted reading to unassisted reading vary considerably in whether radiologists read cases with or without prior studies, under time pressure, and in what order — all of which affect the measured benefit.

Algorithmic Bias and Health Equity in Diagnostic AI

Algorithmic bias in diagnostic AI is not a theoretical concern. It has been documented in published literature across multiple imaging domains. The mechanism is straightforward: models trained predominantly on data from certain demographic groups or imaging equipment types learn features that generalize poorly to underrepresented populations.

A well-cited example is chest X-ray interpretation models, where studies have found differential performance across patient sex, race, and insurance status — even when overall AUC appears high. The aggregate metric masks subgroup performance gaps. A model with AUC 0.90 overall may perform at AUC 0.83 in a specific demographic subgroup that was underrepresented in training data.

FDA's AI/ML action plan and subsequent guidance documents have increasingly emphasized the importance of disaggregated performance reporting, but it is not yet uniformly required in 510(k) submissions. Institutions deploying imaging AI tools in diverse patient populations should treat the absence of subgroup performance data as a gap requiring local evaluation, not as evidence that no gap exists.

Workflow Integration: Where Deployment Gets Complicated

The most common point of failure for imaging AI deployment is not model performance — it is workflow integration. Tools that require manual case upload, operate outside the PACS reading environment, or produce outputs in formats that radiologists cannot act on efficiently tend to see low adoption regardless of their technical performance.

PACS integration via DICOM SR (structured reporting) or overlay rendering is the standard expectation for radiology AI tools. Tools that integrate directly into the reading workflow — surfacing AI findings as overlays or structured annotations within the radiologist's existing interface — show higher adoption rates in implementation studies than tools requiring workflow interruption.

Alert fatigue is a real risk. A triage tool that generates frequent low-confidence flags, or a detection tool calibrated for maximum sensitivity in a low-volume research setting, can create a high false-positive burden in a busy clinical environment. Several published deployment reports describe radiologists developing systematic dismissal behavior toward AI outputs after initial alert fatigue — effectively eliminating the tool's clinical benefit.

Reimbursement: The Current State

FDA clearance does not create a reimbursement pathway. This is one of the most persistent practical obstacles to imaging AI adoption. CMS has established specific CPT codes for some AI-assisted diagnostic tasks — most notably for CT fractional flow reserve (FFRCT) and for AI-based quantitative CT lung analysis — but the majority of cleared imaging AI tools do not have dedicated reimbursement codes.

The typical billing model for imaging AI in the US is bundled into the existing professional fee for interpretation — meaning the radiologist bills for the read, and the AI tool's cost is absorbed by the practice or health system. This creates a financial model where AI tools must demonstrate value through efficiency gains or quality improvement rather than direct reimbursement, which affects how institutions evaluate ROI.

What the Evidence Does and Does Not Support

Honest assessment of the current evidence base for AI in medical diagnosis requires distinguishing between what has been demonstrated and what remains plausible but unproven.

Active Research Gaps

Several questions in diagnostic AI remain genuinely open as of mid-2026, not because research is absent but because the existing evidence is insufficient to draw reliable conclusions.

• Long-term patient outcomes from AI-assisted diagnosis. Most studies measure detection metrics. Studies measuring whether AI-assisted diagnosis changes mortality, morbidity, or quality-of-life outcomes are rare and methodologically difficult to conduct.
• Multi-site external validation. Many cleared tools have been validated primarily at the developing institution or on datasets curated by the manufacturer. Independent multi-site validation, particularly across geographically and demographically diverse health systems, is underrepresented in the published literature.
• Model drift in deployed systems. AI model performance can degrade over time as patient populations, equipment, and clinical practices change. Post-market surveillance requirements for imaging AI are still evolving, and systematic monitoring of deployed tools is not standard practice at most institutions.
• Human-AI interaction effects. How radiologists integrate AI outputs into their decision-making — whether they anchor on AI findings, dismiss them, or selectively attend to them — affects realized clinical benefit in ways that performance benchmarks do not capture.