Skip to content
Healthcare SectorDesign: Randomized experimentNot a nudge

AI Assistance in Radiology: Why AI Predictions Did Not Improve Radiologists' Accuracy

Edited by Paweł Raja, PhD · Published · Updated

In a pre-registered experiment with 227 professional radiologists reading 324 historical chest X-ray cases, giving them AI predictions that were more accurate than 78% of the participants did not improve their average diagnostic performance, while giving them clinical history reduced their error by 4.0% (Agarwal et al., 2023).

Average deviation from the diagnostic standard (0-100 probability points; lower is better) · Randomized experiment

X-ray only (control mean): 22.6 → With AI predictions: 22.9 (+0.3 pts)

The estimated AI effect is +0.3 points (standard error 0.4), not statistically significant: AI assistance did not improve accuracy on average.

The intervention did not improve this outcome.

Study details

Study design
Randomized experiment
Problem
AI tools can read some chest X-rays as well as or better than many radiologists, yet they are mostly used as assistants rather than replacements. That only helps if radiologists combine the AI's prediction with their own judgment well. Agarwal, Moehring, Rajpurkar and Salz tested whether they do.
What was tested
Not a nudge: this experiment varied the information radiologists had. They diagnosed real historical chest X-ray cases under randomly assigned conditions: the X-ray alone, the X-ray plus AI predictions (a probability for each pathology from a deep-learning model), the X-ray plus clinical history, or both.
Outcome
Average deviation from the diagnostic standard (0-100 probability points; lower is better): 22.6 (X-ray only (control mean)) → 22.9 (With AI predictions); +0.3 pts
Sample
227 radiologists; 324 patient cases; 41,920 observations
Key takeaway
Radiologists did move toward the AI's predictions, but they underweighted them and treated the AI's information as independent of their own reading of the same X-ray. AI help improved decisions when the AI was confident and hurt them when it was uncertain, so the average gain was about zero. Unless those errors can be corrected, the authors conclude, cases should go to the AI or to a radiologist, but rarely to a radiologist with AI assistance.
Source
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023). Combining human expertise with artificial intelligence: Experimental evidence from radiology. NBER Working Paper No. 31422. doi:10.3386/w31422 ↗ full text (PDF)Numbers checked against: Abstract; Section 4 and Figure 1 (AI beats 78% of radiologists); Table C.6, col. 4 (control mean 0.226; AI effect 0.003, SE 0.004)

What the study did

Participants were recruited mainly through teleradiology companies, plus staff radiologists from the VinMec health system in Vietnam. About 14% were US-based, and all routinely diagnose US patients. The AI was the CheXpert model of Irvin et al. (2019), trained on about 250,000 X-rays. The "right answer" for each case was a diagnostic standard built from the assessments of five board-certified chest radiologists, not an observed ground truth, because definitive tests do not exist for most chest pathologies.

Paying radiologists at market rates makes large samples expensive, so the study used within-participant designs. Each radiologist saw cases under the different information conditions in random order. In one design, each radiologist read the same patient in all four conditions, at least two weeks apart. Some participants were randomly offered bonuses for accuracy, which did not change the results.

With AI predictions, radiologists' reports moved about 18% closer to the AI's, so they did not ignore it. Their average deviation from the diagnostic standard, however, did not change significantly (Table C.6). On its own, the AI (AUC 0.85) was more accurate than 78% of participants. The authors estimate that an optimal combination of human and AI signals would have reached an AUC of 0.88.

Using cases read both with and without AI, the authors estimate how radiologists update their beliefs. Two errors stand out. Radiologists give the AI's prediction less weight than a Bayesian benchmark would, and they treat it as independent of their own reading even though both come from the same image. As a result, AI assistance improved diagnoses when the AI was confident and made them worse when it was uncertain.

Limitations and caveats

  • This is an experiment on retrospective cases in a research interface, not a study of a deployed tool. Clinical workflow, liability and time pressure could change how radiologists use AI in practice.
  • Accuracy is measured against a panel-based diagnostic standard, not a confirmed diagnosis.
  • The results concern one AI model and chest X-rays. Better-calibrated AI output, or training radiologists to update correctly, could change the conclusion; the authors' recommended design depends on whether these errors can be corrected.
  • The source is an NBER working paper (No. 31422, 2023, revised August 2026). We used the revised version.

Further sources

Cite this page

Raja, P. (Ed.). (2026, 8 Oct 2026). AI Assistance in Radiology: Why AI Predictions Did Not Improve Radiologists' Accuracy. Behavioral Economics Lab. https://www.behavioraleconomicslab.com/findings/radiology-ai-diagnostic

Primary source: Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023). Combining human expertise with artificial intelligence: Experimental evidence from radiology. NBER Working Paper No. 31422.