Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Nikhil Agarwal, Alex Moehring, Pranav Rajpurkar, Tobias Salz
NBER Working Paper Series · 2023
Edited by Paweł Raja, PhD · Published · Updated
Paper details
- Authors
- Nikhil Agarwal, Alex Moehring, Pranav Rajpurkar, Tobias Salz
- Published
- 2023, NBER Working Paper No. 31422
- DOI
- 10.3386/w31422
- Study design
- Controlled experiment (lab or online)
- Full text
- Free PDF · Free PDF of the working-paper version (NBER Working Paper No. 31422), hosted at nber.org. Cite the published version and check page numbers against it.
Summary
Examines how expert radiologists combine their own judgment with recommendations from an AI model. The authors find that radiologists treat AI predictions as independent information and underweight them, which often leads to collaborative results that underperform the AI model alone.
Methodology
A randomized trial where radiologists evaluated chest X-rays. Radiologists in the treatment group were shown AI-generated probability scores of pathologies, and their diagnostic accuracy was compared to both human-alone and AI-alone baselines.
Findings
Radiologists consistently underweighted AI inputs, treating them as if they were uncorrelated with their own signals. This correlation neglect and conservatism meant that the joint human-AI diagnostic accuracy was lower than the AI model working in isolation.
Applied nudge
Design joint human-AI workflows so that experts are nudged to treat the AI's inputs not as separate opinions, but as integrated priors or structured checklists.
How to cite this paper
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023). Combining human expertise with artificial intelligence: Experimental evidence from radiology. National Bureau of Economic Research, Working Paper No. 31422. https://doi.org/10.3386/w31422
BibTeX
@techreport{agarwal2023combining,
author = {Nikhil Agarwal and Alex Moehring and Pranav Rajpurkar and Tobias Salz},
title = {Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology},
institution = {NBER Working Paper Series},
type = {Working Paper},
number = {31422},
year = {2023},
doi = {10.3386/w31422}
}Cite this page
Raja, P. (Ed.). (2026, 19 Oct 2026). Summary: Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology. Behavioral Economics Lab. https://www.behavioraleconomicslab.com/journals/radiology-ai-collaboration
Where this research is used
More research: AI & Trust
- Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them ErrBerkeley J. Dietvorst, Joseph P. Simmons, Cade Massey (2015)
- Algorithm Appreciation: People Prefer Algorithmic to Human JudgmentJennifer M. Logg, Julia A. Minson, Don A. Moore (2019)
- Human Trust in Artificial Intelligence: Review of Empirical ResearchElla Glikson, Anita Williams Woolley (2020)
- Human Decisions and Machine PredictionsJon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan (2018)