Skip to content
AI & Trust

Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology

Nikhil Agarwal, Alex Moehring, Pranav Rajpurkar, Tobias Salz

NBER Working Paper Series · 2023

Edited by Paweł Raja, PhD · Published · Updated

Paper details

Authors
Nikhil Agarwal, Alex Moehring, Pranav Rajpurkar, Tobias Salz
Published
2023, NBER Working Paper No. 31422
DOI
10.3386/w31422
Study design
Controlled experiment (lab or online)
Full text
Free PDF · Free PDF of the working-paper version (NBER Working Paper No. 31422), hosted at nber.org. Cite the published version and check page numbers against it.

Summary

Examines how expert radiologists combine their own judgment with recommendations from an AI model. The authors find that radiologists treat AI predictions as independent information and underweight them, which often leads to collaborative results that underperform the AI model alone.

Methodology

A randomized trial where radiologists evaluated chest X-rays. Radiologists in the treatment group were shown AI-generated probability scores of pathologies, and their diagnostic accuracy was compared to both human-alone and AI-alone baselines.

Findings

Radiologists consistently underweighted AI inputs, treating them as if they were uncorrelated with their own signals. This correlation neglect and conservatism meant that the joint human-AI diagnostic accuracy was lower than the AI model working in isolation.

Applied nudge

Design joint human-AI workflows so that experts are nudged to treat the AI's inputs not as separate opinions, but as integrated priors or structured checklists.

How to cite this paper

Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023). Combining human expertise with artificial intelligence: Experimental evidence from radiology. National Bureau of Economic Research, Working Paper No. 31422. https://doi.org/10.3386/w31422

BibTeX
@techreport{agarwal2023combining,
  author = {Nikhil Agarwal and Alex Moehring and Pranav Rajpurkar and Tobias Salz},
  title = {Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology},
  institution = {NBER Working Paper Series},
  type = {Working Paper},
  number = {31422},
  year = {2023},
  doi = {10.3386/w31422}
}

Cite this page

Raja, P. (Ed.). (2026, 19 Oct 2026). Summary: Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology. Behavioral Economics Lab. https://www.behavioraleconomicslab.com/journals/radiology-ai-collaboration

Where this research is used

More research: AI & Trust