Public Policy · Natural field experimentTaxpayers paying within 23 days (Experiment Two): Standard reminder letter 33.6% → Letter + local & debt norm 38.6% (+5.0 pts)
Sample: 119,527 taxpayers in Experiment Two; 101,471 in Experiment One
In a natural field experiment with 119,527 UK taxpayers who owed Self Assessment tax, 33.6% of those sent the standard HMRC letter paid within 23 days; adding a local-and-debt social norm raised payment by 5.0 percentage points, a 14.9% relative increase (Hallsworth et al., 2017).
Biases: Social Proof · Source: Hallsworth, M., List, J. A., Metcalfe, R. D., & Vlaev, I. (2017). The behavioralist as tax collector: Using natural field experiments to enhance tax compliance. Journal of Public Economics, 148, 14-31.
Full case study, design and limitations →Finance · Before–after (participants)Average saving rate of SMarT participants (% of pay): Before joining SMarT 3.5% → After four pay raises (40 months) 13.6% (+10.1 pts)
Sample: First implementation at one midsize manufacturing firm: 315 eligible employees; 162 joined SMarT (78% of those offered the plan)
In the first implementation of Save More Tomorrow at a midsize US manufacturing company, 78% of the employees offered the plan joined, and their average saving rate rose from 3.5% to 13.6% of pay over 40 months and four pay raises (Thaler & Benartzi, 2004). This is a before-and-after comparison of participants, not a randomized experiment.
Biases: Present Bias, Loss Aversion, Status Quo Bias · Source: Thaler, R. H., & Benartzi, S. (2004). Save More Tomorrow: Using behavioral economics to increase employee saving. Journal of Political Economy, 112(S1), S164-S187.
Full case study, design and limitations →Healthcare · Online experimentRespondents agreeing to be organ donors (online experiment): Opt-in default 42% → Opt-out default 82% (+40.0 pts)
Sample: 161 online respondents; cross-country consent data for European countries
In Johnson and Goldstein's (2003) online experiment with 161 respondents, 42% agreed to be organ donors when the default was not to donate (opt-in), compared with 82% when donation was the default (opt-out) and 79% when they had to choose with no default.
Biases: Default Effect, Status Quo Bias · Source: Johnson, E. J., & Goldstein, D. (2003). Do defaults save lives? Science, 302(5649), 1338-1339.
Full case study, design and limitations →Energy · Natural field experimentHousehold energy use relative to the randomized control group: Households receiving reports: −2.0% vs. control households without reports
Sample: About 600,000 treatment and control households across the United States
Using randomized natural field experiments with about 600,000 treatment and control households across the United States, Allcott (2011) estimates that Opower's Home Energy Reports cut energy use by 2.0% on average, from 6.3% in the highest-usage decile to 0.3% in the lowest.
Biases: Social Proof · Source: Allcott, H. (2011). Social norms and energy conservation. Journal of Public Economics, 95(9-10), 1082-1095.
Full case study, design and limitations →Healthcare · Randomized experiment · Not a nudgeAverage deviation from the diagnostic standard (0-100 probability points; lower is better): X-ray only (control mean) 22.6 → With AI predictions 22.9 (+0.3 pts)
Sample: 227 radiologists; 324 patient cases; 41,920 observations
In a pre-registered experiment with 227 professional radiologists reading 324 historical chest X-ray cases, giving them AI predictions that were more accurate than 78% of the participants did not improve their average diagnostic performance, while giving them clinical history reduced their error by 4.0% (Agarwal et al., 2023).
Biases: Algorithm Aversion, Correlation Neglect · Source: Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023). Combining human expertise with artificial intelligence: Experimental evidence from radiology. NBER Working Paper No. 31422.
Full case study, design and limitations →Public Policy · Retrospective ML evaluation · Not a nudgeDefendants jailed before trial, holding the crime rate at the judges' level (policy simulation): Judges' actual decisions 26.4% → Algorithmic release rule (simulated) 15.3% (−11.1 pts)
Sample: 758,027 New York City cases (2008-2013); 110,938-case hold-out test set
Using 758,027 New York City cases, Kleinberg et al. (2018) estimate that an algorithmic release rule could have cut the pre-trial detention rate from 26.4% to 15.3% (a 41.9% decline) without raising crime, or cut crime by 24.7% while jailing the same share of defendants. These figures come from a retrospective simulation, not a field trial.
Source: Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018). Human decisions and machine predictions. Quarterly Journal of Economics, 133(1), 237-293.
Full case study, design and limitations →Technology · Lab experimentParticipants choosing to use the algorithm's forecasts (Study 1): Algorithm could not be changed 32% → Could adjust each forecast by up to 10 percentiles 76% (+44.0 pts)
Sample: Three incentivized experiments: Study 1 in the Wharton Behavioral Lab; Studies 2-3 online (Study 2: 816 participants)
In Study 1 of Dietvorst, Simmons and Massey (2018), 32% of participants chose to use an imperfect forecasting algorithm they could not change, compared with 76% when they could adjust its forecasts by up to 10 percentiles and 73% when they could override up to 10 of its 20 forecasts. Those who could modify it performed better as a result.
Biases: Algorithm Aversion · Source: Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155-1170.
Full case study, design and limitations →