Do Nudges Work? The Meta-Analysis and the Bias Critique

Key Points

  • A corrected 2021 PNAS meta-analysis of 447 effect sizes from 212 publications reported an overall nudge effect of Cohen's d = 0.43.
  • The authors reported publication bias, and their own sensitivity analyses put the effect at 0.31 under moderate bias and 0.08 under severe bias.
  • A July 19, 2022 letter in the same journal applied a Bayesian adjustment and reported a combined estimate of 0.04 and argued that no evidence of an effect remains.
  • In their reply, the original authors agreed that publication bias is a problem and said their article was not meant as the final word.

Published .

Close view of a gray-haired man in a light blue button-down shirt and patterned tie, speaking, with a small clip-on microphone on his tie.
Economist Richard Thaler, photographed for the NIH Record between January 20 and February 3, 2012. Photo: Ernie Branson for the National Institutes of Health Record, public domain, via Wikimedia Commons, cropped and resized, photo source.

In a December 30, 2021 paper in the Proceedings of the National Academy of Sciences, Stephanie Mertens and colleagues pooled the results of studies on nudges, changes to how choices are designed, and reported that they change behavior by a small to medium amount1. A letter published in the same journal on July 19, 2022 reanalyzed the same data and argued that once publication bias is corrected for, no evidence remains that nudges are effective3. The sources below show what each side measured and where they agree.

What a nudge is

The paper describes nudging, or choice architecture, as an approach that aims to change behavior by redesigning the physical, social or psychological environment in which people make decisions while preserving their freedom of choice1. It credits Thaler and Sunstein's 2008 book Nudge: Improving Decisions about Health, Wealth, and Happiness with introducing the approach widely, and notes that choice architecture interventions have seen an immense increase in popularity since that first edition1.

The 2021 meta-analysis, as corrected

After a 2022 correction, the analysis covers 447 effect sizes from 212 publications with a combined sample of 2,148,439, and puts the overall effect at Cohen’s d = 0.43, with a 95 percent confidence interval from 0.38 to 0.482. Using conventional criteria, the authors classify that as a small to medium effect2.

The correction notes that the original sample had included four observations from a 2012 paper by Shu and colleagues that has since been retracted, erroneous values for two observations from another paper, and several coding errors2. It replaces the 0.45 and the other figures in the abstract and the results with the revised ones, including 0.432.

In the corrected table, the techniques that target the organization of choices (decision structure) average d = 0.54, against 0.34 for decision information and 0.28 for decision assistance, and the default technique has the highest estimate among the nine techniques at 0.622. By domain, food is highest at 0.65 and finance lowest at 0.242.

The publication bias warning

The authors reported an Egger's test of b = 2.10 (95 percent confidence interval 1.31 to 2.89), which they read as a publication bias favoring the reporting of successful implementations4. The funnel plot caption in the correction describes an asymmetric distribution that favors successful results in studies with small sample sizes2. Their sensitivity analyses suggested that the overall effect would shrink to d = 0.31 under moderate publication bias and to d = 0.08 under severe publication bias4. The original abstract itself reported a moderate publication bias toward positive results in the literature1.

The 2022 reanalysis

The letter by Maximilian Maier, Frantisek Bartos, T. D. Stanley, David Shanks, Adam Harris and Eric-Jan Wagenmakers applied a robust Bayesian meta-analysis designed to adjust for publication bias3. For the combined sample the adjusted estimate was 0.04 (credible interval 0.00 to 0.14), against 0.43 in the unadjusted random effects model, and the Bayes factor for the absence of an effect was 0.95, which the letter reads as an absence of evidence for an overall effect3.

By intervention category, the adjusted estimate was 0.00 for information interventions (Bayes factor 33.84 in favor of no effect) and 0.01 for assistance interventions (9.05), while the letter calls the evidence for structure interventions undecided, with an estimate of 0.12 and a Bayes factor of 1.123. Using only the most precise estimate from each study, the combined estimate was 0.11 (credible interval 0.00 to 0.24) and the letter describes weak evidence for an overall effect3. The letter also reports strong evidence for publication bias in every subdomain except food when using only the most precise estimates3.

The authors add that all categories and domains except finance show evidence of heterogeneity, which they say implies that some nudges might be effective even when the average effect is not supported3.

The original authors respond

In their reply, Mertens and colleagues write that their analysis points to a publication bias favoring successful results and that the three letters underline that issue; they also write that the letters report smaller effect sizes or even null effects under robust Bayesian meta-analysis, Andrews-Kasy, WAAP or trim and fill methods4. They state that the objective of their article was not to write the final word on choice architecture interventions, and they agree that publication bias in the literature is problematic4. They call for appropriately powered studies and open science practices such as preregistered reports and the publication of null results4.

What is and is not established

The sources support these statements. A pooled analysis of 447 effect sizes reported an average effect of d = 0.43 before any bias adjustment. Both the original authors and the critics agree that publication bias is present in this literature. A robust Bayesian reanalysis puts the bias adjusted average near zero, with the strength of that conclusion differing by category and by how the estimates are chosen.

The sources do not establish that no nudge works. The reanalysis itself reports evidence of heterogeneity, and the original authors point to unexplained variability in effect sizes. They also do not settle which adjustment method is right, since the reply and the letter present different readings of the same data. Only these three papers and the correction were opened for this article, so the other two letters named in the reply are not described here.

Trivia

  1. In which year was the first edition of Thaler and Sunstein's Nudge published? 20081.
  2. Which journal published the 2021 meta-analysis of choice architecture interventions and the 2022 letter that reanalyzed it? Proceedings of the National Academy of Sciences3.
  3. After the 2022 correction, what was the meta-analysis's overall effect size, in Cohen's d? 0.432.
  4. Which of the nine techniques had the highest estimate in the corrected table? Defaults (d = 0.62)2.
  5. How many publications does the corrected analysis include? 2122.

Sources

  1. Stephanie Mertens, Mario Herberz, Ulf J. J. Hahnel and Tobias Brosch, The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains, Proceedings of the National Academy of Sciences, 119(1): e2107346118, December 30, 2021. Accessed November 17, 2026.
  2. Correction for Mertens et al., The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains, Proceedings of the National Academy of Sciences, 119(19): e2204059119, 2022. Accessed November 17, 2026.
  3. Maximilian Maier, Frantisek Bartos, T. D. Stanley, David R. Shanks, Adam J. L. Harris and Eric-Jan Wagenmakers, No evidence for nudging after adjusting for publication bias, Proceedings of the National Academy of Sciences, 119(31): e2200300119, July 19, 2022. Accessed November 17, 2026.
  4. Stephanie Mertens, Mario Herberz, Ulf J. J. Hahnel and Tobias Brosch, Reply to Maier et al., Szaszi et al., and Bakdash and Marusich: The present and future of choice architecture research, Proceedings of the National Academy of Sciences, 119(31): e2202928119, July 19, 2022. Accessed November 17, 2026.