Examining the replicability of online experiments selected by a decision market

Felix Holzmeister; Magnus Johannesson; Colin F. Camerer; Yiling Chen; Teck Hua Ho; Suzanne Hoogeveen; Juergen Huber; Noriko Imai; Taisuke Imai; Lawrence Jin; Michael Kirchler; Alexander Ly; Benjamin Mandl; Dylan Manfredi; Gideon Nave; Brian A. Nosek; Thomas Pfeiffer; Alexandra Sarafoglou; Rene Schwaiger; Eric Jan Wagenmakers; Viking Waldén; Anna Dreber

doi:10.1038/s41562-024-02062-9

Examining the replicability of online experiments selected by a decision market

Felix Holzmeister
, Magnus Johannesson
, Colin F. Camerer
, Yiling Chen
, Teck Hua Ho
, Suzanne Hoogeveen
, Juergen Huber
, Noriko Imai
, Taisuke Imai
, Lawrence Jin
, Michael Kirchler
, Alexander Ly
, Benjamin Mandl
, Dylan Manfredi
, Gideon Nave
, Brian A. Nosek
, Thomas Pfeiffer
, Alexandra Sarafoglou
, Rene Schwaiger
, Eric Jan Wagenmakers

Viking Waldén, Anna Dreber^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.

Original language	English
Article number	e124
Pages (from-to)	316–330
Number of pages	15
Journal	Nature Human Behaviour
Volume	9
Issue number	2
Early online date	19 Nov 2024
DOIs	https://doi.org/10.1038/s41562-024-02062-9
Publication status	Published - 2025

Bibliographical note

Publisher Copyright:
© The Author(s) 2024.

Funding

We thank A. Andevall for helping with the data collection and programming of experiments and R. Willer for helpful advice on defining the IP address check and exclusion criteria used to exclude individuals from participating to minimize low-quality participant data. For financial support, we thank the Austrian Science FWF (grant SFB F63 to J.H. and M.K.), Jan Wallander and Tom Hedelius Foundation (grants P21-0091 and P23-0098 to A.D.), Knut and Alice Wallenberg Foundation and Marianne and Marcus Wallenberg Foundation (Wallenberg Scholar grant to A.D.) and Riksbankens Jubileumsfond (grant P21-0168 to M.J.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the paper. One author (V.W.) is currently employed by Sveriges Riksbank but did this work before being employed by Sveriges Riksbank; the opinions expressed in this article are the sole responsibility of the authors and should not be interpreted as reflecting the views of Sveriges Riksbank.

Funders	Funder number
Jan Wallanders och Tom Hedelius Stiftelse samt Tore Browaldhs Stiftelse (Jan Wallander and Tom Hedelius Foundation and Tore Browaldh Foundation)	SFB F63
Austrian Science FWF	P21-0091, P23-0098
Jan Wallander and Tom Hedelius Foundation
Knut and Alice Wallenberg foundation
Marianne and Marcus Wallenberg Foundation (Wallenberg Scholar grant)	P21-0168
Riksbankens Jubileumsfond

Access to Document

10.1038/s41562-024-02062-9Licence: CC BY

s41562-024-02062-9Final published version, 10 MBLicence: CC BY

Cite this

Holzmeister, F., Johannesson, M., Camerer, C. F., Chen, Y., Ho, T. H., Hoogeveen, S., Huber, J., Imai, N., Imai, T., Jin, L., Kirchler, M., Ly, A., Mandl, B., Manfredi, D., Nave, G., Nosek, B. A., Pfeiffer, T., Sarafoglou, A., Schwaiger, R., ... Dreber, A. (2025). Examining the replicability of online experiments selected by a decision market. Nature Human Behaviour, 9(2), 316–330. Article e124. https://doi.org/10.1038/s41562-024-02062-9

@article{199095273dd6478b851e4994320566d0,

title = "Examining the replicability of online experiments selected by a decision market",

abstract = "Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83\% for the top-12 and 33\% for the bottom-12 group. Overall, 54\% of the studies were successfully replicated, with replication effect size estimates averaging 45\% of the original effect size estimates. The replication rate varied between 54\% and 62\% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.",

author = "Felix Holzmeister and Magnus Johannesson and Camerer, \{Colin F.\} and Yiling Chen and Ho, \{Teck Hua\} and Suzanne Hoogeveen and Juergen Huber and Noriko Imai and Taisuke Imai and Lawrence Jin and Michael Kirchler and Alexander Ly and Benjamin Mandl and Dylan Manfredi and Gideon Nave and Nosek, \{Brian A.\} and Thomas Pfeiffer and Alexandra Sarafoglou and Rene Schwaiger and Wagenmakers, \{Eric Jan\} and Viking Wald{\'e}n and Anna Dreber",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2024.",

year = "2025",

doi = "10.1038/s41562-024-02062-9",

language = "English",

volume = "9",

pages = "316–330",

journal = "Nature Human Behaviour",

issn = "2397-3374",

publisher = "Nature Publishing Group",

number = "2",

}

Holzmeister, F, Johannesson, M, Camerer, CF, Chen, Y, Ho, TH, Hoogeveen, S, Huber, J, Imai, N, Imai, T, Jin, L, Kirchler, M, Ly, A, Mandl, B, Manfredi, D, Nave, G, Nosek, BA, Pfeiffer, T, Sarafoglou, A, Schwaiger, R, Wagenmakers, EJ, Waldén, V & Dreber, A 2025, 'Examining the replicability of online experiments selected by a decision market', Nature Human Behaviour, vol. 9, no. 2, e124, pp. 316–330. https://doi.org/10.1038/s41562-024-02062-9

TY - JOUR

T1 - Examining the replicability of online experiments selected by a decision market

AU - Holzmeister, Felix

AU - Johannesson, Magnus

AU - Camerer, Colin F.

AU - Chen, Yiling

AU - Ho, Teck Hua

AU - Hoogeveen, Suzanne

AU - Huber, Juergen

AU - Imai, Noriko

AU - Imai, Taisuke

AU - Jin, Lawrence

AU - Kirchler, Michael

AU - Ly, Alexander

AU - Mandl, Benjamin

AU - Manfredi, Dylan

AU - Nave, Gideon

AU - Nosek, Brian A.

AU - Pfeiffer, Thomas

AU - Sarafoglou, Alexandra

AU - Schwaiger, Rene

AU - Wagenmakers, Eric Jan

AU - Waldén, Viking

AU - Dreber, Anna

PY - 2025

Y1 - 2025

N2 - Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.

AB - Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.

UR - https://www.scopus.com/pages/publications/85209596963

U2 - 10.1038/s41562-024-02062-9

DO - 10.1038/s41562-024-02062-9

M3 - Article

AN - SCOPUS:85209596963

SN - 2397-3374

VL - 9

SP - 316

EP - 330

JO - Nature Human Behaviour

JF - Nature Human Behaviour

IS - 2

M1 - e124

ER -

Examining the replicability of online experiments selected by a decision market

Abstract

Bibliographical note

Funding

Access to Document

Other files and links

Fingerprint

Cite this