TY - JOUR
T1 - How best to quantify replication success?
T2 - A simulation study on the comparison of replication success metrics
AU - Muradchanian, Jasmine
AU - Hoekstra, Rink
AU - Kiers, Henk
AU - van Ravenzwaaij, Don
N1 - © 2021 The Authors.
PY - 2021
Y1 - 2021
N2 - To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.
AB - To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.
U2 - 10.1098/rsos.201697
DO - 10.1098/rsos.201697
M3 - Article
C2 - 34017596
SN - 2054-5703
VL - 8
JO - Royal Society Open Science
JF - Royal Society Open Science
IS - 5
M1 - 201697
ER -