Predicting and reasoning about replicability using structured groups

Bonnie C. Wintle; Eden T. Smith; Martin Bush; Fallon Mody; David P. Wilkinson; Anca M. Hanea; Alexandru Marcoci; Hannah Fraser; Victoria Hemming; Felix Singleton Thorn; Marissa McBride; Elliot Gould; Andrew Head; Daniel G.  Hamilton; Steven Kambouris; Libby Rumpff; Rink Hoekstra; Mark Burgman; Fiona Fidler

doi:10.1098/rsos.221553

Predicting and reasoning about replicability using structured groups

Bonnie C. Wintle^*, Eden T. Smith, Martin Bush, Fallon Mody, David P. Wilkinson, Anca M. Hanea, Alexandru Marcoci, Hannah Fraser, Victoria Hemming, Felix Singleton Thorn, Marissa McBride, Elliot Gould, Andrew Head, Daniel G. Hamilton, Steven Kambouris, Libby Rumpff, Rink Hoekstra, Mark Burgman, Fiona Fidler

^*Corresponding author for this work

Research and Evaluation of Educational Effectiveness

Research output: Contribution to journal › Article › Academic › peer-review

2 Citations (Scopus)

37 Downloads (Pure)

Abstract

This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol (‘investigate’, ‘discuss’, ‘estimate’ and ‘aggregate’). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion.

Original language	English
Article number	221553
Number of pages	25
Journal	Royal Society Open Science
Volume	10
Issue number	6
DOIs	https://doi.org/10.1098/rsos.221553
Publication status	Published - Jun-2023

Access to Document

10.1098/rsos.221553Licence: CC BY

wintle-et-al-2023-predicting-and-reasoning-about-replicability-using-structured-groupsFinal publisher's version, 1.5 MBLicence: CC BY

Handle.net

Persistent link

Cite this

Wintle, B. C., Smith, E. T., Bush, M., Mody, F., Wilkinson, D. P., Hanea, A. M., Marcoci, A., Fraser, H., Hemming, V., Thorn, F. S., McBride, M., Gould, E., Head, A., Hamilton, D. G., Kambouris, S., Rumpff, L., Hoekstra, R., Burgman, M., & Fidler, F. (2023). Predicting and reasoning about replicability using structured groups. Royal Society Open Science, 10(6), Article 221553. https://doi.org/10.1098/rsos.221553

@article{338517a81ae4445db9bb6e1a5ccf4e04,

title = "Predicting and reasoning about replicability using structured groups",

abstract = "This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol ({\textquoteleft}investigate{\textquoteright}, {\textquoteleft}discuss{\textquoteright}, {\textquoteleft}estimate{\textquoteright} and {\textquoteleft}aggregate{\textquoteright}). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion.",

author = "Wintle, {Bonnie C.} and Smith, {Eden T.} and Martin Bush and Fallon Mody and Wilkinson, {David P.} and Hanea, {Anca M.} and Alexandru Marcoci and Hannah Fraser and Victoria Hemming and Thorn, {Felix Singleton} and Marissa McBride and Elliot Gould and Andrew Head and Hamilton, {Daniel G.} and Steven Kambouris and Libby Rumpff and Rink Hoekstra and Mark Burgman and Fiona Fidler",

year = "2023",

month = jun,

doi = "10.1098/rsos.221553",

language = "English",

volume = "10",

journal = "Royal Society Open Science",

issn = "2054-5703",

publisher = "ROYAL SOC",

number = "6",

}

Wintle, BC, Smith, ET, Bush, M, Mody, F, Wilkinson, DP, Hanea, AM, Marcoci, A, Fraser, H, Hemming, V, Thorn, FS, McBride, M, Gould, E, Head, A, Hamilton, DG, Kambouris, S, Rumpff, L, Hoekstra, R, Burgman, M & Fidler, F 2023, 'Predicting and reasoning about replicability using structured groups', Royal Society Open Science, vol. 10, no. 6, 221553. https://doi.org/10.1098/rsos.221553

TY - JOUR

T1 - Predicting and reasoning about replicability using structured groups

AU - Wintle, Bonnie C.

AU - Smith, Eden T.

AU - Bush, Martin

AU - Mody, Fallon

AU - Wilkinson, David P.

AU - Hanea, Anca M.

AU - Marcoci, Alexandru

AU - Fraser, Hannah

AU - Hemming, Victoria

AU - Thorn, Felix Singleton

AU - McBride, Marissa

AU - Gould, Elliot

AU - Head, Andrew

AU - Hamilton, Daniel G.

AU - Kambouris, Steven

AU - Rumpff, Libby

AU - Hoekstra, Rink

AU - Burgman, Mark

AU - Fidler, Fiona

PY - 2023/6

Y1 - 2023/6

N2 - This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol (‘investigate’, ‘discuss’, ‘estimate’ and ‘aggregate’). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion.

AB - This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol (‘investigate’, ‘discuss’, ‘estimate’ and ‘aggregate’). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion.

U2 - 10.1098/rsos.221553

DO - 10.1098/rsos.221553

M3 - Article

SN - 2054-5703

VL - 10

JO - Royal Society Open Science

JF - Royal Society Open Science

IS - 6

M1 - 221553

ER -

Predicting and reasoning about replicability using structured groups

Abstract

Access to Document

Handle.net

Fingerprint

Cite this