Subjective evidence evaluation survey for many-analysts studies

Alexandra Sarafoglou*, Suzanne Hoogeveen, Don van den Bergh, Balazs Aczel, Casper J. Albers, Tim Althoff, Rotem Botvinik-Nezer, Niko A. Busch, Andrea M. Cataldo, Berna Devezer, Noah N.N. van Dongen, Anna Dreber, Eiko I. Fried, Rink Hoekstra, Sabine Hoffmann, Felix Holzmeister, Jürgen Huber, Nick Huntington-Klein, John Ioannidis, Magnus JohannessonMichael Kirchler, Eric Loken, Jean-Francois Mangin, Dora Matzke, Albert J. Menkveld, Gustav Nilsonne, Don van Ravenzwaaij, Martin Schweinsberg, Hannah Schulz-Kuempel, David R. Shanks, Daniel J Simons, Barbara A Spellman, Andrea H. Stoevenbelt, Barnabas Szaszi, Darinka Trübutschek, Francis Tuerlinckx, Eric L. Uhlmann, Wolf Vanpaemel, Jelte Wicherts, Eric-Jan Wagenmakers

*Corresponding author voor dit werk

OnderzoeksoutputAcademicpeer review

2 Citaten (Scopus)
14 Downloads (Pure)

Samenvatting

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
Originele taal-2English
Artikelnummer240125
Aantal pagina's28
TijdschriftRoyal Society Open Science
Volume11
Nummer van het tijdschrift7
DOI's
StatusPublished - jul.-2024

Vingerafdruk

Duik in de onderzoeksthema's van 'Subjective evidence evaluation survey for many-analysts studies'. Samen vormen ze een unieke vingerafdruk.

Citeer dit