Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel; Barnabas Szaszi; Gustav Nilsonne; Olmo R van den Akker; Casper J Albers; Marcel Alm van Assen; Jojanneke A Bastiaansen; Daniel Benjamin; Udo Boehm; Rotem Botvinik-Nezer; Laura F Bringmann; Niko A Busch; Emmanuel Caruyer; Andrea M Cataldo; Nelson Cowan; Andrew Delios; Noah N N van Dongen; Chris Donkin; Johnny B van Doorn; Ann Dreber; Gilles Dutilh; Gary F Egan; Morton Ann Gernsbacher; Rink Hoekstra; Sabine Hoffmann; Felix Holzmeister; Juergen Huber; Magnus Johannesson; Kai J Jonas; Alexander T Kindel; Michael Kirchler; Yoram K Kunkels; D Stephen Lindsay; Jean-Francois Mangin; Dora Matzke; Marcus R Munafò; Ben R Newell; Brian A Nosek; Russell A Poldrack; Don van Ravenzwaaij; Jörg Rieskamp; Matthew J Salganik; Alexandra Sarafoglou; Tom Schonberg; Martin Schweinsberg; David Shanks; Raphael Silberzahn; Daniel J Simons; Barbara A Spellman; Samuel St-Jean; Jeffrey J Starns; Eric Luis Uhlmann; Jelte Wicherts; Eric-Jan Wagenmakers

doi:10.7554/eLife.72185

Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel^*, Barnabas Szaszi^*, Gustav Nilsonne, Olmo R van den Akker, Casper J Albers, Marcel Alm van Assen, Jojanneke A Bastiaansen, Daniel Benjamin, Udo Boehm, Rotem Botvinik-Nezer, Laura F Bringmann, Niko A Busch, Emmanuel Caruyer, Andrea M Cataldo, Nelson Cowan, Andrew Delios, Noah N N van Dongen, Chris Donkin, Johnny B van Doorn, Ann DreberGilles Dutilh, Gary F Egan, Morton Ann Gernsbacher, Rink Hoekstra, Sabine Hoffmann, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Kai J Jonas, Alexander T Kindel, Michael Kirchler, Yoram K Kunkels, D Stephen Lindsay, Jean-Francois Mangin, Dora Matzke, Marcus R Munafò, Ben R Newell, Brian A Nosek, Russell A Poldrack, Don van Ravenzwaaij, Jörg Rieskamp, Matthew J Salganik, Alexandra Sarafoglou, Tom Schonberg, Martin Schweinsberg, David Shanks, Raphael Silberzahn, Daniel J Simons, Barbara A Spellman, Samuel St-Jean, Jeffrey J Starns, Eric Luis Uhlmann, Jelte Wicherts, Eric-Jan Wagenmakers

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

26 Citations (Scopus)

113 Downloads (Pure)

Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Original language	English
Article number	72185
Pages (from-to)	1-17
Number of pages	13
Journal	eLife
Volume	10
Early online date	9-Nov-2021
DOIs	https://doi.org/10.7554/eLife.72185
Publication status	Published - 9-Nov-2021

Keywords

multi-analyst
metascience
statistical practice
science forum
expert consensus
analytical variability
None
REPRODUCIBILITY
DELPHI

Access to Document

10.7554/eLife.72185Licence: CC BY

Consensus-based guidance for conducting and reporting multi-analyst studiesFinal publisher's version, 468 KBLicence: CC BY

Handle.net

Persistent link

Cite this

Aczel, B., Szaszi, B., Nilsonne, G., van den Akker, O. R., Albers, C. J., van Assen, M. A., Bastiaansen, J. A., Benjamin, D., Boehm, U., Botvinik-Nezer, R., Bringmann, L. F., Busch, N. A., Caruyer, E., Cataldo, A. M., Cowan, N., Delios, A., van Dongen, N. N. N., Donkin, C., van Doorn, J. B., ... Wagenmakers, E.-J. (2021). Consensus-based guidance for conducting and reporting multi-analyst studies. eLife, 10, 1-17. Article 72185. https://doi.org/10.7554/eLife.72185

@article{918aa9272d814727adf2f143993cbffe,

title = "Consensus-based guidance for conducting and reporting multi-analyst studies",

abstract = "Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.",

keywords = "multi-analyst, metascience, statistical practice, science forum, expert consensus, analytical variability, None, REPRODUCIBILITY, DELPHI",

author = "Balazs Aczel and Barnabas Szaszi and Gustav Nilsonne and {van den Akker}, {Olmo R} and Albers, {Casper J} and {van Assen}, {Marcel Alm} and Bastiaansen, {Jojanneke A} and Daniel Benjamin and Udo Boehm and Rotem Botvinik-Nezer and Bringmann, {Laura F} and Busch, {Niko A} and Emmanuel Caruyer and Cataldo, {Andrea M} and Nelson Cowan and Andrew Delios and {van Dongen}, {Noah N N} and Chris Donkin and {van Doorn}, {Johnny B} and Ann Dreber and Gilles Dutilh and Egan, {Gary F} and Gernsbacher, {Morton Ann} and Rink Hoekstra and Sabine Hoffmann and Felix Holzmeister and Juergen Huber and Magnus Johannesson and Jonas, {Kai J} and Kindel, {Alexander T} and Michael Kirchler and Kunkels, {Yoram K} and Lindsay, {D Stephen} and Jean-Francois Mangin and Dora Matzke and Munaf{\`o}, {Marcus R} and Newell, {Ben R} and Nosek, {Brian A} and Poldrack, {Russell A} and {van Ravenzwaaij}, Don and J{\"o}rg Rieskamp and Salganik, {Matthew J} and Alexandra Sarafoglou and Tom Schonberg and Martin Schweinsberg and David Shanks and Raphael Silberzahn and Simons, {Daniel J} and Spellman, {Barbara A} and Samuel St-Jean and Starns, {Jeffrey J} and Uhlmann, {Eric Luis} and Jelte Wicherts and Eric-Jan Wagenmakers",

note = "{\textcopyright} 2021, Aczel et al.",

year = "2021",

month = nov,

day = "9",

doi = "10.7554/eLife.72185",

language = "English",

volume = "10",

pages = "1--17",

journal = "eLife",

issn = "2050-084X",

publisher = "ELIFE SCIENCES PUBLICATIONS LTD",

}

Aczel, B, Szaszi, B, Nilsonne, G, van den Akker, OR, Albers, CJ, van Assen, MA, Bastiaansen, JA, Benjamin, D, Boehm, U, Botvinik-Nezer, R, Bringmann, LF, Busch, NA, Caruyer, E, Cataldo, AM, Cowan, N, Delios, A, van Dongen, NNN, Donkin, C, van Doorn, JB, Dreber, A, Dutilh, G, Egan, GF, Gernsbacher, MA, Hoekstra, R, Hoffmann, S, Holzmeister, F, Huber, J, Johannesson, M, Jonas, KJ, Kindel, AT, Kirchler, M, Kunkels, YK, Lindsay, DS, Mangin, J-F, Matzke, D, Munafò, MR, Newell, BR, Nosek, BA, Poldrack, RA, van Ravenzwaaij, D, Rieskamp, J, Salganik, MJ, Sarafoglou, A, Schonberg, T, Schweinsberg, M, Shanks, D, Silberzahn, R, Simons, DJ, Spellman, BA, St-Jean, S, Starns, JJ, Uhlmann, EL, Wicherts, J & Wagenmakers, E-J 2021, 'Consensus-based guidance for conducting and reporting multi-analyst studies', eLife, vol. 10, 72185, pp. 1-17. https://doi.org/10.7554/eLife.72185

TY - JOUR

T1 - Consensus-based guidance for conducting and reporting multi-analyst studies

AU - Aczel, Balazs

AU - Szaszi, Barnabas

AU - Nilsonne, Gustav

AU - van den Akker, Olmo R

AU - Albers, Casper J

AU - van Assen, Marcel Alm

AU - Bastiaansen, Jojanneke A

AU - Benjamin, Daniel

AU - Boehm, Udo

AU - Botvinik-Nezer, Rotem

AU - Bringmann, Laura F

AU - Busch, Niko A

AU - Caruyer, Emmanuel

AU - Cataldo, Andrea M

AU - Cowan, Nelson

AU - Delios, Andrew

AU - van Dongen, Noah N N

AU - Donkin, Chris

AU - van Doorn, Johnny B

AU - Dreber, Ann

AU - Dutilh, Gilles

AU - Egan, Gary F

AU - Gernsbacher, Morton Ann

AU - Hoekstra, Rink

AU - Hoffmann, Sabine

AU - Holzmeister, Felix

AU - Huber, Juergen

AU - Johannesson, Magnus

AU - Jonas, Kai J

AU - Kindel, Alexander T

AU - Kirchler, Michael

AU - Kunkels, Yoram K

AU - Lindsay, D Stephen

AU - Mangin, Jean-Francois

AU - Matzke, Dora

AU - Munafò, Marcus R

AU - Newell, Ben R

AU - Nosek, Brian A

AU - Poldrack, Russell A

AU - van Ravenzwaaij, Don

AU - Rieskamp, Jörg

AU - Salganik, Matthew J

AU - Sarafoglou, Alexandra

AU - Schonberg, Tom

AU - Schweinsberg, Martin

AU - Shanks, David

AU - Silberzahn, Raphael

AU - Simons, Daniel J

AU - Spellman, Barbara A

AU - St-Jean, Samuel

AU - Starns, Jeffrey J

AU - Uhlmann, Eric Luis

AU - Wicherts, Jelte

AU - Wagenmakers, Eric-Jan

PY - 2021/11/9

Y1 - 2021/11/9

N2 - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

AB - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

KW - multi-analyst

KW - metascience

KW - statistical practice

KW - science forum

KW - expert consensus

KW - analytical variability

KW - None

KW - REPRODUCIBILITY

KW - DELPHI

U2 - 10.7554/eLife.72185

DO - 10.7554/eLife.72185

M3 - Article

C2 - 34751133

SN - 2050-084X

VL - 10

SP - 1

EP - 17

JO - eLife

JF - eLife

M1 - 72185

ER -

Consensus-based guidance for conducting and reporting multi-analyst studies

Abstract

Keywords

Access to Document

Handle.net

Fingerprint

Cite this