Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

Anya Belz; Craig Thomson; Ehud Reiter; Gavin Abercrombie; Jose M. Alonso-Moral; Mohammad Arvan; Anouck Braggaar; Mark Cieliebak; Elizabeth Clark; Kees van Deemter; Tanvi Dinkar; Ondřej Dušek; Steffen Eger; Qixiang Fang; Mingqi Gao; Albert Gatt; Dimitra Gkatzia; Javier González-Corbelle; Dirk Hovy; Manuela Hürlimann; Takumi Ito; John D. Kelleher; Filip Klubička; Emiel Krahmer; Huiyuan Lai; Chris van der Lee; Yiru Li; Saad Mahamood; Margot Mieskes; Emiel van Miltenburg; Pablo Mosteiro; Malvina Nissim; Natalie Parde; Ondřej Plátek; Verena Rieser; Jie Ruan; Joel Tetreault; Antonio Toral; Xiaojun Wan; Leo Wanner; Lewis Watson; Diyi Yang

doi:10.18653/v1/2023.insights-1.1

Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

Anya Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M. Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, Tanvi Dinkar, Ondřej Dušek, Steffen Eger, Qixiang Fang, Mingqi Gao, Albert Gatt, Dimitra Gkatzia, Javier González-Corbelle, Dirk Hovy, Manuela HürlimannTakumi Ito, John D. Kelleher, Filip Klubička, Emiel Krahmer, Huiyuan Lai, Chris van der Lee, Yiru Li, Saad Mahamood, Margot Mieskes, Emiel van Miltenburg, Pablo Mosteiro, Malvina Nissim, Natalie Parde, Ondřej Plátek, Verena Rieser, Jie Ruan, Joel Tetreault, Antonio Toral, Xiaojun Wan, Leo Wanner, Lewis Watson, Diyi Yang

Computational Linguistics (CL)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

31 Citations (Scopus)

12 Downloads (Pure)

Abstract

We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.

Original language	English
Title of host publication	Proceedings of the Fourth Workshop on Insights from Negative Results in NLP
Editors	Shabnam Tafreshi, Reddy Akula Arjun, Joao Sedoc, Aleksandr Drozd, Anna Rogers, Anna Rumshisky
Publisher	Association for Computational Linguistics, ACL Anthology
Pages	1-10
Number of pages	10
ISBN (Electronic)	9781959429494
DOIs	https://doi.org/10.18653/v1/2023.insights-1.1
Publication status	Published - 2023
Event	4th Workshop on Insights from Negative Results in NLP, INSIGHTS 2023 - Dubrovnik, Croatia Duration: 5-May-2023 → …

Conference

Conference	4th Workshop on Insights from Negative Results in NLP, INSIGHTS 2023
Country/Territory	Croatia
City	Dubrovnik
Period	05/05/2023 → …

Access to Document

10.18653/v1/2023.insights-1.1Licence: CC BY

Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLPFinal publisher's version, 180 KBLicence: CC BY

Handle.net

Persistent link

Cite this

Belz, A., Thomson, C., Reiter, E., Abercrombie, G., Alonso-Moral, J. M., Arvan, M., Braggaar, A., Cieliebak, M., Clark, E., van Deemter, K., Dinkar, T., Dušek, O., Eger, S., Fang, Q., Gao, M., Gatt, A., Gkatzia, D., González-Corbelle, J., Hovy, D., ... Yang, D. (2023). Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP. In S. Tafreshi, R. Akula Arjun, J. Sedoc, A. Drozd, A. Rogers, & A. Rumshisky (Eds.), Proceedings of the Fourth Workshop on Insights from Negative Results in NLP (pp. 1-10). Association for Computational Linguistics, ACL Anthology. https://doi.org/10.18653/v1/2023.insights-1.1

Belz, Anya ; Thomson, Craig ; Reiter, Ehud et al. / Missing Information, Unresponsive Authors, Experimental Flaws : The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP. Proceedings of the Fourth Workshop on Insights from Negative Results in NLP. editor / Shabnam Tafreshi ; Reddy Akula Arjun ; Joao Sedoc ; Aleksandr Drozd ; Anna Rogers ; Anna Rumshisky. Association for Computational Linguistics, ACL Anthology, 2023. pp. 1-10

@inproceedings{7c7521b399c24dc491c1889e0720d3ef,

title = "Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP",

abstract = "We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.",

author = "Anya Belz and Craig Thomson and Ehud Reiter and Gavin Abercrombie and Alonso-Moral, {Jose M.} and Mohammad Arvan and Anouck Braggaar and Mark Cieliebak and Elizabeth Clark and {van Deemter}, Kees and Tanvi Dinkar and Ond{\v r}ej Du{\v s}ek and Steffen Eger and Qixiang Fang and Mingqi Gao and Albert Gatt and Dimitra Gkatzia and Javier Gonz{\'a}lez-Corbelle and Dirk Hovy and Manuela H{\"u}rlimann and Takumi Ito and Kelleher, {John D.} and Filip Klubi{\v c}ka and Emiel Krahmer and Huiyuan Lai and {van der Lee}, Chris and Yiru Li and Saad Mahamood and Margot Mieskes and {van Miltenburg}, Emiel and Pablo Mosteiro and Malvina Nissim and Natalie Parde and Ond{\v r}ej Pl{\'a}tek and Verena Rieser and Jie Ruan and Joel Tetreault and Antonio Toral and Xiaojun Wan and Leo Wanner and Lewis Watson and Diyi Yang",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 4th Workshop on Insights from Negative Results in NLP, INSIGHTS 2023 ; Conference date: 05-05-2023",

year = "2023",

doi = "10.18653/v1/2023.insights-1.1",

language = "English",

pages = "1--10",

editor = "Shabnam Tafreshi and {Akula Arjun}, Reddy and Joao Sedoc and Aleksandr Drozd and Anna Rogers and Anna Rumshisky",

booktitle = "Proceedings of the Fourth Workshop on Insights from Negative Results in NLP",

publisher = "Association for Computational Linguistics, ACL Anthology",

}

Belz, A, Thomson, C, Reiter, E, Abercrombie, G, Alonso-Moral, JM, Arvan, M, Braggaar, A, Cieliebak, M, Clark, E, van Deemter, K, Dinkar, T, Dušek, O, Eger, S, Fang, Q, Gao, M, Gatt, A, Gkatzia, D, González-Corbelle, J, Hovy, D, Hürlimann, M, Ito, T, Kelleher, JD, Klubička, F, Krahmer, E, Lai, H, van der Lee, C, Li, Y, Mahamood, S, Mieskes, M, van Miltenburg, E, Mosteiro, P, Nissim, M, Parde, N, Plátek, O, Rieser, V, Ruan, J, Tetreault, J, Toral, A, Wan, X, Wanner, L, Watson, L & Yang, D 2023, Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP. in S Tafreshi, R Akula Arjun, J Sedoc, A Drozd, A Rogers & A Rumshisky (eds), Proceedings of the Fourth Workshop on Insights from Negative Results in NLP. Association for Computational Linguistics, ACL Anthology, pp. 1-10, 4th Workshop on Insights from Negative Results in NLP, INSIGHTS 2023, Dubrovnik, Croatia, 05/05/2023. https://doi.org/10.18653/v1/2023.insights-1.1

Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP. / Belz, Anya; Thomson, Craig; Reiter, Ehud et al.
Proceedings of the Fourth Workshop on Insights from Negative Results in NLP. ed. / Shabnam Tafreshi; Reddy Akula Arjun; Joao Sedoc; Aleksandr Drozd; Anna Rogers; Anna Rumshisky. Association for Computational Linguistics, ACL Anthology, 2023. p. 1-10.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Missing Information, Unresponsive Authors, Experimental Flaws

T2 - 4th Workshop on Insights from Negative Results in NLP, INSIGHTS 2023

AU - Belz, Anya

AU - Thomson, Craig

AU - Reiter, Ehud

AU - Abercrombie, Gavin

AU - Alonso-Moral, Jose M.

AU - Arvan, Mohammad

AU - Braggaar, Anouck

AU - Cieliebak, Mark

AU - Clark, Elizabeth

AU - van Deemter, Kees

AU - Dinkar, Tanvi

AU - Dušek, Ondřej

AU - Eger, Steffen

AU - Fang, Qixiang

AU - Gao, Mingqi

AU - Gatt, Albert

AU - Gkatzia, Dimitra

AU - González-Corbelle, Javier

AU - Hovy, Dirk

AU - Hürlimann, Manuela

AU - Ito, Takumi

AU - Kelleher, John D.

AU - Klubička, Filip

AU - Krahmer, Emiel

AU - Lai, Huiyuan

AU - van der Lee, Chris

AU - Li, Yiru

AU - Mahamood, Saad

AU - Mieskes, Margot

AU - van Miltenburg, Emiel

AU - Mosteiro, Pablo

AU - Nissim, Malvina

AU - Parde, Natalie

AU - Plátek, Ondřej

AU - Rieser, Verena

AU - Ruan, Jie

AU - Tetreault, Joel

AU - Toral, Antonio

AU - Wan, Xiaojun

AU - Wanner, Leo

AU - Watson, Lewis

AU - Yang, Diyi

PY - 2023

Y1 - 2023

N2 - We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.

AB - We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.

UR - http://www.scopus.com/inward/record.url?scp=85184667620&partnerID=8YFLogxK

U2 - 10.18653/v1/2023.insights-1.1

DO - 10.18653/v1/2023.insights-1.1

M3 - Conference contribution

AN - SCOPUS:85184667620

SP - 1

EP - 10

BT - Proceedings of the Fourth Workshop on Insights from Negative Results in NLP

A2 - Tafreshi, Shabnam

A2 - Akula Arjun, Reddy

A2 - Sedoc, Joao

A2 - Drozd, Aleksandr

A2 - Rogers, Anna

A2 - Rumshisky, Anna

PB - Association for Computational Linguistics, ACL Anthology

Y2 - 5 May 2023

ER -

Belz A, Thomson C, Reiter E, Abercrombie G, Alonso-Moral JM, Arvan M et al. Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP. In Tafreshi S, Akula Arjun R, Sedoc J, Drozd A, Rogers A, Rumshisky A, editors, Proceedings of the Fourth Workshop on Insights from Negative Results in NLP. Association for Computational Linguistics, ACL Anthology. 2023. p. 1-10 doi: 10.18653/v1/2023.insights-1.1

Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

Abstract

Conference

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this