Why we habitually engage in null-hypothesis significance testing: A qualitative study

Jonah Stunt*, Leonie van Grootel, Lex Bouter, David Trafimow, Trynke Hoekstra, Michiel de Boer

*Corresponding author for this work

    Research output: Contribution to journalArticleAcademicpeer-review

    39 Downloads (Pure)

    Abstract

    Background: Null Hypothesis Significance Testing (NHST) is the most familiar statistical procedure for making inferences about population effects. Important problems associated with this method have been addressed and various alternatives that overcome these problems have been developed. Despite its many well-documented drawbacks, NHST remains the prevailing method for drawing conclusions from data. Reasons for this have been insufficiently investigated. Therefore, the aim of our study was to explore the perceived barriers and facilitators related to the use of NHST and alternative statistical procedures among relevant stakeholders in the scientific system.

    Methods: Individual semi-structured interviews and focus groups were conducted with junior and senior researchers, lecturers in statistics, editors of scientific journals and program leaders of funding agencies. During the focus groups, important themes that emerged from the interviews were discussed. Data analysis was performed using the constant comparison method, allowing emerging (sub)themes to be fully explored. A theory substantiating the prevailing use of NHST was developed based on the main themes and subthemes we identified.

    Results: Twenty-nine interviews and six focus groups were conducted. Several interrelated facilitators and barriers associated with the use of NHST and alternative statistical procedures were identified. These factors were subsumed under three main themes: the scientific climate, scientific duty, and reactivity. As a result of the factors, most participants feel dependent in their actions upon others, have become reactive, and await action and initiatives from others. This may explain why NHST is still the standard and ubiquitously used by almost everyone involved.

    Conclusion: Our findings demonstrate how perceived barriers to shift away from NHST set a high threshold for actual behavioral change and create a circle of interdependency between stakeholders. By taking small steps it should be possible to decrease the scientific community's strong dependence on NHST and p-values.

    Original languageEnglish
    Article numbere0258330
    Number of pages23
    JournalPLoS ONE
    Volume16
    Issue number10
    DOIs
    Publication statusPublished - 15-Oct-2021

    Cite this