Dataset decay and the problem of sequential analyses on open datasets

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Open data allows researchers to explore pre-existing datasets in new ways. However, if many researchers reuse the same dataset, multiple statistical testing may increase false positives. Here we demonstrate that sequential hypothesis testing on the same dataset by multiple researchers can inflate error rates. We go on to discuss a number of correction procedures that can reduce the number of false positives, and the challenges associated with these correction procedures.

Related collections

Most cited references 31

Record: found
Abstract: not found
Article: not found

Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing

Yoav Benjamini, Yosef Hochberg (1995)

0 comments Cited 25572 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning.

Tal Yarkoni, Jacob Westfall (2017)

Psychology has historically been concerned, first and foremost, with explaining the causal mechanisms that give rise to behavior. Randomized, tightly controlled experiments are enshrined as the gold standard of psychological research, and there are endless investigations of the various mediating and moderating variables that govern various behaviors. We argue that psychology's near-total focus on explaining the causes of behavior has led much of the field to be populated by research programs that provide intricate theories of psychological mechanism but that have little (or unknown) ability to predict future behaviors with any appreciable accuracy. We propose that principles and techniques from the field of machine learning can help psychology become a more predictive science. We review some of the fundamental concepts and tools of machine learning and point out examples where these concepts have been used to conduct interesting and important psychological research that focuses on predictive research questions. We suggest that an increased focus on prediction, rather than explanation, can ultimately lead us to greater understanding of behavior.

0 comments Cited 454 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

An Agenda for Purely Confirmatory Research.

Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom … (2012)

The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology's academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result-a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label "confirmatory," and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled "exploratory." We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.

0 comments Cited 314 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

William Hedley Thompson:

ORCID: https://orcid.org/0000-0002-0533-6035

Chris I Baker: Role: Reviewing Editor

Peter Rodgers: Role: Senior Editor

Journal

Journal ID (nlm-ta): eLife

Journal ID (iso-abbrev): Elife

Journal ID (publisher-id): eLife

Title: eLife

Publisher: eLife Sciences Publications, Ltd

ISSN (Electronic): 2050-084X

Publication date (Electronic, pub): 19 May 2020

Publication date Collection: 2020

Volume: 9

Electronic Location Identifier: e53498

Affiliations

[1 ]Department of Psychology, Stanford University StanfordUnited States

[2 ]Department of Clinical Neuroscience, Karolinska Institutet StockholmSweden

[3 ]Department of Philosophy, Stanford University StanfordUnited States

National Institute of Mental Health, National Institutes of Health United States

eLife United Kingdom

National Institute of Mental Health, National Institutes of Health United States

University of Glasgow United Kingdom

Author information

William Hedley Thompson https://orcid.org/0000-0002-0533-6035

Jessey Wright https://orcid.org/0000-0001-5003-0572

Russell A Poldrack https://orcid.org/0000-0001-6755-0259

Article

Publisher ID: 53498

DOI: 10.7554/eLife.53498

PMC ID: 7237204

PubMed ID: 32425159

SO-VID: ac6446e7-8d41-434b-a35f-5b0289a32340

License:

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

History

Date received : 12 November 2019

Date accepted : 04 May 2020

Funding

Funded by: FundRef http://dx.doi.org/10.13039/501100004063, Knut och Alice Wallenbergs Stiftelse;

Award ID: 2016.0473

Award Recipient : William Hedley Thompson

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Custom metadata

Author impact statement Open data provides an opportunity to perform new analyses on preexisting data, but trade-offs are required to limit an increase in false positives.

Template 5

ScienceOpen disciplines: Life sciences

Keywords: open data,sequential testing,multiple comparisons,multiple comparison correction,meta-research,human

Data availability:

ScienceOpen disciplines: Life sciences

Keywords: open data, sequential testing, multiple comparisons, multiple comparison correction, meta-research, human

Dataset decay and the problem of sequential analyses on open datasets

Read this article at

Abstract

Related collections

Open Data in Research

Most cited references 31

Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing

Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning.

An Agenda for Purely Confirmatory Research.

Author and article information

Contributors

Journal

Affiliations

Author information

Article

History

Funding

Categories

Custom metadata

Comments

Comment on this article

Similar content 67

Cited by 25

Most referenced authors 925