How Many Replicators Does It Take to Achieve Reliability? Investigating Researcher Variability in a Crowdsourced Replication release_4xuqjtoeszbetc2iqwxopv2llu

by Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Hung H. V. Nguyen, Muna Adem, Jule Adriaans, Esra Akdeniz, Amalia Alvarez-Benjumea, Henrik Andersen, Daniel Auer, Flavio Azevedo, Oke Bahnsen (+174 others)

Released as a post by Center for Open Science.

2021  

Abstract

The paper reports findings from a crowdsourced replication. Eighty-four replicator teams attempted to verify results reported in an original study by running the same models with the same data. The replication involved an experimental condition. A "transparent" group received the original study and code, and an "opaque" group received the same underlying study but with only a methods section and description of the regression coefficients without size or significance, and no code. The transparent group mostly verified the original study (95.5%), while the opaque group had less success (89.4%). Qualitative investigation of the replicators' workflows reveals many causes of non-verification. Two categories of these causes are hypothesized, routine and non-routine. After correcting non-routine errors in the research process to ensure that the results reflect a level of quality that should be present in 'real-world' research, the rate of verification was 96.1% in the transparent group and 92.4% in the opaque group. Two conclusions follow: (1) Although high, the verification rate suggests that it would take a minimum of three replicators per study to achieve replication reliability of at least 95% confidence assuming ecological validity in this controlled setting, and (2) like any type of scientific research, replication is prone to errors that derive from routine and undeliberate actions in the research process. The latter suggests that idiosyncratic researcher variability might provide a key to understanding part of the "reliability crisis" in social and behavioral science and is a reminder of the importance of transparent and well documented workflows.
In application/xml+jats format

Archived Files and Locations

application/pdf  845.4 kB
file_unm3szd4p5byvhwmfvss6mqzky
files.osf.io (publisher)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  post
Stage   unknown
Date   2021-05-18
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: f84aa6cf-141d-4ed7-8733-c383b2ddc42b
API URL: JSON