Bio

• B.A., Cognitive Science, Brown University
• M.S. & Ph.D., Psychology (Cognitive Neuroscience), Carnegie Mellon University
• Human brain development, perception & action, computational modeling, machine vision, big data, open science
• Founding Director of Human Imaging, Penn State Social, Life & Engineering Sciences Imaging Center (SLEIC)
• Co-Founder/Co-Director of Databrary.org data library
• gilmore-lab.github.io
• banjo player, actor, cyclist, backpacker, poet, ham (K3ROG), native Coloradoan

Acknowledgments

• Karen Adolph (NYU)
• LEGO Foundation, Society for Research in Child Development

Roadmap

• There’s something happening here; what it is ain’t exactly clear…
• Reproducibility in Psychology
• Reproducibility in Neuroscience
• Toward a better future

Is there a reproducibility crisis?

• Yes, a significant crisis.
• Yes, a slight crisis.
• No, there is no crisis.
• Don’t know.

Have you ever failed to reproduce an experiment?

• Someone else’s
• My own

Have you ever tried to publish a reproduction attempt?

• Published
• Failed to publish

What factors contribute to irreproducible research?

Solvable problems

• Selective reporting
• Pressure to publish
• Low power
• Replication in original labs
• Mentoring/oversight
• Methods, code, data unavailable
• Poor design
• Fraud
• Insufficient peer review
• Variability of reagents
• Bad luck

Psychology is harder than physics

Behavior has multiple, nested dimensions

Adolph, K., Tamis-LeMonda, C. & Gilmore, R.O. (2016). PLAY Project: Webinar discussions on protocol and coding. Databrary. Retrieved February 17, 2017 from https://nyu.databrary.org/volume/232

Data are sensitive, hard(er) to share

• Protect participant’s identities
• Protect from harm/embarrassment

Sampling is biased

• Much (lab-based) data collected are from Western, Educated Industrialized, Rich, Democratic (WEIRD) populations (Henrich, Heine, and Norenzayan 2010)

Even on Mechanical Turk

The sin of unreliability

• Studies are underpowered (Maxwell 2004) and (Szucs and Ioannidis 2016)
• Direct replications rare, discouraged, hard to publish
• Critical methodological details go unreported

The sin of data hoarding

• Data sharing is not universal practice
• Even after a publication has gone to press
• Despite implicit agreement to do so if publishing in certain journals (e.g., American Psychological Association)

& hasn’t improved since 2006

The sin of corruptibility

• Stapel was Dean of the School of Social and Behavioral Sciences at Tilburg University, teacher of Scientific Ethics course
• Fraud investigation launched when 3 grad students noticed anomalies – duplicate entries in data tables
• Stapel confessed, lost position, gave up Ph.D., wrote a book

Self-reported data fabrication, falsification, and alteration

Self-reports of questionable research practices

• Marc Hauser
• Evolutionary/Comparative Psychologist, Professor at Harvard
• Resigned 2011 after internal investigation found him responsible for research misconduct
• Details see 2012 report by NIH Office of Research Integrity (ORI) and Hauser’s response.

Mistakes, e.g., flexible “stopping” rules

Errors/omissions in data

• C. Reinhart and K. Rogoff paper, http://www.nber.org/papers/w15639.pdf?new_window=1
  • Omitted data in Excel spreadsheet used to model effect of public debt on economic growth. https://www.bloomberg.com/news/articles/2013-04-18/faq-reinhart-rogoff-and-the-excel-error-that-changed-history
  • Error only found when data were shared.

The sin of bias…

*“This article reports 9 experiments, involving more than 1,000 participants, that test for retroactive influence by”time-reversing" well-established psychological effects so that the individual’s responses are obtained before the putatively causal stimulus events occur.“*

(Bem 2011)

“We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem’s p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.”

(E.-J. Wagenmakers et al. 2011)

Reproducibility “Crisis”

• Reproducibility Project: Psychology
• Attempt to replicate 100 experimental and correlational studies published in three psychology journals in 2008 using high-powered designs and original materials when available.
• Materials, data, protocols, analysis code shared via the Open Science Framework (OSF) from the Center for Open Science (COS)

Results published in (Collaboration 2015)

• 39/98 (39.7%) replication attempts were successful
• 97% of original studies reported statistically significant results vs. 36% of replications

So, did the studies replicate?

• Samples not equal
  • Sampling error differences predicts < 100% reproducibility
• Protocols not identical
  • Only 69% of original PIs “endorsed” replication protocol. Replication rate 4x higher (59.7% vs. 15.4%) in studies with endorsed protocol.
• (Collaboration 2015) “…seriously underestimated reproducibility of psychological science.”

Underpowered studies

Risks of false positives

Multiple (> 69K) computational pathways

Tools for openness and transparency

• Open Science Framework (OSF)
  • Reproducibility Project
  • Pre-registration Challenges
  • Pre-print services

Changing journal, funder practices

• Transparency and opennness promotion (TOP) guidelines, (B. A. Nosek et al. 2015)

• Data, materials, code citation
• Data transparency
• Analytic methods (code) transparency
• Design and analysis transparency
• Preregistration of studies
• Preregistration of analysis plans
• Replication

Large-scale replication efforts

Improving methodology training

• Literate Programming
  • RStudio and R Markdown
  • Jupyter notebooks
  • Electronic lab notebooks, talks, reports, articles…
• Software Carpentry
• Version control (e.g., GitHub)
• PSY 511 Spring 2017

Data publication

• Inter-university Consortium for Political and Social Research (ICPSR)
• Dataverse
• Dryad Data Repository

OpenNeuro.org

Databrary.org

Databrary.org

• Specializes in storing, sharing video
• Video captures behavior unlike other methods, but is identifiable
• Policy framework for sharing identifiable data
  • Permission to share -> builds on informed consent
  • Restricted access for (institutionally) authorized researchers
• Datavyu, free, open-source video coding tool

Toward a databservatory…

The Human Project

https://www.nytimes.com/2017/06/04/nyregion/human-project-new-york-city-study.html

Open Humans

Neurosynth.org

Social Data Explorer

WordBank

Allen Brain Atlas

A vision of our open science future…

• All data, materials, code shared
  • when paper goes to press or at end of grant period
• Shared in repositories that encourage data linkage (w/ permission)
  • People, places, times, tasks, behaviors, …

• Commonplace citations of data, materials, code, findings
• Ecosystems for new discovery

What’s your vision?

Stack

This talk was produced on 2017-07-11 in RStudio 1.0.143 using R Markdown and the reveal.JS framework. A BibTex format reference file can be found in bib/psu-repro.bib The code and materials used to generate the slides may be found at https://github.com/gilmore-lab/psu-data-repro-bootcamp-2017-07-10/. Information about the R Session that produced the code is as follows:

## R version 3.4.0 (2017-04-21)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Sierra 10.12.5
## 
## Matrix products: default
## BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] dplyr_0.5.0   ggplot2_2.2.1
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.10     knitr_1.16.4     magrittr_1.5     munsell_0.4.3   
##  [5] colorspace_1.3-2 R6_2.2.0         stringr_1.2.0    plyr_1.8.4      
##  [9] tools_3.4.0      revealjs_0.9     grid_3.4.0       gtable_0.2.0    
## [13] DBI_0.6-1        htmltools_0.3.6  yaml_2.1.14      lazyeval_0.2.0  
## [17] rprojroot_1.2    digest_0.6.12    assertthat_0.2.0 tibble_1.3.0    
## [21] evaluate_0.10    rmarkdown_1.5    labeling_0.3     stringi_1.1.5   
## [25] compiler_3.4.0   scales_0.4.1     backports_1.0.5