Type I & Type II Errors and Statistical Power

Purpose

To discuss Type I and Type II errors and how they relate to the idea of statistical power.

Making decisions

knitr::include_graphics("https://mk0abtastybwtpirqi5t.kinstacdn.com/wp-content/uploads/type-1-2-errors.png")

Setting decision thresholds

• Goal: Minimize the probability of making errors
• Types of errors
  • False positives: Nothing’s going on, but we say otherwise (Type I)
  • False negatives: Something’s going on, but we miss it (Type II)

knitr::include_graphics("https://i.stack.imgur.com/R0ncP.png")

What do we control? What do we measure?

• Control
  • Sample size
  • Decision criteria
  • How big a difference (effect size) actually matters
  • Width of “null” or reference distribution
• Measure
  • Outcomes and their variability

Power analysis on sex differences data

https://gilmore-lab.github.io/sex-differences-in-motion-perception/analysis/preregistration_power_analysis.html

Bottom line

• Need bigger samples to detect smaller effects with confidence