Hypothesis Testing, P-Values & Type I/II Errors
A board-focused walkthrough of hypothesis testing: how to read a p-value against α, the Type I/II error truth table, power, and confidence-interval rules — with vignettes drilling the classic p-value misinterpretation and underpowered-study traps.
The Hypothesis-Testing Framework
Every study starts with a null hypothesis (H₀) — there is no difference or association between groups — and an alternative hypothesis (H₁) — a real difference exists. A study never "proves" H₁; it only gathers evidence to reject or fail to reject H₀. Before collecting data you set a significance level (α), conventionally 0.05 — the maximum false-positive rate you will tolerate. After the study you compute a p-value: the probability of observing your result, or one more extreme, if H₀ were true. If p < α, the result is "statistically significant" and you reject H₀; if p ≥ α, you fail to reject it. Critically, failing to reject H₀ is not proof that H₀ is true — absence of evidence ≠ evidence of absence. The board-favorite trap: a p-value is not the probability that H₀ is true, and not the probability your result occurred "by chance" — it is calculated assuming chance (H₀) is already the explanation.
- P-value: probability of the observed (or more extreme) result assuming H₀ is true; p < 0.05 → reject H₀ (statistically significant)
- α (significance level): pre-set threshold, usually 0.05 = probability of a Type I error (false positive)
- β: probability of a Type II error (false negative); conventionally set at 0.20 (→ 80% power)
- Power = 1 − β: probability of correctly detecting a true effect (rejecting a false H₀); target ≥ 0.80
- Statistically significant ≠ clinically significant — a trivially small effect can reach significance with a large enough sample
- Lowering α (e.g., 0.05 → 0.01) → fewer false positives but more false negatives (↑β, ↓power)
- A p-value tells you whether an effect likely exists, not how large it is — magnitude comes from the effect size / confidence interval
Type I vs Type II Error — The Truth Table
| Decision | H₀ actually TRUE (no real effect) | H₀ actually FALSE (real effect) |
|---|---|---|
| Reject H₀ ("positive" study) | Type I error = α (false positive) | Correct — Power = 1 − β |
| Fail to reject H₀ ("negative" study) | Correct (true negative = 1 − α) | Type II error = β (false negative) |
Courtroom analogy (H₀ = "the defendant is innocent"):
- Type I error (α) = convicting an innocent person → a false positive (you "found" an effect that isn't real).
- Type II error (β) = acquitting a guilty person → a false negative (you missed a real effect).
"Cry wolf" version:
- Type I: villagers rush in but there's no wolf = false positive.
- Type II: villagers ignore the boy but a wolf is really there = false negative.
Memory hook: α is the 1st Greek letter → Type I; β is the 2nd → Type II.
Vignette: A randomized trial compares a new antihypertensive to placebo. Systolic BP falls 12 mmHg more with the drug; the difference has p = 0.03. A student concludes, "There is only a 3% chance the drug doesn't work."
- Why that's wrong: p = 0.03 means that if the drug truly had no effect (H₀ true), there is a 3% probability of seeing a difference this large or larger by chance. It is not the probability that H₀ is true.
- Decision: because p (0.03) < α (0.05), reject H₀ → the result is statistically significant.
- Next best (interpretive) step: examine the effect size and 95% CI to judge whether 12 mmHg is clinically meaningful and how precise the estimate is — significance alone never establishes clinical importance.
Vignette: A small trial (n = 40) finds lower mortality with a new drug vs placebo, but p = 0.09 and the 95% CI for the relative risk is 0.55–1.10. The investigators conclude "the drug doesn't work."
- Interpretation: p ≥ 0.05, so they fail to reject H₀ — but the study is likely underpowered. A non-significant result is not proof of no effect; if a true benefit was missed, this is a Type II error (β).
- CI rule: a 95% CI for a ratio (RR/OR/HR) that crosses 1.0 → non-significant (for a difference in means, the equivalent is crossing 0). The CI (0.55–1.10) is wide → imprecise → small sample.
- Next best step: increase the sample size — the single most reliable way to raise power (1 − β) and narrow the CI.
Ways to INCREASE power (↓ Type II error):
- ↑ Sample size — the most commonly tested answer
- ↑ Effect size (larger true difference between groups)
- ↓ Variability / measurement error (more precise measures)
- ↑ α (e.g., 0.05 → 0.10) — raises power but increases Type I error (a trade-off, not a quality gain)
- One-tailed test — more power than two-tailed in the pre-specified direction, but rarely the intended board answer and invalid if the direction wasn't set in advance
Confidence-interval quick rules:
- 95% CI ↔ α = 0.05; if the null value lies outside the 95% CI, then p < 0.05
- CI for a difference (means) that includes 0 → not significant
- CI for a ratio (RR, OR, HR) that includes 1 → not significant
- Narrower CI = more precision (from larger n / lower variance)
Practice Biostatistics now
Board-style questions, spaced-repetition flashcards, and a Socratic AI tutor — free to start.