📊 P-Values & Statistical Significance

The Null Hypothesis

Start with an assumption: H₀ (null hypothesis) states "there is no effect" or "the groups are the same".

Example H₀: "Drug A and placebo produce the same symptom relief."

Your test collects data and asks: "How unlikely is this data if H₀ is true?" If unlikely enough, reject H₀ and claim a difference exists (statistical significance).

What is a P-value?

P-value: Probability of observing your data (or more extreme) assuming H₀ is true.

Low p-value = your data is surprising under H₀ = evidence against H₀.

Significance Level (α)

Threshold for "unlikely enough". Usually α = 0.05 (5% false positive rate acceptable). If p < α, reject H₀ and claim statistical significance.

Note: α = 0.05 means 1 in 20 tests will falsely reject H₀ even if it's true (false positive / Type I error).

What P-values Are NOT

• NOT probability the result is "real". It's the probability of the data under H₀, not the probability of H₀ itself.
• NOT the probability you made a mistake. That's not quantified by p-values alone.
• NOT effect size. A p-value of 0.001 does NOT mean a large effect; it means the effect is unlikely under H₀. The effect could be tiny but detected in large sample.
• NOT reproducibility. Low p-value ≠ result will replicate. Many low-p studies fail to replicate (publication bias, overfitting).

Type I and Type II Errors

• Type I (false positive): Reject H₀ when it's actually true. Probability = α (e.g., 0.05).
• Type II (false negative): Fail to reject H₀ when it's actually false. Probability = β. Power = 1−β (probability of detecting real effect).

Trade-off: Lowering α (fewer false positives) increases β (more false negatives). Larger sample sizes reduce both.

Degrees of Freedom & Critical Values

Tests compare your statistic (t, χ², F, Z) to a threshold based on the distribution. The distribution depends on degrees of freedom (df) = roughly sample size minus constraints.

More df = larger critical value needed for significance = harder to reject H₀ (makes sense: larger sample should show real effects more clearly).

One-Tailed vs. Two-Tailed Tests

Two-tailed: "Is A different from B?" (either direction). Splits α in half (α/2 at each tail). More conservative.

One-tailed: "Is A better than B?" (specific direction). Uses all of α in one tail. Only use if you had strong reason to expect direction before seeing data.

Default: two-tailed. One-tailed risks p-hacking (if result went the other way, you'd ignore it).

P-Hacking & Multiple Comparisons

If you test many hypotheses, some will reach p < 0.05 by random chance alone. Running 20 tests at α=0.05 expects ~1 false positive.

Solutions:

• Bonferroni correction: Divide α by number of tests (e.g., α = 0.05/20 = 0.0025). Very conservative.
• FDR (False Discovery Rate): Control expected proportion of false positives among all positives. Less conservative.
• Pre-registration: Declare test plan before seeing data. Prevents cherry-picking.

Effect Size

P-value doesn't capture effect magnitude. Report both:

• Difference in means: Mean A − Mean B (useful if means interpretable)
• Cohen's d: Standardized difference = (Mean A − Mean B) / pooled SD. d ≈ 0.2 small, 0.5 medium, 0.8 large.
• Correlation (r): Strength of linear relationship, −1 to +1.
• Odds ratio / Risk ratio: For binary outcomes (e.g., probability of disease given treatment).

Confidence Intervals

95% confidence interval (CI): If you ran the same study 100 times, ~95 of them would produce CI's containing the true parameter.

Misinterpretation: NOT "95% probability the true value is in this interval" (Bayesian; requires a prior).

Use: Better than p-values alone because they show the range of plausible effects.

Reproducibility Crisis

Many published "significant" results don't replicate. Causes:

• Publication bias: Only significant results published
• P-hacking: Testing many hypotheses, reporting winners
• Low power: Small sample, low probability of detecting real effects
• Measurement error: Noisy outcome, inflates noise and Type I error

Fixes: Pre-register studies, report all tests (Registered Reports), use large samples, replicate before publishing.