🧪 A/B Testing

Core Concept

Split users into two groups: control (A, the current version) and treatment (B, the new version). Measure a metric (conversion rate, click-through rate, engagement). Statistical test determines if observed difference is real or just random noise.

Setup

• Hypothesis: "Version B will improve conversion by at least X%"
• Metric: What to measure (e.g., purchase rate, time on page, clicks)
• Sample size: How many users in each group, calculated in advance to detect desired effect size
• Duration: Run test for fixed time or until sample size reached
• Significance level (α): Usually 0.05, meaning 5% false positive rate acceptable

Null Hypothesis & P-values

Null hypothesis (H₀): "Versions A and B perform identically"

P-value: Probability of seeing the observed difference (or larger) if H₀ were true. If p < α (e.g., p < 0.05), reject H₀. Difference is statistically significant.

Important: p-value ≠ probability the result is real. It's a measure of how surprised we'd be under H₀.

Statistical Tests

T-test (continuous metrics)

Compares means of two groups. Assumes approximately normal distributions. Use for metrics like session duration, revenue per user.

Chi-squared test (binary metrics)

Compares counts in categories (e.g., converted vs. not converted). Standard for A/B tests on conversion rates.

Bayesian approach

Instead of p-values, compute posterior probability that B is better than A. More intuitive; allows stopping early. Requires specifying a prior.

Common Pitfalls

• Peeking / Multiple comparisons: Stopping early if the test "looks significant" inflates false positive rate. Run for the planned duration.
• Winner's curse: If you run many A/B tests, some will show false significance by random chance. Adjust significance level (Bonferroni correction).
• Confounding factors: If you change multiple things (copy + button color), you can't tell which drove the effect.
• Segment mismatch: Maybe the new version works great for mobile users but hurts desktop. Check segment-level results.
• Temporal effects: Day-of-week or seasonal trends can confound results if A and B don't run simultaneously.

Sample Size Calculation

Larger effect sizes need fewer users. If you want 80% power (detecting a real effect 80% of the time) and α=0.05:

Where p₁, p₂ are baseline and target conversion rates. Larger n if effect size is small.

When to Run A/B Tests

• High-traffic product changes (enough users to reach significance quickly)
• Uncertain about impact (no clear winner a priori)
• Metric is well-defined and not too noisy
• Users can be randomly assigned (not dependent on user history in a biasing way)

When NOT to Run A/B Tests

• Low traffic (test might never reach significance)
• Effect expected to be huge (run a smaller pilot test first)
• Metric is very noisy (need enormous sample sizes)
• One variant is clearly better on principle (security, legal, accessibility)