A/B Testing
Applying hypothesis testing to compare two product/design variants — sample size, statistical power, and the traps of stopping early.
Try answering in your head first, then click a question to check the model answer.
Q1.Why is random assignment the core mechanism that lets an A/B test claim causation, not just correlation?(show answer)
Random assignment ensures that, on average, the control and treatment groups are identical in every other respect (demographics, behavior, timing) except which version they saw — so any measured difference in outcomes can be attributed to the treatment itself rather than a confounding variable. Without randomization, any observed difference could be explained by pre-existing differences between the groups instead.
Q2.What would you check before trusting an A/B test's result, beyond just the p-value?(show answer)
Sample Ratio Mismatch (is the actual traffic split close to what was configured, e.g. 50/50), whether the test ran long enough to capture a full business cycle (avoiding day-of-week effects), whether a novelty or primacy effect might be inflating early results, and whether multiple metrics/segments were tested without correcting for multiple comparisons.
Q3.Why does "peeking" at A/B test results daily and stopping as soon as p < 0.05 inflate the false positive rate?(show answer)
The significance threshold assumes a single test performed after data collection is complete. Checking repeatedly and stopping at the first significant result is effectively running many tests, and by chance alone one of those checks is likely to cross the threshold even with no real effect — the true false positive rate across the whole peeking process ends up much higher than the nominal 5%. Sequential testing methods (designed to allow valid peeking) or a pre-committed sample size/duration avoid this.
Q4.How would you determine the required sample size before running an A/B test?(show answer)
A power analysis, which needs: the baseline conversion rate, the minimum effect size worth detecting (the smallest lift that matters practically), the desired significance level (commonly α = 0.05), and the desired statistical power (commonly 80%). Smaller effect sizes and higher power requirements both increase the needed sample size — this is decided before the test to avoid the bias of choosing a stopping point based on results already seen.