A/B Testing
Applying hypothesis testing to compare two product/design variants — sample size, statistical power, and the traps of stopping early.
What it is
A/B testing randomly splits users between a control (A) and a variant (B), then applies hypothesis testing to decide whether an observed difference in a metric (conversion rate, click-through, revenue) is likely real rather than noise.
Sample size and power
Before running a test, compute the minimum sample size needed to detect an effect of a meaningful size with adequate power (commonly 80%) — running a test until "it looks significant" without this upfront calculation dramatically inflates the false-positive rate.
The peeking problem
Checking results daily and stopping as soon as p < 0.05 ("peeking") is one of the most common real-world A/B testing mistakes — each additional look is another chance for random noise to cross the threshold, inflating the true false-positive rate far above the nominal 5%. Sequential testing methods exist specifically to allow valid early stopping.
Randomization is what makes it work
Without random assignment, any observed difference could be explained by a confound (e.g. variant B happened to be shown more to mobile users) rather than the variant itself — randomization is what lets you attribute the difference to the change being tested.
Prerequisite
Directly applies the hypothesis-testing framework to a specific, high-stakes business use case.