Skip to content
Tech Interview Prep
Technical interview guide

A/B Testing

Applying hypothesis testing to compare two product/design variants: sample size, statistical power, and the traps of stopping early.

Read
46 min
Practice MCQs
25
Interview Q&A
25
Edition
v6
Editorial status
Reviewed

Scope: Practitioner and standards references current as of 2026-09: the experimentation-platform literature, NIST/SEMATECH e-Handbook, the 2016 ASA statement on p-values, SciPy, and statsmodels.

Curated: · Written: · Reviewed:

25 questions

MCQ-2A team's experiment plan names query success rate as the primary metric and p95 latency, crash rate, and sample ratio as guardrails. Which statement correctly captures the operational role of the guardrails and distinguishes them from the primary metric?
MCQ-24A team pre-registers a decision rule before launching a checkout redesign: ship only if the 95% CI for the conversion lift has a lower bound above +1.0 pp; revert if p99 checkout latency exceeds 400 ms; otherwise do not ship. Two weeks in, the lift is +1.4 pp with a 95% CI of [+0.2 pp, +2.6 pp], and p99 latency is 380 ms against a 350 ms baseline. What does the pre-registered rule require?
MCQ-7A team pre-registers checkout conversion (user level) as the primary metric for an A/B test, with the decision rule: ship if the lower bound of the 95% CI on the absolute lift exceeds 0 and no guardrail is breached. At the end of the planned two-week run the analysis shows: primary metric +0.4 pp, 95% CI [-0.1 pp, +0.9 pp]; average order value +3.0%, 95% CI [+1.0%, +5.0%]; mobile-only checkout conversion +2.1 pp, 95% CI [+0.3 pp, +3.9 pp]; no guardrail breached. What should the team do?
MCQ-25A checkout redesign ran as an A/B test for 9 days. Assignment was configured 50/50; the logs show 48,200 sessions in control and 41,800 in treatment. Purchase rate came back +4.1% (95% CI +0.3% to +7.9%), and the p95 latency guardrail held (380 ms → 410 ms, threshold 500 ms). The team asks you to sign off on shipping. Which read of this experiment is correct?
MCQ-5A 50/50 experiment logs 51,200 control and 48,800 treatment assignments out of 100,000 users. The primary metric shows a +3.2% lift with p = 0.04, and the team notes the assignment imbalance is only 2.4%. What should they do?
MCQ-13Your checkout A/B test has baseline conversion 3.2%. You want 80% power to detect a 10% relative lift (to 3.52%) at two-sided alpha 0.05. Each arm gets 25,000 users per week. Which conclusion follows?
MCQ-22A signup page gets 400 signups a week at a 20% baseline conversion rate. The team wants to A/B test a copy change they expect to lift conversion by about 1 percentage point; their power analysis puts the smallest lift the 4-week test can detect at 80% power at about 8 percentage points. What should they do?
MCQ-6Your team is validating a new experimentation platform before it goes into production. Each A/A run splits 20,000 users 50/50 by the assignment hash and runs a two-sample t-test on revenue per session, with the variance estimated by the delta method. Across 1,000 runs, the p-values have mean 0.44, 137 fall below 0.05, and a KS test of the 1,000 p-values against Uniform(0,1) returns p = 0.0003. What should you conclude?
MCQ-20A checkout redesign runs on 240,000 sessions. The pre-registered primary analysis, overall conversion, moves from 3.00% to 3.12% (two-sided p = 0.04). The team then cuts the same data by device (3 levels) and country (4 levels): 12 segments, none pre-registered, no interaction test. Tablet users in Germany are up 9% with p = 0.03. What should the team conclude?
MCQ-9A consumer app ships about 40 features a year, each with its own randomized control arm, and every one posts a 1–2% win on the primary metric in its two-week test. Two years in, the company-level metric is flat. The team adds a persistent 1% holdout that receives none of the shipped treatments. What does that holdout measure that the per-experiment control arms cannot?
MCQ-12A checkout-prompt change ran for five weekdays (Mon–Fri) with concurrent randomisation: treatment lifted conversion by +0.8pp (p = 0.03). Two earlier tests of the same prompt format put the weekend effect at −0.4pp. Weekdays carry 70% of weekly traffic. What is the best estimate of the change's effect on the full week?
MCQ-21A team ramps a new ranking model on 1M sessions/day. Days 1–2 (a weekend, when conversion runs at 2.0%): 1% of sessions get the model, 99% control. Days 3–14 (weekdays, conversion 5.0%): a 50/50 split. Conversion is 2.0% in both arms on the weekend and 5.0% in both arms on weekdays. Pooling all 14 days gives treatment 4.99% vs control 4.26%, +0.73pp, p<0.001. The team reports the +0.73pp as the model's effect. Which reading is correct?
MCQ-10An experimentation team observes that a new UI redesign produced an immediate 8% lift in key click-through metrics during week 1, but the lift dropped to 2% by week 4. How can the platform systematically isolate whether this attenuation stems from a novelty effect rather than an unobserved confounder?
MCQ-11An experiment on a redesigned checkout flow runs for six weeks. The estimated lift is +9% in week 1 and +2% by week 6. Which statement correctly describes how novelty and primacy effects bias the early-window estimate and why that bias decays?
MCQ-3A checkout redesign is tested on 200,000 users, 100,000 per arm. Overall purchase rate is 5.0% in control and 6.0% in treatment. The team wants to restrict the analysis to users who reached the pricing page, a step the redesign changes: 20% of control users and 30% of treatment users reach it, and among those the purchase rate is 25% in control and 20% in treatment. Which conclusion is correct?
MCQ-1A subscription service ships a one-click reorder button to all users and then compares spend. Among 200,000 users, the 20,000 who clicked the button spent $58 on average over the next 30 days; the 180,000 who did not spent $31. A follow-up randomised experiment assigns 25,000 users to see the button and 25,000 not to. Treatment-arm spend averages $33.40, control-arm $32.10, and 22% of the treatment arm clicks. Which conclusion does the experiment support?
MCQ-8A subscription team wants to use 7-day activation rate as a surrogate for 12-month retention. In a past test, a new onboarding flow raised activation by 4 points but left retention flat, while a pricing change moved retention without moving activation. Which property must activation have for the team to trust it as a surrogate in future tests?
MCQ-17A team is running an A/B test on revenue per user. Revenue is heavy-tailed: the top 0.1% of users account for a large share of the total, and one user in the treatment arm made a purchase 200× the median. The design doc must keep the estimated effect interpretable as the mean difference in revenue per user for the whole population while reducing the variance of that estimate. Which approach should the doc specify?
MCQ-14A team runs a two-arm experiment on a 28-day revenue metric. They build a CUPED adjustment using a covariate measured during the experiment window rather than before it, and estimate theta = cov(y, x) / var(x) = 0.4. Treatment raises the metric by 5 units and, because the covariate is measured in-window, also raises the covariate by 10 units. What does the adjusted estimate report?
MCQ-15A feed experiment randomizes 40,000 users, 20,000 per arm. Each user sees 50 impressions, so each arm logs 1,000,000 impressions. Click rate per impression is 4.90% in control and 5.00% in treatment. A two-proportion z-test over impressions returns z = 3.26 (p ≈ 0.001). Per-user click rates have a standard deviation of 0.08 in both arms. Which conclusion does the honest analysis support?
MCQ-16An A/A test, both arms get the identical experience, randomizes 10,000 users per arm. The metric is CTR = total clicks / total impressions; each user sees a variable number of impressions (mean 20, sd 15), so each arm logs about 200,000 impressions. Computing the p-value with the binomial formula over impressions, the team sees a 'significant' difference (p < 0.01) in 28% of 1,000 simulated A/A runs instead of the expected 5%. Which explanation fits?
MCQ-4A food-delivery marketplace tests a new dispatch algorithm that assigns each order to the courier who can deliver it fastest. The team randomises by order, 50/50, and each of the 200 couriers in the test city handles about 20 treatment and 20 control orders a day. After two weeks the measured difference in delivery time is 0.2 minutes (95% CI −0.3 to 0.7), while an offline replay of the same algorithm on last month's orders predicts a 1.5-minute improvement. What should the team conclude?
MCQ-18A food-delivery marketplace runs a two-week experiment: couriers are randomised 50/50 to a per-delivery bonus (treatment) or the existing pay (control). Baseline is 8 deliveries per courier-hour. The bonus raises a treated courier's own rate by 2 deliveries/hour. Because treated couriers claim more of the available orders, control couriers' rate falls by 1 delivery/hour. The analyst compares mean deliveries/hour between arms. What does the comparison report, and what is the true effect of the bonus?
MCQ-19A team wants to know whether a new eviction policy lowers p99 latency on a shared cache cluster. User-level A/B testing is invalid because control users read entries that treatment users wrote, so they run a switchback: the whole cluster alternates between the old and new policy every 30 minutes for 14 days. 672 periods, 336 per arm. Treating periods as independent, the analysis reports p99 down 12 ms with p < 0.001. The cache takes about 2 hours to reach steady state after a policy switch. What should the team do with this result?
MCQ-23A 50/50 A/B test of a new checkout flow runs for five days. Treatment shows 8% lower revenue per session (p = 0.004). Split by payment path: the 62% of sessions that pay with a saved card show identical revenue per session in both arms, while the 38% that type in card details show 21% lower revenue per session in treatment. The treatment page fires its purchase event from a script that runs after the browser redirects to the payment provider's page. What is the most likely explanation?