Skip to content
Tech Interview Prep
Technical interview guide

Hypothesis Testing

The framework for deciding whether an observed effect is likely real or could plausibly be noise: null hypotheses, p-values, and the two ways a test can be wrong.

Read
44 min
Practice MCQs
25
Interview Q&A
25
Edition
v6
Editorial status
Reviewed

Scope: Standards and library references current as of 2026-09: NIST/SEMATECH e-Handbook, the 2016 ASA statement on p-values, SciPy, and statsmodels.

Curated: · Written: · Reviewed:

25 questions

MCQ-1A checkout A/B test runs 40,000 users per arm. Variant B converts at 12.5% against 12.0% for A: a 0.5 percentage-point lift, 95% CI [0.05, 0.95] points, and the two-sided t-test returns p = 0.03. The team had set α = 0.05 before the test. Which statement correctly describes what the 0.03 is?
MCQ-6A checkout team runs an A/B test with 500,000 users per arm. The control flow converts at 8.00% (40,000 users); the new flow converts at 8.15% (40,750). A two-proportion z-test on these counts gives p ≈ 0.006. Before the test the team agreed to ship only if the lift reached 0.5 percentage points: the smallest gain that covers the added infrastructure cost. Which conclusion is correct?
MCQ-2A team replaces its origin with a cheaper CDN and must show the swap did not worsen p99 latency by more than 10 ms. They test H0: p99_new − p99_old = 0 against H1: p99_new − p99_old ≠ 0. Over 5,000 requests per CDN, p99 is 210 ms (new) vs 205 ms (old) and the test returns p = 0.35. They report 'no significant difference, so the CDNs are equivalent.' Which reading is correct?
MCQ-3A checkout team plans an A/B test on a 10% baseline conversion rate. The smallest lift worth shipping is 11%: a 10% relative improvement. They budget 3,000 users per arm and set a two-sided alpha of 0.05. Which conclusion should they act on before launching?
MCQ-5A team plans a two-arm A/B test on a checkout page. Baseline conversion is 5%. They can randomize 20,000 users per arm. They set alpha = 0.05 (two-sided) and power = 0.8. The smallest lift the business would act on is 0.2 percentage points. What does the minimum detectable effect imply for this test?
MCQ-4An A/B test on a checkout page: control converts at 10.0%, treatment at 10.4%, with 20,000 users per arm. A two-sided test at α = 0.05 gives p = 0.19 and a 95% CI for the lift of [-0.2pp, +1.0pp]. The team has said it will ship only a change worth at least +0.5pp. An analyst computes post-hoc power from the observed 0.4pp effect, gets 0.26, and writes "the test was underpowered, so we can't conclude anything." Which conclusion is warranted?
MCQ-13An experimentation platform runs 20 independent metric tests per experiment at α = 0.05 with no correction. A PM sees one metric at p = 0.02 and wants to ship. Which reading of that result is correct?
MCQ-14A team screens 100 metrics for a launch. Ten metrics have p = 0.001; the other 90 have p-values between 0.2 and 1. They apply Bonferroni and Benjamini-Hochberg at alpha = q = 0.05. Which outcome is correct?
MCQ-15A team runs a fixed-horizon A/B test at α = 0.05 but plans to check the p-value after each of 10 equally spaced batches and stop at the first batch with p < 0.05. Treating the 10 looks as independent tests, what is the probability of at least one false positive?
MCQ-16A team runs a group sequential test on checkout conversion with four equally spaced analyses and an overall two-sided alpha of 0.05, using O'Brien-Fleming boundaries. At the first analysis the observed z-statistic is 2.10, a two-sided p-value of 0.036. What should the team do?
MCQ-7A team runs a small pilot comparing two checkout flows. Flow A: 3 of 4 testers completed checkout. Flow B: 1 of 6 testers completed checkout. They want a p-value for the difference in completion rates and plan a 2×2 analysis. Which analysis is appropriate?
MCQ-8A team compares mean latency between two independently sampled groups and plans a pooled-variance Student's t-test. Which set of assumptions must hold for the p-value to be valid?
MCQ-12A team compares mean checkout latency between a control arm (n = 40) and a treatment arm (n = 90). The sample standard deviations are 12 ms and 48 ms. They had planned a pooled two-sample t-test. Which response is correct?
MCQ-21A latency A/B test has 20 requests in the treatment arm (s = 90 ms) and 500 in control (s = 25 ms). The team runs scipy.stats.ttest_ind(treatment, control) with the default equal_var=True, gets p = 0.04, and ships the change. What is the correct read of that p-value?
MCQ-10You are evaluating an A/B test on API execution times where sample sizes are moderate ($n=45$ per variant). Both distributions exhibit severe positive skewness but share equal variance and similar shape. You must evaluate whether execution time differs between variants without relying on normality assumptions. Which non-parametric procedure is most appropriate, and how should its result be interpreted under these conditions?
MCQ-11In which scenario does the standard non-parametric empirical bootstrap fail to produce asymptotically valid confidence intervals?
MCQ-9An online controlled experiment randomizes at the user level. The analyst evaluates the conversion-rate change with an unadjusted two-sample z-test that treats each raw pageview as an independent observation. What is the primary statistical consequence of this unit-of-analysis mismatch?
MCQ-25A team pre-registers a 14-day A/B test on checkout conversion with a 50/50 user-level split and 24,000 users expected. At the end the arms hold 12,340 and 11,660 users, and the primary metric shows a +4.2% relative lift, 95% CI [+0.6%, +7.9%]. What should the report say?
MCQ-20A team is replacing a JSON serialization library and pre-specified an equivalence margin of ±5 ms on p99 latency. They benchmark 180 requests per arm: the observed difference (new − old) is +3.5 ms, the 90% CI is [+2.1, +4.9] ms, the 95% CI is [+1.8, +5.2] ms, and a two-sided test of no difference gives p < 0.001. Which conclusion do the data support?
MCQ-19A query-planner canary runs at 5% of traffic. Control serves 1.2% errors and the canary serves 1.5%, a difference of +0.3 percentage points (pp), with a 95% confidence interval of [-0.1 pp, +0.7 pp] and a two-sided p-value of 0.12. The error budget has 0.5 pp of headroom before the burn-rate page fires, and the team promotes a change only when the run rules out a breach of that headroom. Which line belongs in the review record?
MCQ-17A checkout team files a plan before launch: a two-sided test of conversion at α = 0.05, with 250,000 sessions per arm. After the run, treatment conversion is 0.4% higher and the two-sided p-value is 0.07. An engineer recomputes the same data as a one-sided test in the observed direction, gets p = 0.035, and proposes reporting 'significant at α = 0.05'. Which statement about the proposed report is correct?
MCQ-22An experiment team runs a two-arm test. The pre-registered primary metric is not significant (p=0.31). After seeing the data, they try 3 outlier rules, 4 user segments, 2 time windows, and 3 metric definitions. One combination, the largest user segment, yields p=0.04, and they report that as the confirmatory result. What does the p=0.04 support?
MCQ-23A checkout change was tested on 180,000 users per arm against a 3.0% baseline conversion rate. The primary metric rose 3.9% relative (p = 0.04), and among 24 secondary metrics one moved at p = 0.03. A replication on a fresh 180,000 users per arm showed +0.7% relative (p = 0.71). Which reading of the two runs is most defensible?
MCQ-24A signup-conversion test ran 2,000 users per arm: control 120 conversions, treatment 140. Under a Beta(1,1) prior on each arm the posterior gives P(treatment better) ≈ 0.90. The team's ship rule is P(treatment better) > 0.95. Which conclusion does this result support?
MCQ-18A payments team is about to run one A/B test on a redesigned pricing page. With the sample size they can afford, the false-positive and false-negative rates trade off as follows: at α=0.01, FP 1% and FN 35%; at α=0.05, FP 5% and FN 20%; at α=0.10, FP 10% and FN 12%; at α=0.15, FP 15% and FN 9%; at α=0.20, FP 20% and FN 7%. Shipping a variant that does not actually help costs an estimated $300,000 a year in support load and lost conversion; failing to ship a variant that would have helped costs an estimated $400,000 a year. The team treats the two outcomes as equally likely a priori. Which threshold should the team use?