Hypothesis Testing
The framework for deciding whether an observed effect is likely real or could plausibly be noise — null hypotheses, p-values, and the two ways a test can be wrong.
Try answering in your head first, then click a question to check the model answer.
Q1.Why is "the p-value is the probability the null hypothesis is true" a common but incorrect interpretation?(show answer)
The p-value is calculated assuming the null hypothesis is true — it's the probability of seeing data this extreme (or more) under that assumption, not the probability the assumption itself is correct. Confusing these is a very common statistical error; correctly interpreting a low p-value only tells you the observed data would be unlikely under the null, giving you grounds to reject it, not a direct probability that the null is false.
Q2.Explain the trade-off between Type I and Type II errors.(show answer)
Type I error (false positive) is rejecting a true null hypothesis; Type II error (false negative) is failing to reject a false null hypothesis. Lowering the significance threshold (α) to reduce Type I errors makes the test more conservative, which increases the Type II error rate (lower power) for a fixed sample size — you can't reduce both simultaneously without collecting more data.
Q3.Why does a statistically significant result (p < 0.05) not necessarily mean the effect is practically important?(show answer)
With a large enough sample size, even a tiny, practically meaningless effect can become statistically significant, since the standard error shrinks as sample size grows. Statistical significance only tells you the effect is unlikely to be pure noise — it says nothing about whether the effect size is large enough to matter for a real-world decision. Effect size and confidence intervals should be reported alongside p-values for this reason.
Q4.How would you increase the statistical power of a hypothesis test?(show answer)
Increase the sample size (reduces standard error), increase the significance threshold α (though this raises Type I error risk), target/expect a larger effect size, or reduce measurement variability/noise in the data collection itself. Sample size is usually the most controllable lever in practice.