Regression Analysis
Modeling a relationship between variables — linear regression's assumptions, and what R² does and doesn't tell you.
What it is
Regression fits a model predicting a numeric outcome from one or more input variables. Simple linear regression fits y = β₀ + β₁x + ε, choosing β₀, β₁ to minimize the sum of squared errors (ordinary least squares).
Key assumptions
Linear regression's inference (p-values, confidence intervals) assumes: a genuinely linear relationship, independent errors, constant error variance across the range of x (homoscedasticity), and approximately normally distributed errors. Violating these doesn't necessarily break the point predictions, but it invalidates the statistical inference around them.
Interpreting R²
R² is the proportion of variance in the outcome explained by the model — it is not a measure of whether the model's predictors are the right ones, and a high R² doesn't imply causation. Adding more predictors mechanically increases R² even if they're useless, which is why adjusted R² (which penalizes added predictors) is often reported instead.
Correlation ≠ causation
A statistically significant coefficient means the variables move together in the data — it says nothing about which one is causing the other, or whether a third, unmeasured variable is driving both.
Prerequisite
Builds on distributions (the error-term assumptions) and hypothesis testing (how a coefficient's p-value is computed).