Regression Analysis
Modeling a relationship between variables — linear regression's assumptions, and what R² does and doesn't tell you.
Try answering in your head first, then click a question to check the model answer.
Q1.Why does a high R² not necessarily mean a regression model is good or that the relationship is causal?(show answer)
R² only measures how well the model fits the data it was fit on — a model can achieve high R² through overfitting (fitting noise, especially with many predictors relative to data points) and still generalize poorly to new data. It also says nothing about causation — a high R² can come from confounding variables or spurious correlation, not the predictor actually causing changes in the outcome.
Q2.What problem does multicollinearity cause, and how would you detect it?(show answer)
When predictors are highly correlated with each other, the regression can't cleanly attribute the outcome's variance to one predictor versus another, making individual coefficient estimates unstable (large standard errors, coefficients that flip sign with small data changes) even if the model's overall predictions remain reasonable. It's commonly detected via the Variance Inflation Factor (VIF) — a high VIF (commonly >5 or >10) on a predictor flags it as problematically collinear with others.
Q3.Why is logistic regression's output interpreted as a probability, and what function makes that possible?(show answer)
Logistic regression applies the sigmoid (logistic) function to a linear combination of predictors, which squashes any real-valued input into the range (0, 1) — making the output directly interpretable as a probability of the positive class. Plain linear regression has no such constraint and could output values like 1.4 or -0.2, which aren't valid probabilities.
Q4.What are residuals, and why would you examine a residual plot after fitting a regression model?(show answer)
Residuals are the differences between actual and predicted values. Plotting them (e.g. residuals vs. fitted values) checks the model's core assumptions: residuals should be randomly scattered around zero with no pattern. A visible pattern (like a curve, or a funnel shape indicating growing variance) suggests the linear model is misspecified — e.g. missing a nonlinear term, or the relationship isn't actually linear at all — which R² alone wouldn't reveal.