Q&A-1A linear regression on 1,000 features with only 200 labelled samples overfits badly. Walk me through what PCA changes about the fit, and how you would pick the number of components.
Answer
The 60-second answer. The problem is parameter count against samples. Ordinary least squares on 1,000 features estimates 1,000 coefficients from 200 rows. The matrix XᵀX has rank at most 200, so infinitely many coefficient vectors fit the training data exactly, and the one the solver returns is mostly fitted to noise. PCA fixes the ratio by replacing the 1,000 raw features with k linear combinations, chosen to capture the most variance. If k is 20, the regression estimates 20 coefficients from 200 rows, which is a well-posed problem. I'd choose k by cross-validation on the pipeline, not by a variance rule.
Mechanism. Standardise the features, then take the top k eigenvectors of the covariance matrix. The projected features Z = XW_k are uncorrelated, and their variances are the top k eigenvalues. Regress the target on Z. The model is still linear in the original features, but it can only move along k directions.
Arithmetic (hypothetical figures). Parameters per sample: 1,000 / 200 = 5 for the raw model, 20 / 200 = 0.1 for k = 20. Forming the k-dimensional Gram matrix costs about n·k² = 200 × 400 = 80,000 multiply-adds, against n·p² = 200 × 1,000,000 = 2e8 for the raw features.
Choosing k. A scree plot gives a starting range. Then sweep k over a grid, fitting PCA and the regression inside each cross-validation fold, and pick the k with the best held-out error, preferring the smaller k when scores tie within noise. Stop at the point where extra components stop helping validation error.
Trade-off against supervised reduction. PCA ignores the target. Its top components keep high-variance directions, which may carry none of the signal, and then the model gets worse as k grows toward the useful range. Partial least squares or a ridge penalty on all 1,000 features are the alternatives when the signal sits in low-variance directions. PCA earns its place when the structure is in the high-variance directions, or when a fixed low-dimensional representation is needed for other reasons.
Failure modes. The first is leakage: fitting PCA on all 200 rows before splitting, so the test rows shape the directions and the held-out score is optimistic. Detect it by putting PCA inside the pipeline and checking that the cross-validated score drops after the fix. The second is scale: features in different units let one feature dominate the first component. Detect it by comparing loadings with and without standardisation.
Curated: · Written: · Reviewed: