Skip to content
Tech Interview Prep
Technical interview guideNew

Principal Component Analysis (PCA)

Linear dimensionality reduction that rotates features onto orthogonal directions of maximum variance, so a few components keep most of the information. Interviews probe the covariance and SVD derivation, scaling and leakage discipline, and what PCA cannot do.

Read
14 min
Practice MCQs
25
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: Mathematics is version-independent. Code examples target scikit-learn 1.5 or later (sklearn.decomposition.PCA with svd_solver auto, full, arpack, randomized and covariance_eigen; whiten and n_components as documented). Default behaviour: centres features, does not scale them, and fits on the data passed to fit..

Interview Q&A

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

Q&A-1

A linear regression on 1,000 features with only 200 labelled samples overfits badly. Walk me through what PCA changes about the fit, and how you would pick the number of components.

Answer

The 60-second answer. The problem is parameter count against samples. Ordinary least squares on 1,000 features estimates 1,000 coefficients from 200 rows. The matrix XᵀX has rank at most 200, so infinitely many coefficient vectors fit the training data exactly, and the one the solver returns is mostly fitted to noise. PCA fixes the ratio by replacing the 1,000 raw features with k linear combinations, chosen to capture the most variance. If k is 20, the regression estimates 20 coefficients from 200 rows, which is a well-posed problem. I'd choose k by cross-validation on the pipeline, not by a variance rule.

Mechanism. Standardise the features, then take the top k eigenvectors of the covariance matrix. The projected features Z = XW_k are uncorrelated, and their variances are the top k eigenvalues. Regress the target on Z. The model is still linear in the original features, but it can only move along k directions.

Arithmetic (hypothetical figures). Parameters per sample: 1,000 / 200 = 5 for the raw model, 20 / 200 = 0.1 for k = 20. Forming the k-dimensional Gram matrix costs about n·k² = 200 × 400 = 80,000 multiply-adds, against n·p² = 200 × 1,000,000 = 2e8 for the raw features.

Choosing k. A scree plot gives a starting range. Then sweep k over a grid, fitting PCA and the regression inside each cross-validation fold, and pick the k with the best held-out error, preferring the smaller k when scores tie within noise. Stop at the point where extra components stop helping validation error.

Trade-off against supervised reduction. PCA ignores the target. Its top components keep high-variance directions, which may carry none of the signal, and then the model gets worse as k grows toward the useful range. Partial least squares or a ridge penalty on all 1,000 features are the alternatives when the signal sits in low-variance directions. PCA earns its place when the structure is in the high-variance directions, or when a fixed low-dimensional representation is needed for other reasons.

Failure modes. The first is leakage: fitting PCA on all 200 rows before splitting, so the test rows shape the directions and the held-out score is optimistic. Detect it by putting PCA inside the pipeline and checking that the cross-validated score drops after the fix. The second is scale: features in different units let one feature dominate the first component. Detect it by comparing loadings with and without standardisation.

Curated: · Written: · Reviewed:

Q&A-2

In plain terms, what does the first principal component of a dataset capture, and how would you find it?

Answer

The 60-second answer. The first principal component is the unit direction along which the centred data has the greatest variance. Among all lines through the data, projecting onto it keeps the most spread. You find it by centring the data, forming the covariance matrix, and taking the eigenvector with the largest eigenvalue. That eigenvalue is the variance captured along the direction, and the eigenvectors of a symmetric covariance matrix are orthogonal, so the later components are the remaining orthogonal directions in decreasing order of variance.

Worked 2D example. Four centred points (mean zero already): (2,2), (-2,-2), (1,-1), (-1,1). Using division by n, which is an assumption here:

Var(x) = (4+4+1+1)/4 = 2.5
Var(y) = (4+4+1+1)/4 = 2.5
Cov(x,y) = (4+4-1-1)/4 = 1.5

C = [[2.5, 1.5],
     [1.5, 2.5]]

The eigenvalues satisfy (2.5 - λ)² = 1.5², so λ = 4 and λ = 1. The eigenvector for λ = 4 is (1,1)/√2, and for λ = 1 it is (1,-1)/√2. The total variance is 2.5 + 2.5 = 5 = 4 + 1, so PC1 carries 4/5 = 80% of it.

Project onto PC1:

(2,2)  · (1,1)/√2 =  4/√2 =  2√2 ≈  2.83
(-2,-2)· (1,1)/√2 = -2√2 ≈ -2.83
(1,-1) · (1,1)/√2 =  0
(-1,1) · (1,1)/√2 =  0

The scores have mean 0 and variance (8+8+0+0)/4 = 4, which matches λ = 4. No other line through the origin gives a larger value, because the variance of the projection onto a unit vector u is uᵀCu, and the largest value of that quadratic form on the unit sphere is the largest eigenvalue.

Where it goes wrong. PCA is sensitive to scale. If one feature is in metres and another in millimetres, the first component can simply follow the unit choice, so standardise first when the features are not on comparable scales. The directions are also linear, so curved structure is not captured by PC1. Variance here means spread, not importance for a label: a direction can have high variance and be useless for prediction.

Curated: · Written: · Reviewed:

Q&A-3

Fit centred PCA with one component on (9,-1), (11,1), (9.5,0.5) and (10.5,-0.5). Then score the batch (11,-1), (13,1), (12.5,0.5), (13.5,-0.5) two ways: centring with the training mean, or centring with the batch's own mean. Which total residual is the right one, and what do you ship?

Answer

The 60-second answer. Centring is part of the fitted transformation, so it uses the training mean and nothing else. Centring with the batch's own mean makes the batch look well reconstructed, but only because it throws away the batch's location, which is the signal you need to detect drift. Ship the training-mean pipeline and treat its larger residual as the alarm.

Fit. The training mean is (10,0). Centred, the points are (-1,-1), (1,1), (-0.5,0.5), (0.5,-0.5). The covariance with divisor n is [[0.625, 0.375], [0.375, 0.625]], with eigenvalues 1.00 and 0.25. PC1 is (1,1)/√2 and PC2 is (1,-1)/√2. Keeping one component means the residual is the projection onto PC2, which is (x − y)/√2.

Pipeline A: training mean. The batch mean is (12.5, 0), so the shifted points are (1,-1), (3,1), (2.5,0.5), (3.5,-0.5). Their PC2 residuals are (x − y)/√2, which gives 2/√2, 2/√2, 2/√2 and 4/√2. The squared residuals are 2, 2, 2 and 8, for a total of 14. The PC1 scores are 0, 4, 3 and 3 over √2, with mean 2.5/√2 ≈ 1.77. That offset is the shift in the batch, projected onto PC1.

Pipeline B: batch mean. The batch centred on its own mean is (-1.5,-1), (0.5,1), (0,0.5), (1,-0.5). The PC2 residuals are -0.5/√2, -0.5/√2, -0.5/√2 and 1.5/√2. The squared residuals sum to 0.125 + 0.125 + 0.125 + 1.125 = 1.5. The PC1 scores now have mean zero, so the offset is gone.

The two totals reconcile exactly. The batch's shift from the training mean is (2.5, 0), and its PC2 component is 2.5/√2, with squared length 3.125 per point. Four points give 12.5, and 1.5 + 12.5 = 14.

import numpy as np
mu = np.array([10.0, 0.0])
W = np.array([1, 1]) / np.sqrt(2)          # PC1, fitted on training
B = np.array([[11, -1], [13, 1], [12.5, 0.5], [13.5, -0.5]])
for c in (mu, B.mean(axis=0)):
    Z = B - c
    s = Z @ W
    r = np.sum((Z - np.outer(s, W))**2)
    print(c, round(s.mean(), 3), round(r, 3))  # residual 14.0 vs 1.5

Why B is the wrong pipeline. The transform is a fixed function of the training data. Re-centring per batch makes it a different function, so a model downstream gets scores that no longer mean the same thing from one batch to the next. Pipeline B also lowers the residual you would use to detect drift. In this example the residual falls by roughly a factor of nine precisely because the batch moved.

Standardising. Scaling each feature by its training standard deviation must follow the same rule. Compute the scale on training data only and reuse it. Recomputing the scale on the batch would make the residual look fine while hiding a change in variance.

Failure modes. (1) Per-batch centring, which hides location drift. Detect it by comparing the mean PC1 score on each batch with zero. A nonzero mean means the batch was not centred on the training mean. (2) A residual threshold set from training data that is too tight. Detect it by checking the training residual distribution before deciding what counts as drift.

Curated: · Written: · Reviewed:

Q&A-4

Your PCA runs on two channels, one logged in metres and one in millimetres. Do you standardise before running PCA, and how does the unit choice change the first component?

Answer

The 90-second answer. PCA finds the orthogonal directions of greatest variance in the centred data. Those directions are the eigenvectors of the covariance matrix. Standardising each feature first turns that matrix into the correlation matrix. The choice matters because covariance-matrix PCA measures variance in whatever units the data was logged in. Before I run anything, I ask whether the raw units carry meaning. If both channels share a physical unit and their spread is signal, I keep the raw scale. If the units differ, or the spread just reflects how each sensor was logged, I standardise.

Worked example (hypothetical figures). Channel x is a length in metres with standard deviation 0.2 m. Channel y is an offset in millimetres with standard deviation 50 mm. Their correlation is 0.6, so the covariance is 0.6 × 0.2 × 50 = 6, and the covariance matrix is

[[0.04, 6], [6, 2500]]

The eigenvalues are about 2500.01 and 0.026. PC1 is roughly (0.0024, 1), and it explains 99.999% of the variance. PC1 is essentially the millimetre channel. Now log y in metres, so its standard deviation is 0.05 m and the covariance is 0.006. The matrix becomes [[0.04, 0.006], [0.006, 0.0025]], with eigenvalues about 0.0409 and 0.0016. PC1 becomes roughly (0.99, 0.14) and explains about 96%. Nothing about the data changed, only the unit, and the first component moved from the y axis toward the x axis.

Standardising gives the correlation matrix [[1, 0.6], [0.6, 1]], with eigenvalues 1.6 and 0.4. PC1 is (1, 1)/√2 and explains 80%. That direction does not depend on units, so the answer no longer depends on which unit was chosen.

The trade-off. Standardising suits incomparable units, because raw variance would only reflect the largest unit. It is the wrong default when the units are shared and a large swing in one channel is real signal. There, standardising gives a quiet channel the same weight as a loud one, and it can lift sensor noise into a component. Check this by comparing each component's explained variance with the variance you would expect from noise alone.

Practical note. scikit-learn's PCA does not standardise by default. Adding StandardScaler in front of it gives correlation-matrix PCA. Fit the scaler on the training split only, so test statistics do not leak into the components.

Curated: · Written: · Reviewed:

Q&A-5

Walk me through deriving the covariance matrix of centred data, showing that it is symmetric and positive semidefinite, why its eigenvectors are orthogonal, and what the eigenvalues mean for variance.

Answer

Start with the shape of the data. X has n rows and d features, and each column is centred so its mean is zero. The sample covariance is S = X^T X / (n-1), a d by d matrix whose (j,k) entry is the covariance of feature j with feature k. Dividing by n instead changes the eigenvalues by a constant factor but not the eigenvectors.

Symmetry. (X^T X)^T = X^T X, so S^T = S. Covariance is symmetric because cov(j,k) = cov(k,j).

Positive semidefinite. For any vector v, v^T S v = (Xv)^T (Xv) / (n-1) = ||Xv||^2 / (n-1) >= 0. Xv is the projection of every row onto v, so for a unit vector v the quantity v^T S v is the variance of the data along v. It cannot be negative, which is why the eigenvalues of S are non-negative.

Orthogonality. Let S v1 = l1 v1 and S v2 = l2 v2 with l1 not equal to l2. Symmetry gives l1 (v1^T v2) = (S v1)^T v2 = v1^T S^T v2 = v1^T S v2 = l2 (v1^T v2). So (l1 - l2)(v1^T v2) = 0, and v1^T v2 = 0. Eigenvalues are real for the same reason. When eigenvalues repeat, the eigenspace is still a subspace, and the spectral theorem lets you choose an orthonormal basis inside it, so the full set of eigenvectors can always be orthonormal.

Eigenvalues as variance. For a unit eigenvector v with S v = l v, the variance of the projections is v^T S v = l. The covariance between the projections onto two eigenvectors is v1^T S v2 = l2 (v1^T v2) = 0, so the components are uncorrelated. Summing over the basis, the eigenvalues add up to trace(S), the total variance, and a rotation to the eigenbasis does not change that total.

# Hypothetical worked example, 4 points in 2D
# (1,2), (2,4), (3,6), (4,8); centred: mean (2.5, 5)
# deviations: (-1.5,-3), (-0.5,-1), (0.5,1), (1.5,3)
# Sxx = 5, Syy = 20, Sxy = 10; n-1 = 3
S = [[5/3, 10/3], [10/3, 20/3]]
# trace = 25/3, det = 100/9 - 100/9 = 0
# eigenvalues: 25/3 and 0
# eigenvector for 25/3: (1,2)/sqrt(5)
# variance along it: (1/5) * (1,2) S (1,2)^T = (1/5)(25/3)(5) = 25/3
# variance along the orthogonal direction (2,-1)/sqrt(5): 0

The data lie on one line, so one eigenvalue carries all the variance and the other is zero. Note that the zero eigenvalue is exact here only because the points are collinear by construction. With real measurements, expect a small positive number.

Failure modes.

  1. Forgetting to centre. X^T X is then a second-moment matrix about the origin, and its top eigenvector points toward the mean rather than along the spread. Detect it: the projections have a non-zero mean, and the first component is nearly parallel to the mean vector.
  2. A non-symmetric or non-PSD matrix. Pairwise deletion of missing values can produce a matrix that is not symmetric or has clearly negative eigenvalues. Detect it by checking symmetry and the smallest eigenvalue before decomposing. Tiny negative values such as -1e-15 are rounding; a large negative value means the input was not a valid covariance.
  3. Using a general eigen solver. numpy.linalg.eig makes no orthogonality promise and can return complex values for near-repeated eigenvalues. For a symmetric matrix, numpy.linalg.eigh returns real eigenvalues in ascending order and orthonormal eigenvectors. Check that the eigenvector matrix Q satisfies Q^T Q = I to within rounding.

Curated: · Written: · Reviewed:

Q&A-6

Why must you centre the features before running PCA?

Answer

The short answer. PCA takes the eigenvectors of the sample covariance matrix, and that matrix is only the covariance when each feature has mean zero. Centring makes the covariance matrix S = Xᵀ X / (n − 1) equal to the quantity whose eigenvectors are the principal components. Each eigenvalue is then the variance of the data along its eigenvector, and the top eigenvector is the direction of greatest spread. Without centring, the matrix you decompose is no longer the covariance, so the components do not carry that meaning.

The rule. For centred data, the variance of the projection of X onto a unit vector v is vᵀ S v. The largest value of that quantity over unit vectors is the largest eigenvalue of S, reached at its eigenvector. This is the fact that makes the top component the variance-maximising direction, and it holds only under the centring assumption.

What goes wrong otherwise. If the features are not centred, the matrix Xᵀ X / n is a second-moment matrix. It mixes the spread of the data with the location of its mean, so its top eigenvector can point toward the mean. Reading that direction as the direction of maximum variance is the error.

Other assumptions to state. The n − 1 denominator is the unbiased sample convention. It rescales every eigenvalue by the same factor and leaves the eigenvectors unchanged. Standardising the features is a separate choice that changes S into a correlation matrix, and it should be named whenever the features have different units.

Curated: · Written: · Reviewed:

Q&A-7

You centre a 40-sample by 5,000-feature matrix and run PCA via SVD. How many components can have nonzero variance, which singular vectors do you get without forming a 5,000 by 5,000 matrix, and what goes wrong if you skip centring on this data?

Answer

The 90-second answer. Assume X is 40 by 5,000, already centred by column, so every column sums to zero. Each row is a sample. Centring forces the rows to sum to zero, so the rows are linearly dependent and rank(X) is at most n-1 = 39. The covariance C = X^T X/(n-1) is 5,000 by 5,000 but has rank at most 39. So at most 39 components have nonzero variance, and the other 4,961 eigenvalues are zero up to rounding. Forming C is wasteful and also misleading, because its zero eigenvalues are numerical noise.

The cheap route works on the 40 by 40 Gram matrix K = X X^T. Take its eigendecomposition K = U_n Λ U_n^T, with Λ = S^2 for the nonzero part. Then the left singular vectors are the eigenvectors u_i of K, the singular values are s_i = sqrt(λ_i), and the principal axes are v_i = X^T u_i / s_i. Check the step: X^T u_i / s_i has length sqrt(u_i^T X X^T u_i)/s_i = s_i/s_i = 1, and it is orthogonal to v_j for j not equal to i. So the 39 right singular vectors come out of a 40 by 40 problem, at cost about n^2 p = 8e6 multiplications for the Gram matrix, against p^2 n = 1e9 for C.

Worked figures, hypothetical. Take a toy case with n = 4 samples, p = 3. After centring, the rows sum to zero, so rank(X) is at most 3. Suppose the Gram matrix has eigenvalues 9, 4, 0 for its nonzero part and a zero eigenvalue for the dependent direction. Then s = (3, 2), and the covariance eigenvalues are 9/3 = 3 and 4/3, about 1.33. The third eigenvalue is exactly zero in exact arithmetic. In floating point it appears as about 1e-15 times the largest, so you must set a rank threshold, for example s_i below 1e-10 times s_1, and drop those components. Reading the zero eigenvalues as variance would report a direction with variance of about 1e-16 as if it were real structure.

Failure modes. (1) Skipping centring: the first right singular vector then points roughly toward the sample mean, so PC1 describes the location of the data and not its spread. The check is that column means are near zero before factorising. (2) Using the wrong divisor: with n = 40 the eigenvalues are off by a factor 40/39 if you divide by n, which changes the explained-variance ratios slightly and breaks any comparison with a library that uses n-1. (3) Trusting the nonzero count: the number of components that can be nonzero is n-1 after centring, not p. Asking for 5,000 components silently returns 4,961 zeros, and a downstream model reads them as features with no information.

Trade-off. The Gram route is exact in real arithmetic, but it squares the condition number just as X^T X does, because K = X X^T has eigenvalues s_i^2. The thin SVD of X avoids that squaring and is the safer path when small singular values matter. For 40 by 5,000 data, most libraries do the SVD of X directly, which costs about n^2 p, the same order as the Gram route.

Curated: · Written: · Reviewed:

Q&A-8

You have n = 300 samples and d = 20,000 features, and you need the top 50 principal components. Compare computing them from the covariance eigendecomposition with computing them from the SVD of the centred data matrix, and say which one you would run.

Answer

The 90-second answer. Both routes return the same principal axes. The right singular vectors of the centred matrix X (n by d) are the eigenvectors of C = XᵀX/(n−1), so the question is only how each route gets there. The covariance route forms C explicitly, a d by d matrix, and eigendecomposes it. The SVD route factorises X directly and never builds C. At n = 300 and d = 20,000 the difference is large: C has 4×10⁸ entries, about 3.2 GB in float64, and its eigendecomposition costs on the order of d³ = 8×10¹² operations. An SVD of X costs on the order of n·d·min(n,d) = 300 × 20,000 × 300 = 1.8×10⁹. I would run the SVD, and I would not form C.

Cost. Forming C costs n·d² = 300 × 4×10⁸ = 1.2×10¹¹ operations, and eigendecomposing it costs d³. The SVD costs O(n·d·min(n,d)). Because min(n,d) = n here, the SVD never touches a d-by-d matrix, and it is cheaper by roughly two orders of magnitude. The Gram matrix G = XXᵀ (300 by 300) is another way to the same answer: decompose G, then recover the right singular vectors as v = Xᵀu/σ. That is the cheapest arithmetic at this shape, but it inherits the accuracy problem below, so it is a fallback, not the default.

# numpy 1.26, Python 3.11; X is centred, shape (n, d)
U, s, Vt = np.linalg.svd(X, full_matrices=False)   # O(n d min(n,d))
components = Vt[:50]
explained = s[:50]**2 / (n - 1)                   # eigenvalues of C

Stability. Forming C squares the condition number: κ(XᵀX) = κ(X)². A singular value ratio of 10⁴ in X becomes an eigenvalue ratio of 10⁸ in C, and float64 keeps about 16 significant digits, so the small eigenvalues of C lose accuracy once κ(X) approaches 10⁸. The SVD works on X directly, so its error tracks κ(X). The Gram route squares the conditioning too, because G = XXᵀ has the same squared singular values. Ill-conditioned data, such as near-collinear features or columns on very different scales, is where the covariance route returns trailing components that look plausible and are wrong.

Detection. Check the SVD result by reconstruction, ‖X − UΣVᵀ‖ relative to ‖X‖, and by orthogonality, VᵀV ≈ I. If the covariance route is used, compare its eigenvalues against the squared singular values of X. Disagreement in the trailing ones is the signature of squared conditioning.

When the Gram route wins. Only when the n by n decomposition is the bottleneck and the top components are well separated, for example when X is already well conditioned and many more features than samples arrive repeatedly. At 300 samples the d³ cost of the covariance route is the real problem, so the decision is between an SVD and a Gram route, not between the covariance route and anything else.

Failure mode to name. Forgetting to centre X. The SVD of uncentred data gives the top direction of the second-moment matrix, which tracks the mean rather than the variance. Detection: the first component's loadings line up with the mean vector, and the explained-variance ratio looks suspiciously high.

Curated: · Written: · Reviewed:

Q&A-9

Eight PCA eigenvalues are 4.2, 2.1, 0.9, 0.5, 0.2, 0.1, 0.06 and 0.04. Walk me through the explained variance ratio and the cumulative ratio, then tell me what keeping four components does and does not guarantee about a downstream model.

Answer

The 60-second answer. The explained variance ratio of component k is its eigenvalue divided by the sum of all eigenvalues of the covariance matrix of the centred data. Each eigenvalue is the variance of the data along its principal axis, so the ratio is the fraction of total feature variance that axis carries. The cumulative ratio is the running sum after sorting eigenvalues from largest to smallest. It tells you how much of the spread in the features the first k components keep. It does not tell you whether a model trained on those k components predicts the target better, and the two can move in opposite directions.

Worked example. Hypothetical eigenvalues from an 8-feature covariance matrix, total 8.10:

keigenvalueratiocumulative
14.2051.9%51.9%
22.1025.9%77.8%
30.9011.1%88.9%
40.506.2%95.1%
50.202.5%97.5%
60.101.2%98.8%
70.060.7%99.5%
80.040.5%100.0%
# Python 3.12, numpy 2.x
import numpy as np
lam = np.array([4.2, 2.1, 0.9, 0.5, 0.2, 0.1, 0.06, 0.04])
ratio = lam / lam.sum()
cum = np.cumsum(ratio)
k = int(np.argmax(cum >= 0.95)) + 1   # 4

Four components clear 95%, and they discard 4.9% of the variance. That discarded 4.9% is the number to look at, not the 95.1%.

Failure modes.

  1. The discarded direction carries the label. Variance is not relevance. If the target depends mostly on a low-variance direction, say the eighth, a regression on the top four components can lose most of its accuracy while the ratio still reads 95%. Detect it with cross-validated score against k, not with the ratio.

  2. Unscaled features pick the threshold for you. The eigenvalues are in the units of the inputs. A feature in millimetres dominates the spectrum that a feature in metres does not. Standardise first, or the cumulative ratio measures units, not structure.

  3. In-sample ratio on the training set. The ratio is computed from whatever data you fit on. A fit on a small or unrepresentative sample gives a confident ratio for a spectrum that will move on new data.

Alternatives and when to use each. A variance threshold (95%, 99%) needs no labels and is cheap, so it suits exploration, compression and monitoring. Choosing k by downstream cross-validation costs a training run per candidate but measures the thing you ship. In practice I use the ratio to set a budget and cross-validation to pick k inside it.

What the ratio guarantees. For the centred data, the first k components give the best rank-k linear reconstruction in squared error, and the ratio is exactly the share of that error budget kept. It guarantees nothing about labels.

Curated: · Written: · Reviewed:

Q&A-10

Kaiser's rule keeps principal components whose eigenvalue exceeds what value, and on which kind of data is that threshold meaningful?

Answer

The threshold is 1, and the rule is meaningful only on standardised data.

When each feature is scaled to unit variance, the correlation matrix has trace p, the number of features. Its eigenvalues sum to p, so the average eigenvalue is 1. Kaiser's rule keeps the components whose eigenvalue is above that average: each such component explains more variance than a single original standardised feature would.

# scikit-learn 1.5, X_std already standardised per column
from sklearn.decomposition import PCA
pca = PCA().fit(X_std)
k_kaiser = int((pca.explained_variance_ > 1).sum())

On unstandardised data the eigenvalues are in the squared units of the inputs. A threshold of 1 then depends on the units of measurement: a feature recorded in millimetres can push its eigenvalues far above 1 while a feature in metres pushes them far below. The rule has no meaning there.

The rule is a fast default, not a proof of the right count. It tends to keep too many components when features are only weakly correlated, because eigenvalues then cluster near 1.

Curated: · Written: · Reviewed:

Q&A-11

You have 400 features and 20,000 rows, and you need to keep some principal components for a downstream model. Describe how you would choose k using cumulative variance, a validation metric and a stability check, and what you would report.

Answer

The 90-second answer. Cumulative variance gives me a ceiling on how much information I discard. It does not tell me how much the model needs. So I use it to shortlist k, pick k with a cross-validated score for the actual model, and then check that the components I keep are stable enough to report. First question: are the features on comparable scales? If not, I standardise, and I fit the scaler and PCA inside each training fold, never on the full data.

Step 1: the variance curve. With 20,000 rows and 400 features, the training covariance is 400 by 400 and cheap to eigendecompose. Eigenvalues come from the SVD of the centred training matrix, so I never need the 20,000 by 20,000 matrix. The explained ratio for component i is lambda_i divided by the sum of all lambdas. Keeping k components leaves a relative reconstruction error equal to the discarded share of the total variance (Eckart-Young). I shortlist the smallest k where cumulative ratio reaches 0.90, 0.95 and 0.99, and I read the elbow in the scree plot. That gives three candidates, not an answer.

Step 2: the validation metric. For each candidate k, I fit scaler and PCA on the training folds, project, train the downstream model and score the held-out fold. I run this over a grid of k, not only the three shortlisted values, because the curve often has a flat region. I pick the k with the best mean score, and if the scores within one standard error of the best are flat, I take the smallest k in that band. Variance is a property of the features. The metric is a property of the task. A direction with little variance can carry most of the label signal, so a 0.99 cumulative ratio can still lose accuracy relative to k equal to 40.

Step 3: stability. Principal components are only defined up to sign, and near-equal eigenvalues make individual components rotate freely. I refit PCA on 20 bootstrap resamples or on the CV folds and compare the k-dimensional subspaces, using the principal angles between them or the norm of the difference of the projection matrices. I also check the eigengap between lambda_k and lambda_(k+1). A tight gap with unstable subspaces means the cut point is arbitrary, and I should either pick a k that sits at a clean gap or report the subspace, not individual components.

Failure mode I watch for. Fitting the scaler and PCA on all 20,000 rows before splitting leaks held-out variance into the basis. The CV score then looks better than production will be, and the gap grows with k. I detect it by comparing the leaky and leak-free scores on the same folds.

What I report. The chosen k and the rule that selected it. The cumulative ratio at k, and at the rejected candidates. The CV score with its spread across k, so the reader can see the flat region. The stability measure with the eigengap. The scaling choice and whether it was fitted per fold. That k was chosen for the downstream metric, so the components are not claimed to be the true latent factors.

Curated: · Written: · Reviewed:

Q&A-12

Walk me through reconstructing a centred data matrix from its first k principal components: write Z and X_hat, explain why X_hat is the orthogonal projection onto the subspace, and say what the discarded components carry.

Answer

Start with the shapes. X is n by d, mu is the training mean, and W_k is d by k with orthonormal columns taken from the top k right singular vectors (or eigenvectors of the covariance). The scores are Z = (X - mu) W_k, an n by k matrix, and the reconstruction is X_hat = Z W_k^T + mu. The mean is added back because the subspace passes through mu, not through the origin.

Why this is a projection. Let P = W_k W_k^T. Since W_k^T W_k = I, P is symmetric and P^2 = P, so it is an orthogonal projector onto the column space of W_k. Then X_hat - mu = (X - mu) P, and the residual R = X - X_hat = (X - mu)(I - P). Each residual row r satisfies W_k^T r = 0, because W_k^T (I - W_k W_k^T) = W_k^T - W_k^T = 0. The residual is orthogonal to every retained direction, which is the defining property of an orthogonal projection. The reconstruction is the closest point in the subspace to each sample, and Pythagoras gives ||x - mu||^2 = ||x_hat - mu||^2 + ||r||^2 for every row.

# numpy 1.26, Python 3.11
mu = X.mean(axis=0)
Xc = X - mu
_, s, Vt = np.linalg.svd(Xc, full_matrices=False)
Wk = Vt[:k].T
Z = Xc @ Wk
X_hat = Z @ Wk.T + mu
R = X - X_hat
assert np.allclose(R @ Wk, 0, atol=1e-8)

What the discarded components carry. With the sample covariance S = Xc^T Xc / (n - 1), the variance of the discarded directions sums to the squared residual. Take the eigenvalues 4.2, 2.1, 0.9, 0.5, 0.2, 0.1, 0.06, 0.04 (total 8.10). Keeping four retains 7.7 and discards 0.4, so the summed squared residual is (n - 1) x 0.4, and the mean per sample is 0.4 x (n - 1) / n. This is the best possible rank-4 linear reconstruction in squared error (Eckart-Young), so no other linear map of that rank does better on the training data.

The trade-off is what the residual can hide. Low-variance directions are not necessarily noise. A feature with small variance can still separate two classes, and PCA chooses directions by variance, not by the target. The residual also says nothing about nonlinear structure: a curved manifold can have a large residual under a linear subspace even when it is well described by a nonlinear model.

Failure mode to watch: fit mu and W_k on training data, then measure residuals on new data. If the test residual is much larger than the training residual, the subspace does not describe the new distribution, which usually means drift or a scaling change. Track the per-sample residual distribution, not just its mean, because a few out-of-subspace rows can dominate.

Curated: · Written: · Reviewed:

Q&A-13

Keep k of d principal components of centred data. Derive the total squared reconstruction error in terms of the eigenvalues you discarded, and explain why that quantity is the discarded explained variance.

Answer

The 60-second answer. Let X be the n by d centred data matrix and let S = XᵀX/(n−1) be the sample covariance, with eigenvalues λ1 ≥ λ2 ≥ … ≥ λd. Keeping the top k components projects each sample onto the span of the first k eigenvectors. The squared error discarded is exactly the sum of the discarded eigenvalues, scaled by the sample count. Summed over all samples, the total squared reconstruction error is (n−1) × (λ_{k+1} + … + λ_d). Dividing by the total variance gives the fraction of variance thrown away, so reconstruction error and unexplained variance are the same number viewed two ways. One correction to the brief's form: no common convention gives division by n−1. The error scales with n−1 under the 1/(n−1) covariance and with n under the 1/n covariance. Per-sample mean error is (n−1)/n × discarded sum, or exactly the discarded sum under 1/n.

Derivation. Take the SVD X = UΣVᵀ with singular values σi, so λi = σi²/(n−1) and V is orthogonal. Write X = XVVᵀ = XV_kV_kᵀ + XV_{−k}V_{−k}ᵀ. The first term is the rank-k reconstruction X̂. The residual is R = XV_{−k}V_{−k}ᵀ. Because V_{−k} has orthonormal columns, the squared Frobenius norm is ‖R‖²_F = tr(V_{−k}ᵀXᵀXV_{−k}) = Σ_{i>k} σi² = (n−1) Σ_{i>k} λi. Total variance is tr(S) = Σ λi = ‖X‖²_F/(n−1), so ‖X‖²_F = (n−1) Σ λi. Therefore the fraction of squared norm lost in reconstruction is ‖R‖²_F/‖X‖²_F = Σ_{i>k} λi / Σ λi, which is one minus the cumulative explained variance ratio. The tie holds because projection onto orthonormal directions is Pythagorean: the retained and discarded parts are orthogonal, so their squared norms add to the total.

Worked example (hypothetical figures, n = 101 so n−1 = 100). Eigenvalues 4.2, 2.1, 0.9, 0.5, 0.2, 0.1, 0.06, 0.04 sum to 8.1. Keep k = 4. Discarded sum is 0.2 + 0.1 + 0.06 + 0.04 = 0.40. Explained ratio is (8.1 − 0.40)/8.1 = 0.951, and discarded fraction is 0.049. Summed squared error is 100 × 0.40 = 40. Per-sample mean squared error is 40/101 ≈ 0.396, which equals (100/101) × 0.40. Per-feature average is that value divided by d.

Failure modes. Comparing reconstruction error across two runs with different n or different covariance conventions gives misleading numbers unless both are normalised the same way. The fraction form is the safe one to report. Reconstruction error measures what PCA preserves in squared terms, not what a downstream model needs, so low error does not guarantee a better classifier. Unscaled features make the eigenvalues track units, so the error fraction reflects the largest-unit features, not structure.

Curated: · Written: · Reviewed:

Q&A-14

Which criterion do the top k principal components minimise among all rank-k linear projections, and what does the guarantee assume about the data?

Answer

The 60-second answer. Among all rank-k linear projections, the top k principal components minimise the total squared reconstruction error. The guarantee is the Eckart-Young result: for centred data, the truncated SVD is the best rank-k approximation under the Frobenius norm, and the top k right singular vectors are the top k eigenvectors of the covariance matrix. Two assumptions carry the claim. The loss is squared error summed over features, and the data is centred, so the projection is fitted around the mean. Under a different loss, such as absolute error, the top eigenvectors are no longer guaranteed to be optimal.

Why it holds, in one line. For centred data, the total squared norm equals the variance kept by the subspace plus the residual it discards. The total is fixed, so minimising the residual is the same as maximising the kept variance, and the top eigenvectors do that.

Worked check. Take the centred points (2,0), (-2,0), (0,1), (0,-1). Their covariance-based total sum of squares is 10, with variance 8 along e1 and 2 along e2. Keeping e1 leaves a residual of 2, and keeping e2 leaves 8. The top component is the better choice, as the rule says.

Where the guarantee does not reach. It says nothing about prediction. The components are chosen without the target, so a high-variance direction may carry no signal for a label. It also says nothing about curved structure: a linear subspace is the only shape considered.

Failure mode to check. Without centring, the best subspace passes through the origin instead of the mean, and the first component partly describes the mean rather than the spread.

Curated: · Written: · Reviewed:

Q&A-15

Two features have correlation 0.6 and are standardised, so PC1 is (0.707, 0.707) with eigenvalue 1.6. You then multiply feature B by 10 and refit PCA on the covariance matrix. What happens to each feature's loading on PC1 and to its correlation with PC1, and which number should you report as driver strength?

Answer

What I say first. The number that answers 'which feature drives PC1' is the correlation between each feature and the component, because it does not depend on units. Loadings from a covariance PCA do depend on units, so after the rescale they tell a different story from the one the team had before.

Standardised case. R = [[1, 0.6], [0.6, 1]] has eigenvalues 1.6 and 0.4. The unit-norm PC1 vector is (0.707, 0.707). The scaled loading is 0.707 * sqrt(1.6) = 0.894 for each feature, and that equals corr(feature, PC1) because both features have unit variance. The two features are symmetric, so neither drives PC1 more than the other.

After B is multiplied by 10. The covariance matrix becomes [[1, 6], [6, 100]]. Its trace is 101 and its determinant is 64, so the eigenvalues are about 100.36 and 0.64. The first eigenvector is about (0.060, 0.998). The scaled loadings are 0.060 * sqrt(100.36) = 0.60 for A and 0.998 * 10.02 = 10.00 for B.

The correlations are: corr(A, PC1) = 0.60 / sd(A) = 0.60 / 1 = 0.60, and corr(B, PC1) = 10.00 / sd(B) = 10.00 / 10 = 0.998.

The raw loading on B has grown from 0.89 to 10.0 even though the data's dependence structure has not changed. The B loading looks like a larger driver only because B is measured in larger units. The correlation shows that A's relationship to PC1 has weakened, from 0.89 to 0.60, because B now dominates the component. So I report the correlations, state that the covariance PCA was used, and give the explained variance of each component.

Failure modes.

  1. Reading covariance loadings as importance. A team that compares raw loadings across features with different units will rank the feature with the largest scale first. Detect it by computing correlations and checking whether the ranking changes.
  2. Sign ambiguity. The first vector could come back as (-0.060, -0.998). Compare signs within one component only, and fix the sign convention before reporting.
  3. Mixed reporting. A table that combines unit-norm weights from one run with scaled loadings from another invites the wrong comparison. Name the convention in every column header.

What I report. The convention used, each feature's correlation with each component, the explained variance ratio, and a note that the covariance PCA ranking depends on units.

Curated: · Written: · Reviewed:

Q&A-16

Your PCA on 30 customer-behaviour features gives a first component loading on recency, spend and support tickets, and product wants to label it 'engagement'. Why is the label not yet defensible, and what single check would you run first?

Answer

The 90-second answer. A principal component is a unit vector v in feature space. A row's score on it is the dot product of the standardised row with v. So PC1 is a weighted sum of features, and 'engagement' is a claim about those weights, not a measured quantity. Two properties make the claim fragile. First, v and -v have identical variance, so 'high engagement' could equally be 'low recency' depending on which sign the solver returned. Second, the loadings come from the whole covariance matrix, so removing one feature can move the direction. Before naming it I would run one check: refit PC1 on a bootstrap resample of the training rows, with the sign fixed, and see whether the direction stays put.

Why the sign and the weights matter. With standardised features, the loading on feature j is v_j, and the correlation between feature j and the component is v_j times sqrt(lambda_1). Those correlations are the numbers to read, because they are in the units of the features. A correlation of -0.6 between recency and PC1 means the score rises as recency falls. That is the sentence the product manager needs, and it is more honest than 'engagement'.

# numpy 1.26, Python 3.11; X_train standardised with training mean and sd
rng = np.random.default_rng(0)
vals, V = np.linalg.eigh(np.cov(X_train, rowvar=False))  # ascending order
ref = V[:, -1]
ref *= np.sign(ref[np.argmax(np.abs(ref))])             # fixed sign convention
cos = []
for _ in range(200):
    idx = rng.integers(0, len(X_train), len(X_train))
    _, Vb = np.linalg.eigh(np.cov(X_train[idx], rowvar=False))
    cos.append(abs(Vb[:, -1] @ ref))                     # abs removes the sign
print(np.percentile(cos, [5, 50]))

If the 5th percentile cosine is high, the direction is stable under resampling and a name is worth discussing. If it is low, the top loadings are noise, and the name should be dropped.

Trade-off in the check. Bootstrap stability says the direction is reproducible, but it does not say the direction means anything. A stable component can still be a mix of unrelated features that happen to co-vary in this sample. That is why the name still needs an outside test, such as whether the score separates next-quarter churn on a held-out cohort. Stability is necessary and does not establish meaning.

Failure modes.

  • Naming from the top three loadings with no stability check. Detect: the 5th percentile bootstrap cosine falls well below the median, or lambda_1 and lambda_2 are close enough to swap.
  • Outlier-driven direction. A few extreme rows set the axis. Detect: refit without the top 1% of rows by score and compare cosines.
  • Causal wording. The slide says the component 'drives' engagement. Detect: the label survives no outside test.

Rotation. Varimax on the top k loadings makes each component easier to read, but the rotated components are no longer ordered by variance, and the subspace and reconstruction error do not change. I would rotate for explanation and keep the ordered basis when a downstream step relies on variance ranking.

Curated: · Written: · Reviewed:

Q&A-17

You fit PCA on a training set and get a first component with loadings (0.8, 0.6). Refit on a bootstrap resample and the loadings come back as (-0.8, -0.6). What changed, does it affect the scores or the reconstruction, and how do you compare components across fits?

Answer

The 90-second answer. An eigenvector is defined only up to sign. If v is a principal direction, so is -v, and a solver may return either. The sign carries no information, so nothing changed about the fitted structure. A sign flip negates the score z = Xv and leaves the reconstruction unchanged, because the reconstruction is z times v, and both factors flip together. So the sign only matters when someone compares a raw loading or score across two fits, or reads the sign as meaning something. I would choose a reference fit, align each refit to it by the sign of the dot product, and apply the same sign to the scores before comparing or plotting anything.

Worked trace. Take centred data and a point x = (2, 1.5).

v = ( 0.8,  0.6): z = 2*0.8 + 1.5*0.6 =  2.5   x_hat = 2.5 * ( 0.8,  0.6) = (2.0, 1.5)
v = (-0.8, -0.6): z = 2*(-0.8) + 1.5*(-0.6) = -2.5   x_hat = -2.5 * (-0.8, -0.6) = (2.0, 1.5)

The scores have opposite signs and the reconstructions are identical. A plot of PC1 scores from the two fits is mirrored, not different.

Alignment. Use the full-data or training fit as the reference. For a replicate with loading v_b, compute s = sign(v_b · v_ref). If s is -1, multiply v_b and that replicate's PC1 score column by -1. In the trace, (-0.8, -0.6) · (0.8, 0.6) = -1, so it is flipped back. For train and test, fit once on train, store V, and project test with the stored V. Do not refit on test and match signs by eye.

Where alignment is not enough. If the first two eigenvalues are close, the individual vectors can rotate inside the near-degenerate pair from one resample to the next. Sign alignment does not fix that. Check the eigengap λ2/λ1 before reading a single component. When it is near 1, report the pair as a subspace, or say PC1 and PC2 are not separately identified.

Failure modes and detection. (1) Mirrored plots: two fits look like they disagree because one axis is reversed. Detect by computing the absolute cosine between loading vectors. (2) Sign read as meaning: a sample is called 'high on PC1' when the claim would reverse under negation. Detect by checking whether the statement survives flipping the axis. (3) Library drift: the default sign convention changes across versions, so a saved loading table no longer matches a fresh fit. Pin the convention in code, for example by making the largest-magnitude loading of each component positive, and test it.

What I report. The sign convention and the reference fit, the eigengap, bootstrap intervals on absolute loadings or on |cos| to the reference, and the spread of explained-variance ratios. Plot axes carry the convention in their label, and a sign flip is never reported as a finding.

Curated: · Written: · Reviewed:

Q&A-18

Should a StandardScaler be fitted on the training fold only, or on the full dataset, when you cross-validate a model?

Answer

The 60-second answer. Fit it on the training fold only. A StandardScaler learns a mean and a standard deviation per feature from the data it is fitted on. If those statistics come from all rows, the held-out fold has already influenced the values the model is scored on, so the cross-validated score is optimistic. The rule is simple: every data-dependent preprocessing step is fitted inside the fold, and the test fold is only transformed.

The leaky version.

# Python 3.12, scikit-learn 1.5: leaky
Xs = StandardScaler().fit_transform(X)
scores = cross_val_score(LogisticRegression(max_iter=1000), Xs, y, cv=5)

The correct version.

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(pipe, X, y, cv=5)

The pipeline refits the scaler on each training fold and applies transform alone to the test fold. The same rule covers PCA, which learns its centring vector and components from the data in the same way.

Failure mode to check. Choosing the number of PCA components from variance computed on the full dataset, then cross-validating, leaks through the same path. Tune n_components inside the search over the pipeline instead.

Curated: · Written: · Reviewed:

Q&A-19

Your forecasting pipeline standardises 400 features, keeps enough PCA components for 90% cumulative variance computed on all 150 months, then fits a regressor on months 1 to 120 and tests on months 121 to 150. Which fitted quantities leak, and how do you rebuild the evaluation under time order?

Answer

The 90-second answer. Three things leak, and the time order makes two of them worse than a random split would. The scaler's mean and standard deviation come from all 150 months, so the test months shift the centring and scale of the training rows. The 90% cumulative-variance threshold is computed from the eigenvalues of the full covariance matrix, so k is chosen with knowledge of the test period's variance. The components themselves are eigenvectors of that same full matrix. The rank constraint is the subtle part. With 120 training rows, centring leaves rank at most 119, so a fold-clean PCA has at most 119 nonzero components. If k was chosen on all 150 rows and comes out at, say, 130, the pipeline either fails or silently keeps directions that carry no variance in training. Rebuild it as one pipeline fitted on months 1 to 120 only, with k chosen by an expanding-window validation inside that period, and score the 30 test months once.

# scikit-learn 1.5, Python 3.12
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import TimeSeriesSplit, GridSearchCV

train, test = slice(0, 120), slice(120, 150)
pipe = make_pipeline(StandardScaler(), PCA(), RidgeCV(alphas=[0.1, 1, 10]))
search = GridSearchCV(pipe, {'pca__n_components': [10, 20, 40, 60, 90]},
                      cv=TimeSeriesSplit(n_splits=4))
search.fit(X[train], y[train])   # every fitted quantity sees past rows only
score = search.score(X[test], y[test])

Size of the bias. Hypothetical figures for illustration: the leaky pipeline reports R² of 0.66 on months 121 to 150, while the rebuilt pipeline reports 0.49. The gap depends on how far the test regime sits from the training regime. A stationary series gives a small gap. A series whose variance concentrates in new directions after month 120 gives a large one, because the leaked components were chosen to describe a period the model has not seen. The gap also grows with the ratio of features to rows, since with 400 features and 150 rows the top eigenvectors are dominated by sampling noise.

Where the time order changes the fix. Random k-fold inside the training window is still optimistic, because validation folds then borrow future months. Use expanding windows. The scaler and PCA must be refitted in every fold, and the rank cap must be checked in every fold, since the early folds have fewer than 119 rows.

Failure modes and detection. (1) k above the training rank: PCA warns or returns components with zero variance; log explained_variance_ and reject any k where the tail is zero. (2) Component signs flipping between refits: coefficients that look stable in one split reverse in another, so compare absolute loadings across folds. (3) Test score that improves as the leak is added: a sign the pipeline is memorising the test regime. Re-run with a shifted cutoff and check whether the gap moves with it.

Curated: · Written: · Reviewed:

Q&A-20

Your PCA on two-feature data shows the two classes overlapping on PC1, and the classes are concentric rings. Why does PCA fail here, and would you switch to kernel PCA or keep PCA for a downstream classifier?

Answer

Clarify first: is the goal a classifier or an exploratory picture? I will assume a classifier on the reduced features, with two input features x and y, and the rings are centred at the origin.

PCA keeps directions of largest variance and discards the rest. Its covariance depends only on second moments. Take two rings of radius 1 and 3 with equal counts and uniform angles. For a ring of radius r, E[x^2] = r^2/2 and E[xy] = 0. Mixing the rings 50/50 gives E[x^2] = (1/2)(1/2) + (1/2)(9/2) = 2.5, and likewise E[y^2] = 2.5 with E[xy] = 0. The covariance is 2.5 times the identity. Both directions have equal variance, so PC1 is arbitrary, and projecting onto it stacks the two rings in one interval. The class signal sits in x^2 + y^2, a nonlinear function that second moments cannot see.

Kernel PCA can recover this. With an RBF kernel, it implicitly maps points to a space where the rings separate along a few components. It computes the Gram matrix K, centres it as HKH with H = I - 11^T/n, and eigendecomposes it. Cost is the trade-off: the decomposition is O(n^3) time and the Gram matrix needs O(n^2) memory. At n = 5,000 that is 25 million float64 entries, about 200 MB before workspace. Scoring a new row evaluates the kernel against every training point, so serving cost grows with the training set.

# scikit-learn 1.5, illustrative
from sklearn.decomposition import KernelPCA, PCA
from sklearn.datasets import make_circles
X, y = make_circles(n_samples=2000, factor=0.3, noise=0.05, random_state=0)
Z_lin = PCA(n_components=1).fit_transform(X)
Z_rbf = KernelPCA(n_components=2, kernel='rbf', gamma=2.0).fit_transform(X)

My decision: check the classifier on both. If a simple classifier on linear PCA features is no better than chance, and kernel PCA with a bandwidth chosen on a validation set clearly beats it at the same k, use kernel PCA for this data at this size. If the data is large, or the structure is not clearly curved, keep PCA or go straight to a classifier on the raw features, since a nonlinear model may do the separation without a separate reduction step. Autoencoders are justified when data is plentiful and a fast learned encoder for new rows is needed, but that is a later step, not the first one.

Curated: · Written: · Reviewed:

Q&A-21

One extreme point on a 20-feature sensor dataset seems to be turning PC1 toward itself. How would you confirm that one point is responsible, and what would you use instead of the sample covariance if it is?

Answer

The 60-second answer. PCA's eigenvectors come from the sample covariance, which is a sum of squared deviations. A point far from the bulk adds its squared distance to every entry it touches, so it can dominate the sum and pull the leading eigenvector toward its own direction. Classical PCA only sees variance, so it does not tell me whether that point is an error. I would first confirm the influence by refitting without the point and comparing the two PC1 directions. If one point is responsible, I would report a robust covariance fit alongside the classical one, and the trade-off is compute and an assumption that the clean majority is roughly elliptical.

Worked example (hypothetical figures). Take 100 clean, centred points with variance 1.0 along x, variance 0.01 along y, and no correlation. Add one point at (0, 20). The mean moves to y = 20/101 ≈ 0.198. The outlier's y-deviation from that mean is about 19.80, so its squared deviation is about 392. The clean points add about 1 from their own spread plus 100 × 0.198² ≈ 3.9 from the mean shift. Dividing by n−1 = 100 gives Sxx ≈ 1.0, Syy ≈ (392 + 1 + 3.9)/100 ≈ 4.0, and Sxy ≈ 0, since the outlier's x-deviation is zero. The eigenvalues are about 4.0 and 1.0, so PC1 flips from (1, 0) to (0, 1). The outlier supplies almost all of the variance along the new PC1.

Confirming the cause. Refit PCA n times, leaving out one point each time, and record the angle between the full-data PC1 and each refit. Here dropping the outlier rotates PC1 by 90 degrees, while dropping any clean point moves it by a few degrees at most. This costs n SVDs of the centred matrix, which is cheap at n = 300 and d = 20. A single point standing alone at the end of a score plot is the visual version of the same signal, and it is a prompt to run the refit, not proof by itself.

Robust alternatives, and the trade-off.

  • Minimum Covariance Determinant (MCD). Choose the h points whose sample covariance has the smallest determinant, and use their mean and covariance. scikit-learn's MinCovDet defaults h to about (n+d+1)/2, which gives a breakdown point near 50%. The exact search is combinatorial, so FastMCD uses random starts and concentration steps (Rousseeuw and Van Driessen, 1999). It costs far more than the sample covariance, and at n = 300 with d = 20 it is workable, though its estimates get noisier as d grows toward n.
  • Huber-type M-estimators. Reweight each point by a function of its distance, so distant points count less than they would in the sum of squares. Each pass is a weighted covariance, O(n·d²), which is much cheaper than MCD. Their breakdown point is lower, and a few gross outliers can still pull the estimate.
  • Choosing between them: if contamination is heavy and n is comfortably above d, MCD is the safer choice despite its cost. If the outliers are few and the cost matters, a Huber-type fit is usually enough.

Assumptions. The clean majority is roughly elliptical, and the flagged points are errors rather than signal. If the heavy tails are real, a robust fit drops the structure the business cares about, so I would report the classical and robust fits side by side and settle with the data owner whether the flagged rows are errors. The estimator cannot make that call.

Curated: · Written: · Reviewed:

Q&A-22

Why is PCA a poor fit for a table of one-hot encoded categorical columns, and what method does the job better?

Answer

The 60-second answer. PCA finds directions of maximum variance and assumes the numbers mean something as magnitudes: a difference of 1 is the same kind of difference everywhere. One-hot columns are 0/1 indicators with no such meaning. Two rows that differ in one level are exactly as far apart as any other two different levels, so Euclidean distance says nothing about how similar two categories are. The variance of each indicator also just tracks how common its level is.

The structure makes this worse. A variable with three levels becomes three columns, and every row has exactly one 1, so the columns always sum to 1. That constraint makes the covariance matrix singular: its rows sum to zero, and one eigenvalue is exactly 0 for the constant-sum direction. PCA spends a component on an artefact of the encoding, and the rest reflect marginal frequencies rather than associations you can name.

For purely categorical data, multiple correspondence analysis (MCA) is the method built for this. It decomposes the indicator matrix using chi-square distances, so each row profile is compared relative to the marginal frequencies it sits within. For mixed numeric and categorical data, factor analysis of mixed data (FAMD) standardises the numeric columns and rescales the indicators so both groups contribute comparably before decomposing. Use MCA or FAMD when you want components that describe the table. For a predictive model, regularised regression or a tree handles sparse indicators directly and avoids compressing them at all.

Curated: · Written: · Reviewed:

Q&A-23

Your 30 standardised features are whitened with a covariance estimated on 200 training rows, then a linear model is trained on the output. When does whitening help, and what happens to the smallest-eigenvalue directions on a test batch whose spread along them has drifted?

Answer

The 90-second answer. Whitening rescales PCA scores so every component has unit variance and zero correlation with the others. With C = V Λ Vᵀ and centred x, the whitened vector is z = Λ^(-1/2) Vᵀ (x − μ), so Cov(z) = I. It is an invertible linear map, so it cannot add information. It changes the geometry the downstream method sees, and that is the whole decision. It helps when the method assumes isotropic or decorrelated inputs, such as gradient descent with a shared step size, Euclidean distances, or ICA's sphering step. It hurts when the small eigenvalues are mostly noise, because dividing by √λ multiplies those directions by 1/√λ. I would whiten only after truncating to the components I trust, or regularise the scale to 1/√(λ + ε).

Worked example (hypothetical numbers). Training covariance has eigenvalues 4 and 0.01. Whitening gives z₁ = x₁/2 and z₂ = 10·x₂ in the rotated frame. Training variance of each whitened component is 1, and the condition number drops from 400 to 1. Now a test batch arrives where spread along the second direction has grown to variance 0.04 because of sensor drift. Its whitened variance is 0.04 × 100 = 4, four times the signal direction's variance. The model sees the noise as the loudest feature. Without whitening, the same drift adds 0.04 to a variance of 0.01 and is visible, but it does not dominate the input.

What whitening does to models. For unpenalised least squares with an intercept, predictions are invariant to any invertible linear transform of the features, so whitening changes nothing in the fitted function. It changes conditioning, which matters for gradient-based solvers: iterations scale roughly with the condition number. With a ridge penalty, whitening changes which directions the penalty shrinks, so the penalty must be retuned on the whitened scale. Whitening also amplifies rounding error: an eigenvalue of 1e-12 from floating-point noise becomes a scale of 1e6.

Two forms. PCA whitening uses W = Λ^(-1/2) Vᵀ, which rotates into component coordinates. ZCA whitening uses W = V Λ^(-1/2) Vᵀ, which stays closest to the original axes, so each whitened feature stays interpretable. I would choose ZCA when feature identity matters downstream and PCA whitening when a rotation is harmless.

Failure modes and detection. (1) Test-time drift in low-variance directions dominates the input. Detect by comparing per-component variance on the test batch with the training value of 1: a ratio far above 1 flags amplification. (2) With n = 40 and d = 5,000, the covariance has rank at most 39, so the remaining eigenvalues are zero and whitening divides by zero. Truncate to the nonzero components first. (3) Fitting the whitening on the full dataset leaks test variance into the scale. Fit it inside the training fold only.

Curated: · Written: · Reviewed:

Q&A-24

You have 300 features and a binary target. Why might PCA keep a feature that does not predict the target, and how does feature selection differ?

Answer

The 60-second answer. PCA ranks directions by how much variance they carry in the inputs. It never sees the target, so a high-variance input can take a top component even when it says nothing about the label, and a low-variance input that predicts the label well can be dropped. Feature selection keeps some original columns, and supervised selection uses the target to decide which ones. PCA gives a compact representation that is target-blind; selection gives fewer original columns chosen for the task.

Mechanism. PCA takes the eigenvectors of the covariance of centred features. Component j is z_j = w_j^T x, where w_j maximises the variance of z_j with no reference to y. Selection keeps a subset S of columns. The two orderings can disagree.

A hypothetical case shows the gap. Let x1 and x2 be uncorrelated, with Var(x1) = 100 and Var(x2) = 1, and let y = x2 + noise with noise variance 1. PC1 lies almost on the x1 axis. Keeping one component keeps x1, which explains little of y. Selecting x2 gives R-squared near 0.5 on this setup. Variance rank and predictive rank are different orderings, and PCA only sees the first.

# scikit-learn 1.5, hypothetical data
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif
pca = PCA(n_components=k).fit(X_scaled)        # no y argument
sel = SelectKBest(f_classif, k=k).fit(X_scaled, y)  # uses y

Coefficients on PCA components are not effects of the original inputs in their own units, because each component mixes all inputs. Selected columns keep their meaning.

Failure check. Selection done on the full data before cross-validation leaks the target into the choice, so the selection step belongs inside each fold. Before committing to PCA, fit a model on the discarded components and see whether it predicts y.

Curated: · Written: · Reviewed:

Q&A-25

Your colleague wants to feed 2-D t-SNE coordinates of 5,000 customers into a churn model. Explain what the t-SNE axes mean compared with PCA scores, and what you would feed the model instead.

Answer

The 90-second answer. A PCA score is a linear projection: component j is a unit vector of weights on the original features, and each row's score is its dot product with that vector. Refit on the same data and you get the same weights up to sign. A t-SNE coordinate has no such meaning. It is the output of a layout optimisation that matches neighbour probabilities, and rotating or reflecting the whole embedding leaves that objective unchanged. So the axes are arbitrary, and a churn model fitted on them learns a mapping to a layout that a different seed would draw differently. I would feed the model PCA scores at a k chosen on cumulative variance and checked on validation churn, or the raw features.

Mechanism. t-SNE converts input distances into neighbour probabilities with a Gaussian kernel whose bandwidth is set by perplexity (default 30 in scikit-learn), converts 2-D distances into Student-t probabilities, and minimises the KL divergence between them by gradient descent. Only near-neighbour relations are fitted, so cluster sizes, gaps and densities are not preserved. scikit-learn's TSNE has no transform method, so new rows cannot be embedded without refitting.

# Python 3.11, scikit-learn 1.5
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
Z_pca = PCA(n_components=10).fit(X_train)        # transform() works on new rows
Z_tsne = TSNE(n_components=2, random_state=0).fit_transform(X_train)  # no transform()
PCA scorest-SNE coordinates
Axis meaningweighted combinations of inputsnone
New rowstransformrefit required
Repeat rundeterministic up to signseed-dependent

Failure modes and detection. (1) The model learns coefficients on a layout that a new seed rotates or reflects. Detect: refit with three seeds and compare cluster membership. (2) Fitting the embedding on train plus test leaks test structure into the features. Detect: validation score moves when the embedding is fitted on train only. (3) Gaps in the plot are read as distances. Detect: compare centroid distances with input-space distances on a sample.

What I would ship. PCA scores at the k that holds about 90% cumulative variance, validated against churn, or the raw features. t-SNE stays a visualisation, reported with its seed and perplexity.

Curated: · Written: · Reviewed: