Skip to content
Tech Interview Prep
Technical interview guideNew

Principal Component Analysis (PCA)

Linear dimensionality reduction that rotates features onto orthogonal directions of maximum variance, so a few components keep most of the information. Interviews probe the covariance and SVD derivation, scaling and leakage discipline, and what PCA cannot do.

Read
14 min
Practice MCQs
25
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: Mathematics is version-independent. Code examples target scikit-learn 1.5 or later (sklearn.decomposition.PCA with svd_solver auto, full, arpack, randomized and covariance_eigen; whiten and n_components as documented). Default behaviour: centres features, does not scale them, and fits on the data passed to fit..

Overview

Curated: · Written: · Reviewed:

Principal Component Analysis (PCA)

PCA is the first dimensionality reduction technique most candidates reach for, and the one interviewers use to check whether you can derive the result rather than call a library. Questions cluster on three areas: the covariance and SVD derivation, scaling and leakage discipline, and what PCA cannot do. This guide builds the mechanism first, then the decisions that follow from it.

The one mental model to carry: PCA rotates the coordinate system so the new axes point along the directions the data varies most, orders them by how much variance each carries, and keeps the first k. Everything else (eigendecomposition, explained variance, reconstruction error, the leakage rule) follows from that.

Why reduce dimensions, and what PCA actually buys

A d-dimensional dataset projected onto a k-dimensional subspace (k < d) costs less to store and train on, and often generalises better, because the discarded directions are frequently noise. The practical reasons are speed, noise removal, and visualisation at 2D or 3D. The curse of dimensionality is the underlying problem: as d grows, data gets sparse, distances lose contrast, and models overfit more easily.

PCA's contribution is specific. It finds orthogonal directions that capture the most variance, in order, under a linear map. It is unsupervised, so it ignores labels. It does not find clusters and does not preserve nonlinear neighbourhoods. Say this early, because most wrong answers treat PCA as a general structure finder.

Centring and standardising before PCA

PCA is defined on centred data. Subtract the column mean from each feature so every column has zero mean. Without centring, the first component tends to point toward the mean instead of along the spread, and the covariance formula no longer measures variance around the mean.

Standardising (dividing each column by its standard deviation) is a separate decision. PCA maximises variance, and variance is measured in each feature's units. A feature recorded in grams dominates one recorded in kilograms purely through its scale. Standardise when features have different units or ranges. Skip it when the units are comparable and the magnitudes carry meaning you want PCA to respect, such as pixel intensities.

Worked check: height in metres (variance 0.04) and weight in kilograms (variance 400). Unscaled, PC1 is almost exactly the weight axis and height is effectively discarded. After standardising both to unit variance, the two features enter the covariance on equal footing, and PC1 reflects their actual correlation.

Covariance matrix, eigenvectors and eigenvalues

For centred data X (n samples by d features), the sample covariance is C = XᵀX / (n − 1). It is d by d, symmetric and positive semi-definite. The diagonal holds per-feature variances and the off-diagonals hold pairwise covariances.

The principal components are the eigenvectors of C. For a unit eigenvector v with eigenvalue λ, the sample variance of the data projected onto v is vᵀCv = λ. Maximising vᵀCv subject to ‖v‖ = 1 is a Rayleigh quotient problem, and its maximiser is the eigenvector with the largest eigenvalue. The second component is the eigenvector with the second largest eigenvalue. It is orthogonal to the first because C is symmetric, and eigenvectors of a symmetric matrix with distinct eigenvalues are orthogonal.

Trace it on C = [[3, 1], [1, 3]]. The eigenvalues are 4 and 2, with eigenvectors (1, 1)/√2 and (1, −1)/√2. PC1 carries 4/6 ≈ 67% of the variance. The total variance is the trace, 3 + 3 = 6, which equals the sum of eigenvalues. Interviewers often ask for that identity.

The recipe: centre, form the covariance, eigendecompose, sort eigenvalues in descending order, keep the top k eigenvectors as the columns of W, then compute Z = XW.

The same result through the singular value decomposition

Libraries rarely form XᵀX. They take the SVD of the centred data, X = UΣVᵀ. The columns of V are the principal components, and the singular values relate to the eigenvalues by λᵢ = σᵢ² / (n − 1). The projection is Z = XV_k = U_kΣ_k.

The SVD route is preferred for numerical stability. Forming XᵀX squares the condition number: a matrix with condition number 10⁸ gives XᵀX a condition number near 10¹⁶, close to the limit of double precision, so small eigenvalues lose accuracy. Working on X avoids the squaring.

On cost, the two routes are similar when n ≫ d. Forming the covariance and computing a full SVD both scale with n·d², so the gap is a constant factor, and stability is the stronger argument. When d is large and k is small, truncated or randomised SVD computes only the top k components and is much cheaper than a full decomposition. That is the answer to questions about scale.

Explained variance, scree plots and choosing k

The explained variance ratio for component i is λᵢ / Σⱼλⱼ. The cumulative sum gives the share of total variance kept by the first k components. A common rule keeps enough components to reach 90 to 95% cumulative explained variance. That threshold is a convention, not a theorem, and you should say so.

The scree plot graphs eigenvalues against component index and looks for an elbow. It is a heuristic, and real data often has no clear elbow. Where you can, choose k from a downstream metric, such as validation score of the model that consumes the components. Variance measures how much of the data's spread is kept, not how useful the components are for your task.

Projection, reconstruction and reconstruction error

For a centred point x and the top k eigenvectors W_k, the code is z = W_kᵀx and the reconstruction is x̂ = W_k z + μ, where μ is the mean that was subtracted. The squared error for one point is the squared norm of the discarded part, ‖x − x̂‖² = Σ_{i>k} (vᵢᵀx)².

Summed over the n training points, each discarded component contributes Σ_s (vᵢᵀx_s)² = (n − 1)λᵢ when C = XᵀX / (n − 1). So the total squared reconstruction error on the training set is (n − 1) Σ_{i>k} λᵢ, and the mean per sample is ((n − 1) / n) Σ_{i>k} λᵢ. The sum of discarded eigenvalues equals the mean per-sample error exactly only when the covariance uses divisor n. With the usual n − 1, the factor (n − 1)/n is the correction, and for large n it is close to 1.

This is the optimality property: among all rank-k linear projections, PCA minimises total squared reconstruction error. Interviewers often ask why PCA is the best linear compression, and the answer is this identity.

Reading loadings and why components resist interpretation

Each component is a linear combination of the original features. The coefficients of the eigenvector are sometimes called loadings. Other texts use loadings for the correlations between each feature and each component, which for standardised data equal the eigenvector coefficient times √λ. This guide uses loadings to mean the eigenvector coefficients. Check which definition a source uses before comparing numbers. Loadings from a different scaling or a different library cannot be compared directly.

Components resist interpretation for three reasons. They are mixtures of features rather than measured quantities. Their signs are arbitrary, since v and −v span the same axis. And any rotation of the top k components within their subspace preserves the variance captured, so a single axis is one choice among many. Varimax rotation, common in psychometrics, produces sparser loadings, but it is a separate step applied after PCA.

PCA in a pipeline: fitting on training data only

Fit the scaler and the PCA on training data only, then apply the fitted transforms to validation and test data. If you fit PCA on the full dataset, the components encode directions chosen partly from test points, and the validation score is optimistically biased. The model does not crash. It simply reports a number that is too good.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression

pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95), LogisticRegression())
pipe.fit(X_train, y_train)   # scaler and PCA see training rows only
pipe.score(X_test, y_test)   # fitted transforms reused, not refitted

In scikit-learn, n_components=0.95 keeps the smallest number of components whose explained variance ratio reaches 0.95, which is the threshold rule above in a convenient form. Inside cross-validation, put the whole pipeline in the folds so the scaler and PCA refit per fold. Fitting them once before splitting is the classic error.

Limits: nonlinear structure, outliers, categorical data

PCA is linear. If the data lies on a curved manifold, such as a Swiss roll, the top linear components can flatten it and remove the structure that matters. Kernel PCA and autoencoders handle curvature, at the cost of extra tuning and loss of PCA's closed form and optimality guarantee.

PCA is driven by variance, and variance is inflated by outliers. One extreme point can pull PC1 toward itself. Robust covariance estimators or outlier removal before fitting are the usual responses.

PCA assumes numeric, roughly continuous inputs. Categorical features need encoding first, and one-hot columns form sparse, correlated blocks that PCA represents poorly. Multiple correspondence analysis is the standard alternative for categorical data.

Whitening and what it changes

Whitening divides each projected component by √λᵢ, so the output covariance becomes the identity. This removes the variance ordering: the first component no longer dominates in magnitude. Some algorithms assume decorrelated, unit-scale inputs and benefit from this. The cost is that low-variance components, which are often noise, get amplified to unit scale. Whitening trades the ordering of signal against decorrelation, so it can make a downstream model worse when the small components are noise.

PCA versus feature selection, t-SNE and UMAP

Feature selection keeps a subset of the original columns and so keeps their meaning. PCA builds new combined features and loses direct interpretability. Choose selection when the model must explain its predictions in original terms.

t-SNE and UMAP are nonlinear methods built mainly for visualisation. They preserve local neighbourhoods and often show clusters that a linear projection blurs. Their distances and cluster sizes are not comparable in the way PCA's are, and they are unreliable as inputs to a downstream model. Use PCA for a reusable linear transform with a known reconstruction error, and t-SNE or UMAP to look at the data.

Interview checklist: follow-ups and weak answers

Expect to be asked why the top eigenvector maximises variance, why components are orthogonal, and why SVD is preferred. Be ready to write vᵀCv, state the Rayleigh quotient, and explain that the total variance equals the sum of eigenvalues.

Expect the leakage probe: "you fit the scaler on the full dataset, what goes wrong?" The answer is an optimistic validation estimate, not an error message.

Expect the limits probe: "when would PCA fail?" Name nonlinearity, outliers and categorical data, and say what you would use instead.

Expect the reconstruction probe: "what is the reconstruction error?" Give the discarded-eigenvalue identity with the (n − 1) factor stated, rather than a bare sum of eigenvalues.

A weak answer says PCA "finds the important features" or "removes noise" without saying it is linear and variance-based. Another forgets to standardise mixed-unit features, or calls the components interpretable features. A third fits PCA before the train-test split and does not notice.

A strong answer states the assumptions (linear, variance-based, centred data), gives the covariance-eigenvector link, names the SVD route and why, and states the leakage rule unprompted.

Feedback on this overview