Skip to content
Tech Interview Prep
Technical interview guideNew

UMAP and Nonlinear Embeddings

How UMAP builds a fuzzy k-nearest-neighbour graph and lays it out in low dimensions, how its parameters change the picture, and how to read the result without over-interpreting it.

Read
14 min
Practice MCQs
25
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: umap-learn 0.5.x (Python package, current 0.5 line) and the algorithm as published by McInnes, Healy and Melville (arXiv 1802.03426, 2018). Parameter defaults (n_neighbors=15, min_dist=0.1, n_components=2, metric='euclidean') are those of umap-learn 0.5.x; verify against the release in use before quoting them..

Interview Q&A

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

Q&A-1

Your team needs a 2D picture of 1,500 points that lie on a curled sheet, like a Swiss roll. Why might PCA mislead you here, when would you reach for UMAP instead, and what must you not read off the UMAP plot?

Answer

The 90-second answer. PCA is the right first tool when the structure you care about is linear and the variance is where the signal is. It finds the orthogonal directions that maximise retained variance, so a 2D PCA plot is the best flat picture of the data's dominant linear structure. A Swiss roll breaks that assumption. The sheet is two-dimensional but curled inside three dimensions, so two points on opposite layers of the roll can sit close in Euclidean distance while being far apart along the sheet. A linear projection onto the best 2D plane drops one direction and stacks those layers on top of each other. The regions are distinct on the manifold and collapsed in the plot.

UMAP starts from the opposite assumption: the data lies on a manifold, and distances only mean something locally. It takes each point's k nearest neighbours (default n_neighbors=15), converts their distances into fuzzy membership strengths using a per-point bandwidth, and builds a weighted graph of the local neighbourhoods. It then optimises a 2D layout whose graph has similar edge strengths, pulling neighbours together and pushing non-neighbours apart with a cross-entropy objective. Because neighbourhoods are local, the two layers of the roll stay apart whenever no neighbour edge links them.

# umap-learn 0.5.1, scikit-learn 1.x, Python 3.11
from sklearn.datasets import make_swiss_roll
from sklearn.decomposition import PCA
import umap
X, t = make_swiss_roll(n_samples=1500, noise=0.0, random_state=0)
Z_pca = PCA(n_components=2).fit_transform(X)
Z_umap = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0).fit_transform(X)

Plot t against each embedding. The PCA output folds the roll into an overlapping band. The UMAP output tends to unroll it, though this depends on the neighbourhood size.

What each method preserves. PCA preserves variance under a linear map, so global distances in the projection are faithful to the extent the top components carry the data. UMAP preserves local neighbourhood structure. It does not preserve global distances or densities, so inter-cluster gaps and cluster sizes in the plot are not quantities you can read. Two clusters that look far apart may be closer in the original space than the picture suggests, and a dense-looking island may be a sampling artefact.

Failure modes. The first is over-reading the layout. Treating UMAP coordinates as metric is the most common error. The second is parameter sensitivity: n_neighbors controls the balance between local and global structure, and min_dist controls how tightly points pack. The third is nondeterminism. UMAP is stochastic, so a fixed random_state is needed for reproducibility, and in umap-learn fixing it also disables parallel execution. Detect the first by a sanity check: compare nearest-neighbour rank in the input and the embedding, or run at two n_neighbors values and see what changes.

When not to use it. If the question is about variance, a downstream linear model, or a compressed input for another learner, PCA is the right answer. UMAP is for visual exploration and for clustering that depends on local density.

Curated: · Written: · Reviewed:

Q&A-2

Is UMAP a linear dimensionality reduction method like PCA?

Answer

The 60-second answer. No. PCA is linear: each component is a fixed linear combination of the input features, so the whole transform is a single matrix you can write down and apply to any new point with a dot product after centring. UMAP is nonlinear. It builds a weighted k-nearest-neighbour graph in the input space and then optimises a low-dimensional layout to match it, so no single matrix maps an input point to its embedded position. The consequence a candidate should state next is the one that follows from linearity: PCA's transform is explicit and reusable on new points, while UMAP's transform for new points places them by neighbour search and optimisation, so it approximates a projection rather than applying a fixed map.

# umap-learn 0.5.1, scikit-learn 1.x
from sklearn.decomposition import PCA
import umap

pca = PCA(n_components=2, random_state=0).fit(X_train)
Z_new = pca.transform(X_new)   # X_new centred, then multiplied by components_.T

reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0).fit(X_train)
Z_new_umap = reducer.transform(X_new)  # neighbour placement, not a matrix product

Failure mode. Reading UMAP as a linear projection and treating its plot as a faithful picture of distances. Two clusters that look far apart on a UMAP plot may be close in the input space, because the layout preserves local neighbourhoods and not global distances. Detect it by checking a few cluster-to-cluster distances in the input space against the plot, or by comparing against PCA on the same data.

When the distinction matters. If the output feeds a downstream model that needs a stable linear transform, or you need loadings you can interpret, linearity is the deciding property and PCA is the right choice. UMAP suits visualising local cluster structure.

Curated: · Written: · Reviewed:

Q&A-3

A 1,500-point UMAP run has one sample duplicated 40 times with n_neighbors=15. Walk me through what happens to that group's edge weights, what the 2D plot shows, and how you would confirm it.

Answer

Opening (about 90 seconds). The 40 copies are identical rows, so each copy has 39 other copies at distance 0. With n_neighbors=15, every copy's neighbour list is filled by zero-distance entries, and the fuzzy graph gives every edge inside the block a weight of 1.0. Those edges are maximally strong, so the 40 copies are pulled into one spot. The 2D plot shows one dense point where a reader expects 40 samples. I'd confirm this from the graph UMAP actually optimises, not from the docs.

Neighbour lists. Query the raw data before fitting:

from sklearn.neighbors import NearestNeighbors
import numpy as np
rows = np.where((X == X[dup_idx]).all(axis=1))[0]   # 40 indices
d, idx = NearestNeighbors(n_neighbors=16).fit(X).kneighbors(X[rows[:1]])
print(d[0])  # all zeros
print(np.isin(idx[0][1:], rows).all())  # True if all 15 non-self neighbours are copies

Trace for one copy: 39 candidates sit at distance 0, the query returns 16 entries, self is one of them, so 15 zero-distance copies remain. The kNN list contains no point outside the block.

Edge weights. In umap-learn's smooth_knn_dist, rho_i is the first strictly positive distance. None exists here, so rho_i = 0. The target is log2(15) ≈ 3.907. The binary search over sigma cannot change the outcome because every term is exp(-max(d - rho, 0) / sigma) with d - rho = 0, so each term is 1.0 and the sum stays at 15 for any sigma. Each directed membership is 1.0. Symmetrising with a + b - ab gives 1.0 for internal edges. Outside points whose lists contain a copy add edges into the block, but the copies have no outgoing edges to outside points, so the block connects to the rest only through those inbound edges.

What the plot shows. Spectral initialisation gives identical rows identical coordinates, and the small noise term barely separates them. The 40 copies overlap at one location, so the plot shows one point. Its apparent density comes from overplotting, not from 40 distinct samples.

How I'd confirm it.

  1. Count exact duplicates with np.unique(X, axis=0, return_counts=True).
  2. Check the fitted graph: G = model.graph_.tocsr(), then read G[i].data for a copy and confirm the internal weights are 1.0.
  3. Compare the spread of block coordinates with the spread of distinct points in the embedding.
  4. Refit with two random_state values and check the block stays collapsed.

If NN-descent misses some ties, a copy's list may contain a non-zero neighbour. Step 1 of the trace would catch that, so run it on the actual index build you use.

Fix. Deduplicate, keep the multiplicity as a point weight or marker size, and report the distinct sample count. Jitter hides the tie and should be reported if used. Raising n_neighbours above 40 changes every point's neighbourhood and can merge real local structure.

Curated: · Written: · Reviewed:

Q&A-4

In UMAP, why does each point's connection radius start at its distance to its nearest neighbour, and what goes wrong if you drop that step?

Answer

UMAP starts each point's connection radius at its nearest-neighbour distance so that every point is guaranteed at least one edge at full strength, whatever its local density. Without that anchor, a point in a sparse region can end up with no strong edge at all, and the graph breaks apart before layout begins.

The mechanism: for point i, let rho_i be the distance to its nearest neighbour and sigma_i a per-point bandwidth. The membership strength of the edge from i to neighbour j is v_ij = exp(-max(0, d_ij - rho_i) / sigma_i). Subtracting rho_i means the nearest neighbour always gets exactly 1. sigma_i is found by binary search so that the strengths over the k neighbours sum to log2(k), which makes the local weights comparable across points with different densities.

Worked example (hypothetical distances): point A has k = 3 neighbours at distances 0.5, 1.0 and 2.0. Then rho_A = 0.5, and the discounted distances are 0, 0.5 and 1.5. The target sum is log2(3) = 1.585, so exp(-0.5/sigma) + exp(-1.5/sigma) must equal 0.585. Solving gives sigma of about 0.67, and the strengths are 1.00, 0.48 and 0.11.

# umap-learn 0.5.x, metric='euclidean' is the default
import umap
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1)
embedding = reducer.fit_transform(X)

The directed weights are then symmetrised with the fuzzy union v = a + b - a*b, so an edge kept from either end survives. The subtraction is what makes that union behave for outliers: an outlier's strongest edge is pinned at 1 rather than decaying with its absolute distance.

What goes wrong without the subtraction: keep the per-point search but use d_ij directly. For A, the nearest neighbour at 0.5 now gets exp(-0.5/sigma), which is well below 1 for any sigma that keeps the far neighbours meaningful. The binary search then inflates sigma to hit the target sum, which flattens the weights across all neighbours and blurs which edges matter. In a sparse region where every distance is large, the strengths can fall under the level the layout pulls on, so the point stays isolated.

The failure is checkable before layout. Count connected components of the fuzzy graph, and count points with no edge above a threshold such as 0.5. On data that should form one sheet, a nonzero count points to a fragmented or isolated start, and the plot will show stray dots at the margins or a dense group split in two.

Trade-offs: local connectivity fixes the starting graph per point but says nothing about global distances, so the gaps between clusters in the plot are not measurements. n_neighbors controls the locality. Lowering it fragments local structure, and raising it smears fine detail. A strong answer names which of those two effects it is trading, rather than just naming the parameter.

Curated: · Written: · Reviewed:

Q&A-5

Walk me through how UMAP sets the bandwidth for each point, and why that makes its neighbourhoods behave differently in dense and sparse regions.

Answer

The 90-second answer. UMAP does not use one global length scale. For each point i it finds its k nearest neighbours, where k is n_neighbors, and sets two numbers. The first, rho_i, is the distance to its nearest neighbour, so that point always gets membership strength 1 to that neighbour and every point stays connected to something. The second, sigma_i, is the bandwidth. It is chosen by binary search so that the fuzzy weights over the k neighbours sum to log2(k). That target is the entropy-like budget the method spends on each point's neighbourhood, and it is the same for every point. What differs is how far apart the neighbours are, and that is what sets sigma_i.

The membership strength of neighbour j of point i is w = exp(-max(0, d_ij - rho_i) / sigma_i). Solve sum over j of w_ij = log2(k) for sigma_i. Then symmetrise with the fuzzy union w = w_ij + w_ji - w_ij * w_ji, since the graph is not symmetric to begin with.

A worked example, with hypothetical figures: k = 4, so the target is log2(4) = 2. Suppose the four neighbours sit at distances 1.0, 1.5, 2.0 and 2.5 from point i, so rho_i = 1.0 and the shifted distances are 0, 0.5, 1.0 and 1.5. The nearest neighbour contributes exp(0) = 1 whatever sigma is. The other three must contribute 1 between them. At sigma = 0.5 they sum to about 0.55, at sigma = 1.0 to about 1.20, and at sigma of roughly 0.82 they total 1.0. So sigma_i is about 0.82.

# umap-learn 0.5.x, illustrative check of the target
import numpy as np
from scipy.optimize import brentq
d = np.array([0.0, 0.5, 1.0, 1.5])        # d_ij - rho_i, nearest is 0
target = np.log2(4)
f = lambda s: np.exp(-d / s).sum() - target
print(brentq(f, 1e-3, 10))                # ~0.82

Dense and sparse regions. In a dense cluster the k-th neighbour is close, so the shifted distances are small and sigma_i is small. In a sparse region the same budget of log2(k) has to be spread over neighbours that are far away, so sigma_i grows. The effective scale therefore follows local density, which is the point of the construction: each point gets a neighbourhood of comparable meaning, not comparable absolute distance. The rho shift adds a second effect. A point in a sparse patch whose nearest neighbour is far away has its whole neighbourhood pulled down by rho_i, so it is not isolated by the raw distance alone.

Trade-offs and failure modes. n_neighbors enters through the target. Raising k from 15 to 50 raises log2(k) from about 3.9 to about 5.6, so sigma_i grows and the graph links points across wider distances. That is why larger k favours global structure and smaller k favours local detail. Failure mode one: an outlier with a single distant neighbour gets a large rho and a weight of 1 to that neighbour, so it attaches to the graph even though it belongs nowhere. Detect it by checking per-point degree in the fuzzy graph. Failure mode two: a dataset with many exact duplicates makes rho_i zero for many points and makes the search for sigma degenerate. Detect it by counting duplicates before fitting.

Compare with t-SNE. It fixes perplexity, a related but different target on the same neighbourhood entropy idea, and it uses a Gaussian per point in the high-dimensional space. Both adapt scale per point, but UMAP's rho term and fuzzy union produce a graph that is then laid out, so the layout depends on the graph, not directly on the density estimate.

Curated: · Written: · Reviewed:

Q&A-6

In UMAP, what does the fuzzy union a + b - a*b give for an edge that is present in only one direction, and why is that the reason it is used?

Answer

The 60-second answer. UMAP's kNN graph is directed: point i can list j as a neighbour without j listing i. The layout needs an undirected weighted graph, so the two directed weights a (i to j) and b (j to i) are combined with the probabilistic t-conorm a + b - a*b. When only one direction exists, b = 0, so the union returns a unchanged. The edge keeps its full strength. Averaging would have halved it to a/2, which weakens exactly the links that sparse points rely on.

Worked example (hypothetical weights).

CaseabUnion a+b-abMean
One-directional0.800.80.4
Mutual0.80.50.90.65

For mutual edges the union is larger than both inputs, so the symmetrised graph is denser where both points agree.

Failure mode and detection. Because the union is generous, hub points that many others name as neighbours gain high total weight, and clusters can merge in the layout. Check by setting set_op_mix_ratio to 0 (product-like) and to 1 (union), and see whether the separation changes.

Curated: · Written: · Reviewed:

Q&A-7

Walk me through what UMAP's layout step minimises, and why it samples repulsive pairs rather than computing repulsion for every pair. What does that cost you in exactness?

Answer

The 90-second answer. UMAP builds a fuzzy graph in the input space, then places points in 2D or 3D so that a low-dimensional fuzzy graph matches it. The layout step minimises a sum of binary cross-entropies over every pair of points. Attraction acts on the pairs the input graph says are neighbours. Repulsion acts on non-neighbours, and it is sampled: each positive update is paired with a handful of random points pushed away. That makes an epoch cost roughly linear in the number of edges, not quadratic in n. The price is that the repulsion is an estimate, the embedding is not a true minimiser of the stated objective, and the gaps between clusters become run-dependent.

The objective. Each point i gets a bandwidth sigma_i, found by binary search so that the membership sum over its neighbours hits log2(k), and a distance rho_i to its nearest neighbour, so that v(j|i) = exp(-(d_ij - rho_i) / sigma_i). The edges are symmetrised with a fuzzy union, v = v_ij + v_ji - v_ij * v_ji. In the layout, the membership of two output points is w = 1 / (1 + a * d^(2b)), where a and b are fitted so that the curve sits close to 1 below min_dist and decays on the scale set by spread. With umap-learn defaults (min_dist = 0.1, spread = 1.0), b comes out near 0.9 and a near 1.9. The loss is

CE = sum over pairs of [ v * log(v / w) + (1 - v) * log((1 - v) / (1 - w)) ]

Split it into two parts. The attractive term, -v log w, pulls neighbours together. The repulsive term, -(1 - v) log(1 - w), pushes non-neighbours apart, and it is strong only when w is near 1, which means two output points are already close.

# umap-learn 0.5.x, Python 3.10
import umap
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, negative_sample_rate=5,
                    n_epochs=None, random_state=None)
emb = reducer.fit_transform(X)  # n <= 10000 gets 500 epochs, otherwise 200

Why sample. Computing repulsion for all pairs means about n^2 / 2 terms per epoch. Hypothetical figures: at n = 1,000,000 that is about 5 × 10^11 pair evaluations per epoch. With k = 15 and symmetrised edges, the attractive edge count is roughly 1.5 × 10^7, and with negative_sample_rate = 5 each sampled edge triggers six updates, about 9 × 10^7 per epoch. That is around 5,000 times less work. The approximation is reasonable because most far pairs have w near 0 and contribute almost nothing.

What it costs. The negative draws are uniform over the points. Most land far away and contribute nothing, while the few that land close and matter are rare. So repulsion is under-sampled in exactly the places that decide whether two non-neighbouring clusters separate. Three consequences follow. The output is not a stationary point of the stated CE, and it moves with the random seed. Inter-cluster gaps and cluster sizes are not reliable reads even when the local neighbourhoods look right. And setting random_state makes umap-learn run single-threaded, so reproducibility costs wall-clock time.

How you check it. Run two seeds and compare cluster membership and inter-cluster distances. Raise negative_sample_rate and see whether the gaps move. If they do, the plot's global layout is an artefact of sampling, and the claim has to stay at the level of local neighbourhoods.

Curated: · Written: · Reviewed:

Q&A-8

UMAP's n_neighbors is the main local-versus-global control. What does it change in the graph, what does the plot look like at very small and very large values, and which starting value would you use for 50,000 unlabelled points?

Answer

The 90-second answer. n_neighbors fixes how many neighbours each point's fuzzy graph edges are fitted against. It sets the scale at which UMAP treats points as locally connected. Small values make the layout faithful to tiny neighbourhoods and unable to say how those neighbourhoods relate. Large values make it faithful to the broad arrangement and blurry about fine boundaries. For 50,000 unlabelled points I would start at the default of 15, because it is a tested compromise, and state that the cluster count depends on this choice.

Mechanism. For each point i, UMAP takes the distance to its nearest neighbour as rho_i, then fits a bandwidth sigma_i so that the membership weights of its k nearest neighbours sum to log2(k). Each directed weight is exp(-(d - rho_i) / sigma_i). The graph is symmetrised with p + q - pq, and the 2D layout is optimised by stochastic gradient descent to match it. So k decides how many edges each point gets before the bandwidth is set.

Small k (2 to 5). Each point links to very few others. The graph breaks into many small connected components. The optimiser has no edges that say where one component sits relative to another, so the plot shows islands. Noise in a handful of points can create sub-clusters that disappear at k=15.

Large k (around 200 or more). Each neighbourhood reaches across several groups. Edges that would separate a small cluster from its neighbour are averaged away, so small clusters can merge into larger ones. The broad arrangement is more faithful, but fine structure is lost.

Applying it to 50,000 points. Edge count scales with n times k. At k=15 the graph has about 750,000 directed edges before symmetrisation, which is cheap. At k=200 it has about 10,000,000. The cost is not the main concern here. The main concern is that the cluster count is a function of k, so I would not report a single number. I would run the default, run one larger value such as 50, and report which groups appear in both. Groups that survive both runs are candidates for real structure. Groups that appear in only one are open questions.

Failure modes to name. Fragmentation at small k that looks like biology. Merging at large k that hides a real subgroup. Distances between clusters read as meaningful when they depend on k and min_dist.

Curated: · Written: · Reviewed:

Q&A-9

Two UMAP runs on the same 5,000 cells differ only in min_dist, 0.0 and 0.8. What changes in the plot, what stays fixed in the neighbour graph, and how would you check that?

Answer

The 90-second answer. min_dist does not change which points are neighbours or how strongly they are connected. Those are fixed before optimisation begins by n_neighbors and the per-point bandwidth. min_dist only changes the target shape of the low-dimensional similarity curve, which sets how hard the optimiser pulls neighbours together and how far it lets them drift apart. A small min_dist lets points pack into tight clumps; a large one forces the layout to spread them, so local structure is visibly looser and the clumps dissolve. Read the plot's density as a setting of min_dist, not as evidence about the data.

Mechanism. The high-dimensional graph is built from distances to the n_neighbors nearest points. For each point i, rho_i is the distance to its nearest neighbour, and sigma_i is found by binary search so that the sum of exp(-(d_ij - rho_i) / sigma_i) over its neighbours equals log2(n_neighbors). Symmetrising with a fuzzy union gives weights v_ij in [0, 1]. Nothing in that path reads min_dist.

In the layout, UMAP uses a smooth curve Phi(d) = 1 / (1 + a d^(2b)) for the probability that two embedded points are connected. The constants a and b are fitted in find_ab_params so the curve is about 1 up to distance min_dist and then decays with scale spread (default 1.0). With the defaults min_dist=0.1 and spread=1.0, a is roughly 1.58 and b roughly 0.90. Raising min_dist moves the flat part of the curve outward, so the optimiser stops pulling neighbours together once they are within min_dist. Points that would sit in one tight cluster at 0.0 are held at a visible separation at 0.8.

# umap-learn 0.5.1, Python 3.10
import umap
tight = umap.UMAP(n_neighbors=15, min_dist=0.0, random_state=0).fit_transform(X)
loose = umap.UMAP(n_neighbors=15, min_dist=0.8, random_state=0).fit_transform(X)

Trade-offs. A small min_dist gives sharp, separated-looking clusters, and that is exactly what makes it dangerous to read cluster sizes or gaps as real. A large min_dist shows more of the broad topology but makes each neighbourhood look less dense. Neither setting changes which points were neighbours. If you need the graph to change, you change n_neighbors, the metric or the input representation.

Failure modes and how you catch them.

  1. Reading the clump as a population. Two clusters separated by a visible gap at min_dist=0.0 may just be one population that the optimiser pushed apart. Detect: rerun with a larger min_dist and check whether the gap survives; also check the graph's connected components, which min_dist cannot alter.
  2. Believing min_dist changed the neighbourhood. If the kNN overlap between runs differs, the cause is a different n_neighbors, metric or preprocessing. Detect: compare the kNN index arrays directly, before any embedding is drawn. They should match exactly when only min_dist changed.
  3. Treating a single seed as stable. Set random_state for the comparison, and run at least a few seeds, because the optimiser's result depends on initialisation.

Check. Hold n_neighbors, metric, input and random_state fixed. Confirm the kNN indices and fuzzy weights are identical across the two runs, then compare the embeddings. The only difference should be the density and spacing of the layout.

Curated: · Written: · Reviewed:

Q&A-10

In UMAP, what does n_components control, and what value is it set to by default?

Answer

The 60-second answer. n_components sets how many coordinates UMAP returns for each point. The default is 2, which is the dimension a scatter plot can show directly. Any other value gives a feature matrix with that many columns, which is usually used as input to clustering or a downstream model rather than drawn.

import umap
view = umap.UMAP(n_components=2).fit_transform(X)     # default: plot-ready
features = umap.UMAP(n_components=5).fit_transform(X)  # 5 columns, not plottable as a scatter

What the number does not mean. Each output column is an arbitrary coordinate of the layout. Unlike PCA, the columns are not ordered by variance, so the first column is not the most informative one. A 2D run and a 5D run are separate optimisations, so their coordinates are not directly comparable.

Failure mode. Plotting the first two columns of a 5D output and reading the distances between clusters as real. Those distances come from a different optimisation. To check, rerun with n_components=2 on the same points and compare cluster membership.

Curated: · Written: · Reviewed:

Q&A-11

Your team clusters 50,000 product embeddings with UMAP and the groups change when you switch the metric from euclidean to cosine. How do you explain the change, and which metric would you ship?

Answer

The 90-second answer. The metric is used in one place that matters: finding each point's k nearest neighbours. UMAP turns those distances into fuzzy edge weights, and the layout then optimises a low-dimensional embedding using Euclidean distances in the output space. The metric also feeds the per-point calibration of rho and sigma, but it never touches the layout directly. So a metric change alters which points are connected and how strongly, and nothing else. Two consequences follow. Points the metric calls neighbours get pulled together on the plot even when they are far apart in raw coordinates. And because the graph is local, a metric change can reshape clusters without changing anything you would see in a global view of the data.

# umap-learn 0.5.x; hypothetical 2D points
import numpy as np
a = np.array([1.0, 0.0])
b = np.array([10.0, 0.0])   # same direction as a, ten times longer
c = np.array([0.0, 1.0])    # orthogonal to a, short
euclid = lambda x, y: np.linalg.norm(x - y)
cos = lambda x, y: 1 - x @ y / (np.linalg.norm(x) * np.linalg.norm(y))
print(euclid(a, b), euclid(a, c))  # 9.0 1.414 -> nearest neighbour of a is c
print(cos(a, b), cos(a, c))        # 0.0 1.0   -> nearest neighbour of a is b

For unnormalised text embeddings the norm often tracks document length or term frequency. Euclidean distance then groups short documents with other short documents, and cosine groups them by direction, which is closer to topic. For expression profiles, a correlation metric computed across genes removes each sample's offset and scale, so two samples with the same pattern at different overall expression become neighbours. Euclidean on log-normalised values would separate them. Hamming suits binary features, but distances are integer counts, so many points tie and the k-set among tied points is arbitrary. Check the metric's behaviour on your data before reading a cluster as biology.

Failure modes and how you catch them. First, cosine on vectors that are already unit-normalised ranks neighbours exactly as Euclidean does, since ||a-b||^2 = 2 - 2cos(a,b). If the plots still differ, the difference comes from something else, such as the random seed or a changed min_dist, so fix random_state and compare. Second, the metric absorbs a nuisance variable: cluster membership tracks library size or vector norm. Detect it by correlating cluster labels with that nuisance quantity. Third, a metric that reshuffles neighbours inside a cluster while keeping it intact is harmless; one that splits a cluster is not. Measure the per-point Jaccard overlap of kNN sets between metrics and inspect the points whose sets change most.

I would choose the metric by the invariance the question needs: cosine if scale is noise, correlation if offset and scale are noise, Euclidean only when magnitudes are the signal. Then I would report the kNN overlap alongside the plot.

Curated: · Written: · Reviewed:

Q&A-12

Your team plots the same single-cell dataset with t-SNE and with UMAP, and the two pictures arrange the clusters differently. What does each method preserve locally and globally in practice, and what can you claim about the distance between two clusters in either plot?

Answer

The 90-second answer. Both methods build a high-dimensional neighbour weighting, then optimise a 2D layout to match it. Both are good at local structure: a point's close neighbours usually stay close. Neither guarantees global geometry. The distance between two clusters, their relative sizes and their densities are not reliable readings, and two runs with different seeds can rearrange the clusters. UMAP often keeps the arrangement of clusters somewhat better than t-SNE, for two mechanical reasons: its loss has a repulsive term that acts between all sampled pairs, and its default initialisation is spectral, which starts from the graph's global eigenvectors. That is a tendency to test, not a property to quote.

Mechanism. t-SNE sets a Gaussian bandwidth per point so that its conditional distribution has a chosen perplexity, symmetrises the result, and uses a Student-t kernel with one degree of freedom in 2D. It minimises KL divergence between the high- and low-dimensional distributions. Because the KL term penalises places where close points are placed far apart much more than the reverse, the layout is good at keeping neighbours together, and the heavy tails push unrelated clusters apart only weakly. UMAP builds a k-nearest-neighbour graph with n_neighbors, converts each edge to a fuzzy membership using a per-point distance to the nearest neighbour and a bandwidth, and minimises a cross-entropy with attractive terms on graph edges and repulsive terms on sampled non-edges (negative_sample_rate defaults to 5).

t-SNEUMAP
Neighbourhood controlperplexityn_neighbors
LossKL divergencefuzzy cross-entropy
Repulsionall pairs, via normalised Qsampled non-edges
Default initPCA (scikit-learn 1.2 and later)spectral
Typical global behaviourclusters placed freelyarrangement often more stable
# umap-learn 0.5.x, scikit-learn 1.4
from sklearn.manifold import TSNE
import umap
emb_u = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0).fit_transform(X)
emb_t = TSNE(perplexity=30, init='pca', random_state=0).fit_transform(X)

Setting random_state in umap-learn disables parallel execution, so the UMAP run is slower than the same call without it.

What to check before you claim anything about the plot. Run each method with five seeds and compute the pairwise centroid distances between clusters. If the ranking of distances changes across seeds, the global layout is not a finding. Then vary the perplexity or n_neighbors and see which clusters stay adjacent. Clusters that stay neighbours across settings carry some signal; the exact gap between them does not.

Failure modes. A tight, well-separated cluster in t-SNE can look like it has a meaningful size, but its spread is set by perplexity and the kernel, not by the data. A UMAP run with very small n_neighbors can break one continuous population into several islands, which then appear to be far apart. Both failure modes are caught by the seed and parameter sweep above.

Curated: · Written: · Reviewed:

Q&A-13

Two UMAP runs on the same 3,000 cells put cluster B 4 units from cluster A in one run and 11 units in the other. Can you say the A-to-B gap is biologically meaningful, and what single change would make you trust a gap more?

Answer

The 60-second answer. No, not from these two plots. UMAP builds a fuzzy graph from each point's nearest neighbours and then lays it out by pulling connected points together and pushing sampled non-neighbours apart. The objective constrains local neighbourhoods well. It says nothing about how far apart two clusters should sit, so the 4-versus-11 difference is most likely layout variation rather than a change in the data. The single change I would trust most is the same input rerun with several seeds: if A-to-B stays the nearer pair under every seed, the ordering is worth reporting, and the magnitude is not.

Why the gap is not fixed. Two clusters are joined only by the few edges that cross between them, so when they are well separated the attractive force holding them together is weak. The optimiser runs n_epochs passes (200 by default for large datasets, 500 for small ones) with negative_sample_rate=5 sampled repulsions per attractive edge. Where the clusters settle depends on the spectral starting layout and on which non-neighbours get sampled, and both depend on the seed. Changing n_neighbors also changes the balance: a larger value adds cross-cluster edges and tends to draw clusters closer, while a smaller value keeps the graph local and lets clusters drift further apart.

The check. Keep the data, preprocessing and metric fixed. Run the same embedding across ten seeds and two n_neighbors values, then compare the gaps on a scale-free basis, such as the distance between centroids divided by the mean within-cluster radius:

# umap-learn 0.5.1, Python 3.11
import numpy as np, umap
def gap_ratio(X, y, a, b, seed, k):
    Z = umap.UMAP(n_neighbors=k, random_state=seed).fit_transform(X)
    A, B = Z[y==a], Z[y==b]
    radius = np.sqrt(((A - A.mean(0))**2).sum(1).mean())
    return np.linalg.norm(A.mean(0) - B.mean(0)) / radius

Run the same ratio on the input features, where the separation is real. Hypothetical output, labelled as such: the embedded ratio runs from 3 to 9 across seeds at n_neighbors=15 and from 2.9 to 3.6 at n_neighbors=50, while the input ratio is about 2. The wide range at small n_neighbors says the magnitude is unstable. If the ordering of A-to-B against A-to-C is the same in every run, that ordering can be reported with its range. A single fixed random_state gives reproducibility, not robustness, so it can hide exactly the variation you need to see.

Curated: · Written: · Reviewed:

Q&A-14

In a UMAP plot, one cluster holds 900 points and another holds 90, yet they take up similar areas. Does the bigger-looking cluster have more spread in the original space?

Answer

The 60-second answer. No. Area in a UMAP plot is not a measurement of how many points a cluster holds or how spread out it is in the input. UMAP does not preserve absolute density or absolute distance. For each point it rescales distances by the distance to that point's nearest neighbour, then builds a weighted neighbour graph from those scaled values. The 2D layout is fitted to that graph, not to the raw distances. So each point gets a comparable local scale wherever it sits. A dense group of 900 points and a sparse group of 90 can both settle at similar apparent sizes.

The rule to state: in the plot, the area a group covers and the gaps between groups carry no reliable information about point count or variance. Membership is what the layout is designed to preserve.

What this means for reading the plot.

  1. Point count does not set area. Ten times as many points in a region does not give ten times the footprint. Crowded and sparse groups are drawn at comparable local scales.
  2. Original variance does not carry over. A cluster that is wide in 300 dimensions can appear compact, and a tight one can appear stretched. Only neighbour structure is kept.
  3. Gaps are not distances. Empty space between groups means the graph had few edges across them. It does not give a measured separation in the input.

Failure modes and how you catch them.

  • Reading density as size. A sparse group can look larger than a dense group that has ten times the points. Detect it by counting points per group and comparing the counts with the plot before you report anything.
  • Reading area as variance. Compute the variance or spread of each group in the original space. If the ranking differs from the plot, the plot is not measuring spread.
  • Treating one run as final. Layout shapes depend on settings and random initialisation. Re-run with a different seed and check that group membership holds.

The guarantee is that local neighbourhoods are represented in 2D. The plot does not promise that area, spacing or variance match the input.

Curated: · Written: · Reviewed:

Q&A-15

You fix random_state=42 and rerun UMAP on the same matrix, and one cluster moves to the other side of the plot while another splits in two. What is random_state fixing, what is it not fixing, and how do you tell real structure from a seed artefact?

Answer

The 90-second answer. random_state makes one run reproducible on the same library version, platform and input. It does not make the embedding a property of the data. UMAP has three places where randomness enters: the approximate nearest-neighbour graph, the initialisation, and the stochastic layout optimisation. A fixed seed removes run-to-run noise for one configuration, but changing the seed still moves clusters, because the 2D arrangement is one of many near-equal optima of a non-convex objective. So I would not read cluster positions or the number of clusters from one seed. I would run a sweep of seeds, measure which groupings survive, and only claim structure that does.

# umap-learn 0.5.x, Python 3.11, X is (n, d) float32
import umap
from sklearn.metrics import adjusted_rand_score as ari
emb = {s: umap.UMAP(random_state=s, n_neighbors=15, min_dist=0.1).fit_transform(X) for s in range(10)}
# cluster labels per seed, e.g. from HDBSCAN on each embedding

Where the randomness sits.

  1. Approximate kNN. UMAP builds its graph with NN-descent, seeded from random-projection trees. Neighbours are approximate, so points near a boundary can gain or lose an edge between seeds. This changes the fuzzy graph slightly, most visibly for small or thinly connected groups.
  2. Initialisation. The default init='spectral' takes the leading eigenvectors of the graph Laplacian, then adds a small amount of noise. The eigensolver's start vector is random too. Spectral init fixes the global shape up to rotation and sign, but the noise means the starting layout still differs by seed. init='random' makes the starting point seed-dependent in a stronger way, which is a useful contrast when you want to see how much the result depends on the start.
  3. Negative-sampling SGD. The layout minimises a cross-entropy over edges: attractive terms for graph edges, repulsive terms for sampled non-edges. With negative_sample_rate=5 (the default), each attractive update is paired with five random repulsive pairs. Learning rate starts at 1.0 and decays linearly. In 0.5.x the default epoch count is 500 for datasets of 10,000 points or fewer and 200 above that. Sampling means two seeds follow different trajectories through the same objective.

Why clusters move. Relative placement of well-separated groups is weakly determined: once each group is internally cohesive, many arrangements give almost the same loss. A cluster that is linked to its neighbours only by weak fuzzy edges (membership strengths near zero after the union step) has little force holding it in place, so a different seed can put it on another side. Splits behave differently: a group held together by a few bridge edges can separate when one seed drops a bridge during the kNN approximation or the early epochs push it apart.

What I would check, in order.

  • Cluster agreement across seeds: adjusted Rand index between label sets from, say, ten seeds. A split that appears in 2 of 10 seeds is a warning, not a finding.
  • Graph stability: compare the kNN index sets across seeds and count how many edges differ. If the graph is stable, the randomness is in the layout, not the neighbour search.
  • Sensitivity to optimisation budget: raise n_epochs and see whether the split persists. A split that vanishes with more epochs was under-converged.
  • Exact kNN: rerun with a brute-force neighbour search on a subsample, to separate approximation error from layout randomness.
  • Pre-embedding evidence: check the split in the high-dimensional data with a density or silhouette measure before trusting it in 2D.

If the groups are stable across seeds and budgets, I report them. The distance between two clusters on the page, and the cluster sizes, I do not report as data. The embedding is evidence about neighbourhoods, not a metric on the plot.

Curated: · Written: · Reviewed:

Q&A-16

A notebook sets random_state=42 in umap-learn and the fit runs slower on 16 cores than it did unseeded. What causes the slowdown, and would you keep the seed in a pipeline that rebuilds a shipped embedding each week?

Answer

The 60-second answer. random_state seeds the random draws UMAP makes, mainly the negative samples used in the layout step and the random starts in the approximate nearest-neighbour search. With the same input, library version and environment, a fixed seed gives the same embedding on repeated runs. The cost is that setting random_state overrides n_jobs to 1. The parallel neighbour search and the parallel layout updates both stop, so the slowdown you see is the parallelism being switched off, not the seed doing extra work. I would keep the seed in analysis and in regression tests, and in a weekly pipeline I would drop it from the fit and store the output as an artefact with the seed, versions and parameters recorded.

# umap-learn 0.5.x (hypothetical timings, 16 cores)
umap.UMAP(n_jobs=-1).fit_transform(X)        # parallel, output varies between runs
umap.UMAP(random_state=42).fit_transform(X)  # repeatable, runs single-threaded

Why the parallel version cannot be repeated: the layout updates write to shared coordinates from several threads, and the order of those writes changes from run to run. Floating-point results depend on that order, so the seed alone cannot make the parallel run repeatable. Serialising the updates is the only route to identical output, and that is why the seed and the thread count are tied together.

What the seed guarantees: identical output for repeated runs with the same input, library version, numba build and CPU class. What it does not guarantee: identical output after an upgrade of numba, scikit-learn or NumPy, on a different architecture, or with rows in a different order. The first is documented behaviour. The last three are things I would test rather than assume.

The main failure mode is a single seeded run that looks stable. Run five seeds, compare cluster labels with the adjusted Rand index, and report the spread next to any figure. A low score tells you the picture depends on the seed, and a team that picks the seed that looks cleanest has made the figure less reliable, not more.

For the weekly rebuild, the choice is between a reproducible rerun that takes longer and a parallel run that is faster but cannot be regenerated bit for bit. Storing the artefact with its metadata keeps both the speed and the audit trail.

Curated: · Written: · Reviewed:

Q&A-17

Before you run UMAP on a table of eight clinical measurements in mixed units, would you apply StandardScaler to each column, and why?

Answer

The 90-second answer. UMAP builds its neighbour graph from distances computed on the input features, so each column's scale decides how much it contributes. With the default euclidean metric, a column measured in millimetres and a column holding a 0 to 1 ratio do not compete on equal terms. For this table I would standardise each column to zero mean and unit variance, provided no column has a clinical reason to dominate. The units are arbitrary, and a column's raw spread mostly reflects its unit rather than its information. I would state that assumption to the interviewer and say what would change it: if the clinicians ranked measurements by importance, I would weight them rather than standardise them.

Mechanism. Take two features: height in mm (values near 1,800) and a ratio (values near 0.5). Two patients differing by 10 mm and 0.4 in the ratio have euclidean distance sqrt(10^2 + 0.4^2) ≈ 10.01. The ratio contributes almost nothing, so the graph, and therefore the layout, is built almost entirely from height. Standardising both columns makes a 0.4 ratio difference count as much as a one-standard-deviation height difference. Neither result is more correct. The choice encodes a claim about which differences matter.

# Python 3.11, scikit-learn 1.5, umap-learn 0.5.x
from sklearn.preprocessing import StandardScaler
import umap

X = StandardScaler().fit_transform(clinical_raw)   # clinical_raw: 8 columns, mixed units
emb = umap.UMAP(n_neighbors=15, metric='euclidean').fit_transform(X)

Failure modes and how to catch them.

  1. One outlier-heavy column dominating after scaling. A single extreme value inflates a column's standard deviation and compresses its other values near zero. Detect: for that column, most scaled values sit in a narrow band, and the plot separates on the few extreme patients.
  2. Scaling a column that is really categorical. Standardising a 0/1 flag or an ordinal grade gives it an arbitrary weight. Detect: the layout splits along that column even though it carries little clinical signal.
  3. Treating scaling as a fix for correlated columns. Two near-duplicate measurements count twice in the distance. Detect: the correlation matrix shows near-1 pairs, and dropping one changes the clusters.

Curated: · Written: · Reviewed:

Q&A-18

Why is PCA often run before UMAP on a dataset with thousands of features?

Answer

PCA is run first to shrink the feature count before UMAP builds its neighbour graph. UMAP finds each point's nearest neighbours using distances in whatever space it is given, and those distances are computed across every feature. With 30,000 gene columns, each distance costs about 30,000 operations, so the neighbour search is roughly 600 times more expensive than it would be on 50 principal components. Most of those columns also carry little sample-to-sample signal, so they add noise to every distance and blur the difference between near and far neighbours.

So the usual pattern is: standardise, project onto the top principal components, then run UMAP on those scores. The component count is a choice made by a stated rule, such as a scree elbow or a cumulative variance target, and it is checked by confirming the plot's cluster structure holds when the count changes.

Trade-off: PCA is linear, so it can discard low-variance directions that carry a small real population. Run PCA first to save cost and reduce noise, but do not treat its output as the truth about the data.

Curated: · Written: · Reviewed:

Q&A-19

After you fit UMAP on 20,000 training cells, a new batch of 2,000 cells arrives. How does reducer.transform place them, what can it never change in the existing layout, and how would you tell when its placements are untrustworthy?

Answer

The 60-second answer. transform does not refit anything. It finds each new point's nearest training neighbours in the fitted index, turns those distances into membership strengths, places the new point at a weighted average of its neighbours' training coordinates, and then runs a short optimisation that moves only the new points. The training embedding stays fixed throughout. So a new point can only land where its training neighbours already are. Two consequences follow: the existing layout cannot change, and a point whose neighbours are poor or far away gets placed confidently in the wrong region.

# umap-learn 0.5.x, Python 3.11
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=42).fit(X_train)
Z_train = reducer.embedding_          # fixed from here on
Z_new = reducer.transform(X_new)      # only the new rows are optimised

Why it cannot move the layout. The fitted model stores the training data, the kNN index and the training embedding. The optimisation that places new points takes gradient steps on the query coordinates only, against the frozen training coordinates. A new point is an observer of the map, not a participant in it. If the new batch contains a rare subtype, the training clusters do not split or merge to make room for it.

Off-manifold points. Membership strengths are normalised per query point, so a point 40 units from anything in the training set still gets a full set of weights summing to one over its k neighbours. The placement is the same kind of answer as for a point sitting in the middle of a cluster. Distance to the training data is discarded at the step that matters most.

Detection. Check three things before trusting placements:

  1. Distance profile. Compare each new point's mean distance to its k training neighbours with the same statistic for held-out training points. A new batch with a shifted distribution is off-manifold.
  2. Round-trip. Transform a held-out slice of the training data and compare with its fitted embedding. The residual is an estimate of placement error. It is not exact, because transform re-optimises rather than copying coordinates.
  3. Stability under the seed. Refit with a second random_state. A placement that moves between training runs was never well supported.

When to refit. If the new batch is in-distribution, transform is fine for a dashboard. If it is off-manifold, or if you need the new points to shape cluster structure, fit on the union. Say so in the report, since the training map is no longer the object you analysed.

Trade-off. Transform is cheap and keeps old coordinates stable, which matters for dashboards and for comparing batches. The price is that the map encodes the training sample only, and the new points are a conditional guess given that map.

Curated: · Written: · Reviewed:

Q&A-20

You pass class labels as y to UMAP and the classes come out as clean, separate islands. Can the plot support a claim that the classes are separable in the features?

Answer

The 90-second answer. No. Supervised UMAP is given the labels before it builds the graph, so the islands partly reflect what it was told. Separation in the plot is therefore not independent evidence about the features. For a separability claim I would use a classifier cross-validated on the raw features, and I would use the supervised plot only to see how the labels shape the graph.

Mechanism. Unsupervised UMAP builds a fuzzy graph from k-nearest-neighbour distances, scaling each point's distances by its own local bandwidth, then symmetrises edge weights with the fuzzy union a + b - a*b. The layout then pulls linked points together and pushes sampled non-neighbours apart. Passing y to fit changes only graph construction. In umap-learn the default is target_metric='categorical' with target_weight=0.5, on a 0 to 1 scale. An edge whose endpoints have different labels is down-weighted, so the graph keeps fewer links across classes. Points labelled -1 are treated as unknown and carry no penalty. The layout code is unchanged.

# umap-learn 0.5.x, Python 3.10
import umap
emb_u = umap.UMAP(n_neighbors=15, random_state=0).fit_transform(X)
emb_s = umap.UMAP(n_neighbors=15, target_weight=0.5, random_state=0).fit_transform(X, y=labels)

What the penalty does. Cross-label links are cut, so same-class points stick together and classes pull apart in the plot even where X does not separate them. Two classes that overlap heavily in feature space can still come out as two islands. The gap between islands is not a measurement of anything in X.

When it helps. It is useful for seeing whether a label partition has local structure at all, and for visualising a learned metric when the labels are reliable and the question concerns the label partition itself.

When it misleads. The graph has been told the answer. Noisy labels get written into the geometry. Small classes can form clean islands from few points. A strong penalty can also fragment the graph by label even when the labels are random.

How I would check it. Rerun with shuffled labels at the same target_weight. If islands still appear, the penalty is producing them. Sweep target_weight from 0 to 1 and see where separation begins; separation that appears only at large weights is a property of the penalty. For the separability claim itself, train a classifier on X with held-out folds and report its accuracy with an interval, rather than relying on the plot.

Curated: · Written: · Reviewed:

Q&A-21

Your team must choose between UMAP and t-SNE for a 2D view of 100,000 cells in 50 PCA dimensions. Which one would you ship for a cluster-level overview, and what would you check before trusting the distances between clusters?

Answer

The first 90 seconds. For a cluster-level overview of 100,000 cells I would ship UMAP, with one caveat that I would state in the slide notes: the gaps between clusters are not measurements. Both methods build a neighbour graph in the input space and optimise 2D coordinates to match it. The trade-off that decides the choice is cost against how the method spends its effort. UMAP keeps about n_neighbors edges per point (default 15) and estimates repulsion from sampled pairs. t-SNE keeps about 3 × perplexity affinities per point (perplexity 30 gives roughly 90) and approximates a repulsive term over all points.

Arithmetic (hypothetical sizes, labelled as such). At 100,000 points, 15 neighbours give about 1.5 million edges. With perplexity 30, t-SNE keeps about 9 million affinities, roughly six times as many. Each UMAP epoch visits each edge once and draws negative_sample_rate=5 random pairs per positive sample. The default is 200 epochs at this size. The per-point cost is therefore lower for UMAP, which is why it usually finishes first. The exact seconds depend on hardware and thread count, so I would report the ratio from a timed run on our machine rather than a number from a paper.

# umap-learn 0.5.x, scikit-learn 1.4, Python 3.11
import umap
from sklearn.manifold import TSNE
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=0)
tsne = TSNE(n_components=2, method='barnes_hut', init='pca', perplexity=30, random_state=0)

Stability. Seeds change the stochastic optimiser in both methods, so cluster positions can move between runs. The starting layouts differ: scikit-learn's t-SNE defaults to init='pca' since 1.2, and UMAP's default is spectral initialisation. Neither gives a layout that is identical across seeds, so I would not present a single run as the picture of the data.

Global structure. Both methods preserve local neighbourhoods better than global distances. Spectral initialisation gives UMAP a coarse arrangement that depends on the graph, so the relative placement of two clusters can reflect the initialisation as much as the data. A check: rerun with three seeds and see whether the rank order of cluster-centroid distances holds. If it does not, the gap is not evidence.

Failure modes and checks. A UMAP run with too few epochs leaves clusters under-converged, so I would double n_epochs once and see whether the layout changes beyond seed noise. For t-SNE at this size, a perplexity that is too low fragments clusters, so I would compare the final KL divergence across perplexities before reading the picture.

What I would claim. That UMAP separates the main cell groups at lower cost, and that cluster membership is stable under the checks above. I would not claim that the distance between two clusters measures how different they are.

Curated: · Written: · Reviewed:

Q&A-22

In t-SNE, what does the perplexity setting control?

Answer

The 60-second answer. Perplexity is the effective number of neighbours that t-SNE's Gaussian kernel spreads each point's attention over. Formally, perplexity = 2^H(P_i), where H is the Shannon entropy in bits of the conditional distribution p(j|i) ∝ exp(−‖x_i − x_j‖² / 2σ_i²). Each point gets its own σ_i, found by binary search so that the entropy matches the chosen value. Dense regions get a small σ_i and sparse regions a large one. So perplexity 30 means each point's kernel behaves roughly like 30 equally weighted neighbours. It is a soft count, not a hard cut-off.

The default is 30 in scikit-learn's TSNE, and the value must be below the number of samples.

# scikit-learn 1.x
from sklearn.manifold import TSNE
TSNE(perplexity=30)  # default 30, Gaussian over all points, soft neighbour count

Why it matters for interpretation. Because the count is soft, a point's influence reaches beyond its nearest 30 points through the Gaussian tail. Changing perplexity changes how much of the layout reflects local structure. Do not read a single perplexity value as a claim about true cluster sizes.

Failure mode. Someone treats perplexity as the number of nearest neighbours and then compares t-SNE output to a UMAP plot built with the same number. Perplexity is a soft count tied to σ_i, so the two settings do not produce the same graph.

Curated: · Written: · Reviewed:

Q&A-23

You cluster 50,000 product embeddings by reducing them with UMAP and running HDBSCAN on the output. How do you decide whether the clusters are real, and what do you check in the original feature space before reporting them?

Answer

The 90-second answer. UMAP is a reasonable front end for HDBSCAN because it compresses the space to a few dimensions where density-based clustering is cheaper and less noisy. The risk is that HDBSCAN then measures density in the embedding, not in the original space. UMAP does not preserve densities: it preserves local neighbourhood graphs with per-point scaling, so a dense patch in the embedding can come from a sparse region of the input and the reverse. The clusters HDBSCAN returns are therefore clusters of the embedding's geometry. I would treat them as candidate groups and test them against the raw features before naming them.

What I run. First, a null model. Permute each column of X independently. That keeps every marginal distribution and destroys the joint structure. Run the identical pipeline on the permuted matrix. If it still returns several clusters with a meaningful persistence score, the procedure manufactures structure at this n_neighbors and min_cluster_size.

# umap-learn 0.5.x, hdbscan 0.8.x, Python 3.11
emb = umap.UMAP(n_neighbors=30, n_components=10, min_dist=0.0,
                metric='cosine', random_state=0).fit_transform(X)
labels = hdbscan.HDBSCAN(min_cluster_size=50).fit_predict(emb)

Second, a raw-space baseline: HDBSCAN on PCA-50 of X with cosine geometry, same min_cluster_size. Compare labels on the points both assign with the adjusted Rand index. Report the noise fraction in both spaces. Third, stability: vary n_neighbors over 15, 30 and 60 and the seed over five values, then check whether each cluster survives with a Jaccard overlap above a threshold you fixed before looking.

Finally, separability in the raw features. Train a logistic regression to predict cluster membership from X with five-fold cross-validation, one cluster against the rest. High held-out accuracy means the groups differ in the original space. Accuracy near the base rate means they may be a layout artefact, however tidy the plot looks.

Failure modes. Small n_neighbors makes UMAP break a continuous manifold into fragments, and HDBSCAN then reports them as separate clusters. Large min_dist packs points into blobs that look like clusters. A clean figure from one seed hides a split that moves under another. Each one shows up in the null run, the seed sweep or the held-out classifier.

I would ship the clusters only if they beat the permuted null, agree with the raw-space baseline above an ARI I chose beforehand, and survive the seed sweep. I would describe them as groups that separate in the embedding and in X, and avoid claiming the UMAP distances between groups mean anything.

Curated: · Written: · Reviewed:

Q&A-24

You plan to fit UMAP on all 1,500 rows, then cross-validate a classifier on its output. Why is that a problem, and what would you change?

Answer

The 60-second answer. The problem is leakage. UMAP builds its fuzzy neighbour graph from every row it is given, so if I fit it on all 1,500 rows before cross-validation, each test row's coordinates already reflect the labels and positions of its training neighbours and of other test rows. The classifier then learns from coordinates that were shaped with knowledge it should not have. The fix is to fit the reducer inside each training fold and call transform on the held-out fold.

Mechanism. UMAP computes a k-nearest-neighbour graph with n_neighbors, converts distances to membership strengths, and then optimises a low-dimensional layout to match those strengths. Every row's position depends on its neighbourhood, so a test row placed by a graph that included training rows carries information about them. Labels do not need to be passed in for this to inflate the score: when classes form compact regions in the feature space, the layout reproduces those regions, and a classifier can exploit that shape even though the test point's own label was never used.

# Leaky: reducer sees test rows
Z = umap.UMAP(n_components=8, random_state=42).fit_transform(X)
scores = cross_val_score(clf, Z, y, cv=5)

# Safe: reducer fit on training fold only
for tr, te in StratifiedKFold(5).split(X, y):
    red = umap.UMAP(n_components=8, random_state=42).fit(X[tr])
    clf.fit(red.transform(X[tr]), y[tr])
    scores.append(clf.score(red.transform(X[te]), y[te]))

Trade-off. Refitting per fold costs five UMAP fits instead of one, which matters on large data. For a plot that will not feed a model, fitting on everything is fine, and the leaky version is only wrong when a model's score is the claim. Exploratory work can use the cheaper pipeline if the score is then re-measured with the safe one.

Failure modes and detection. The first is an inflated score that drops once the reducer is refit per fold. Detect it by running both pipelines on identical splits and comparing the gap. The second is that transform places an unseen point near its nearest training neighbours, so a test population that is absent from training is squeezed onto the nearest known group. Detect it by comparing test-to-training nearest-neighbour distances with the training-to-training distribution.

What I would report. The score from the per-fold pipeline, the same split scored with a PCA baseline at the same dimension, and the transform distance check. A UMAP gain that does not survive those is not a gain.

Curated: · Written: · Reviewed:

Q&A-25

Your UMAP of 5,000 single cells shows a 300-cell arm joining two clusters, and a colleague calls it a differentiation path. What in UMAP's construction can draw that arm without any path existing, and what check do you run before agreeing?

Answer

The 90-second answer. An arm of cells between two clusters is not evidence of a path. UMAP builds a k-nearest-neighbour graph, sets each point's bandwidth so its neighbour weights sum to roughly log2(k), and symmetrises the directed weights with the fuzzy union a + b - a*b. A cell that sits between two populations, or a mixture of cells from both, keeps weak but non-zero edges to each side. The layout then attracts connected points and repels sampled non-neighbours, so that chain of weakly connected cells is drawn as a visible arm. The arm can be a continuum of real transitional states, or it can be a set of mixed cells with nothing developmental about it. The plot alone cannot tell these apart.

Mechanism. The fuzzy union keeps an edge if either direction supports it: a directed weight a from one point and b from the other combine to a + b - a*b. A bridge cell whose neighbours are split between A and B therefore receives two modest weights that survive symmetrisation. In the layout, attraction along these edges pulls the bridge cells toward a line between the clusters, while sampled repulsion from distant points keeps them from collapsing into either cluster. Five negative samples per positive edge is the default, so the arm's shape is partly set by which non-neighbours happen to be sampled.

Check, in order.

  1. Neighbourhood mixing. For each arm cell, count the fraction of its k neighbours in cluster A versus B in the original PCA space. A real transitional state has neighbours that sit on a gradient. A mixture has neighbours that split into two groups with few cells in between.
  2. Per-cell expression. Plot a marker of A against a marker of B for arm cells only. Transitional cells usually form a continuum along one state. Mixed cells, or doublets, tend to show two separate populations, each expressing one parent state strongly.
  3. Parameter stability. Rebuild the graph at n_neighbors 15 and 50 and look at whether the arm persists with the same cell membership. An arm that appears at one value and dissolves at another is a graph effect until shown otherwise.
ObservationReading it supports
Mixing fraction changes smoothly along the arm, and per-cell markers form a continuumCandidate transition, to be tested further
Mixing splits into two groups, and marker plot is bimodalMixture or doublets, not a path
Arm disappears at a different n_neighborsGraph artefact

Failure mode to name. Ordering bias: a pseudotime or PAGA run on this graph will always return an ordering from A to B, so the ordering proves nothing on its own. The test is whether the mixing gradient and the per-cell marker continuum agree, and whether they survive a parameter change.

If the checks support a transition, report a continuum connecting A and B, with the cell count on the arm and the three checks named. Do not write 'differentiation from A to B' without a lineage-level experiment.

Curated: · Written: · Reviewed: