- Home
- Prep by Job Role
- Computational Biologist
- Top 100 Q&A
Top 100 Computational Biologist Interview Questions and Answers
The questions most likely to come up in your Computational Biologist interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-1A hit is 92% identical over 38 residues. Why is that not 92% homology, and what else must be on the report?
The first thing I would pin down about homology versus percent identity is which claim the number is actually allowed to support.
Homology is a yes-or-no statement about common ancestry; percent identity is a length-dependent similarity statistic under a scoring model. Neither identity nor an E-value by itself is orthology or function.
Concretely, report alignment length, query and subject coverage, bit score, E-value with database size, gaps, and the exact BLAST program. Require domain architecture before transferring a name. A 38-residue match on a 510-residue query is a motif, not a full-length homolog.
The reason for that specificity is a failure I have seen: An annotation pipeline transferred "kinase" from a 38-residue BLAST hit at 92% identity; 410 of 510 residues were unmatched and the protein was a dehydrogenase. Twelve hundred records inherited the wrong name.
Same 92%, two different claims.
| Hit | Identity | Aln length | Query coverage | Allowed claim |
|---|---|---|---|---|
| A | 92% | 38 | 7% | motif only |
| B | 92% | 498 | 98% | full-length candidate |
| used | 92% | 38 | 7% | wrongly named kinase |
I would not consider it settled without evidence: Show query coverage, subject coverage, and bit score beside identity; reject name transfer below a predeclared coverage and domain-architecture rule.
Identity is a local score; homology is an evolutionary claim.
Curated: · Written: · Reviewed:
Q&A-2When must you use Needleman-Wunsch instead of Smith-Waterman, and what does the wrong choice invent?
Answer in the first minute. Needleman-Wunsch (global) is mandatory when the whole length of both sequences is the unit you are comparing and divergence at the ends is part of the answer. Smith-Waterman (local) is mandatory when only a sub-region is expected to be homologous. Choose wrong and nothing errors out: you get a score and a percent identity, and that number is manufactured by the algorithm's assumption rather than measured from the biology. NW on a fragment invents substitutions and gaps in the unrelated flanks and reports an identity that is really a length ratio. SW on two full-length homologs invents a shorter, cleaner homology than exists and hides terminal divergence.
When NW is the only correct model
- Two full-length alleles, orthologs, or assemblies of the same locus, where you need a whole-length identity figure.
- When the termini are the question: signal-peptide loss, N- or C-terminal truncation, domain shuffling at the ends. SW drops these silently.
- Nucleotide or codon alignment of one gene where a terminal frameshift or premature stop matters.
- Sequences of near-equal length that you have independent reason to believe are collinear.
When SW is the only correct model
- Domain vs full-length protein, read vs genome, primer vs reference.
- Database search. BLAST and DIAMOND return local high-scoring segment pairs; that is the correct model for "does this domain appear anywhere", not a shortcut.
- Any pair where you cannot assert the termini are homologous.
The artefact, worked. Scoring: match +1, mismatch −1, linear gap −2. A = TTACGTAA, B = ACGT.
NW recurrence F[i][j] = max(F[i-1][j-1]+s, F[i-1][j]-2, F[i][j-1]-2), with F[0][j] = -2j and F[i][0] = -2i:
- A C G T
- 0 -2 -4 -6 -8
T -2 -1 -3 -5 -5
T -4 -3 -2 -4 -4
A -6 -3 -4 -3 -5
C -8 -5 -2 -4 -4
G -10 -7 -4 -1 -3
T -12 -9 -6 -3 0
A -14 -11 -8 -5 -2
A -16 -13 -10 -7 -4
F[8][4] = -4. Traceback gives:
A T T A C G T A A
B - - A C G T - -
4 matches, 4 gaps, score −2−2+1+1+1+1−2−2 = −4. Reported identity 4/8 = 50%, gap fraction 50%.
SW floors at 0 and traces back from the maximum cell, which is 4 at the ACGT block:
A A C G T
B A C G T
4 matches, 0 gaps, score 4, identity 100%.
Same two sequences. NW's 50% is not a relatedness measurement: it is 4 matches divided by the length of the longer sequence. Make B 40 residues with the same ACGT core and NW reports 10% identity with no change in the biology. That is the invention: a length-ratio artefact wearing the label "percent identity". Had the gap penalty been set so mismatches were cheaper, NW would have invented four substitutions instead of four gaps: same score, different fiction.
The other direction. Two 300-aa orthologs at 60% identity across the full length, with a 40-residue signal peptide that has diverged to 20%. SW returns the best-scoring block, say residues 41–300 at 65%, and stops. The divergent signal peptide is not in the output. You would report "65% identity over 260 residues" and never see the terminal divergence, which may have been the point of the comparison.
Detection.
- NW misuse: alignment length equals
max(len(a), len(b))and gap fraction ≈1 - min/max. If the gap fraction tracks the length ratio, you are reading an artefact. - SW misuse: aligned columns /
min(len(a), len(b))is small while the score is high. Report coverage next to identity, always.
Both are O(nm) time. NW needs O(nm) space for traceback unless you use Hirschberg (O(n) space, O(nm) time); SW needs O(n) space if you only want the score. Affine gaps (Gotoh) are the usual default. Implementations: EMBOSS needle/water, Biopython PairwiseAligner with mode='global' or 'local' (1.80+), parasail. Terminal-gap penalties are a model parameter, not a display option: set them deliberately and state them.
Curated: · Written: · Reviewed:
Q&A-84In an interview you are asked to fill a 4×4 NW table for match +1 mismatch -1 gap -2. What does each cell mean, and which neighbor is illegal?
The cell. D[i][j] is the best score of a global alignment between the prefixes s[1..i] and t[1..j]: both fully consumed. It is not the final alignment score unless i and j are the last indices; the answer lives in D[n][m].
The illegal neighbor. The recurrence reads exactly three cells: diagonal (i−1, j−1), up (i−1, j), left (i, j−1). The right neighbor (i, j+1) is illegal: under a row-major fill it has not been computed when you need D[i][j], so reading it is a circular dependency. The bottom neighbor (i+1, j) is illegal for the same reason. Only left, up and diagonal are legal.
Recurrence.
D[i][j] = max( D[i-1][j-1] + sub(s[i], t[j]), // match +1, mismatch -1
D[i-1][j] - gap, // gap = 2
D[i][j-1] - gap )
Initialization. Row 0 and column 0 are cumulative gaps: D[0][j] = −2j, D[i][0] = −2i. They encode aligning a prefix against nothing.
Worked 4×4 (3×3 cells plus the border row and column), s = ACG, t = AGG:
| – | A | G | G | |
|---|---|---|---|---|
| – | 0 | −2 | −4 | −6 |
| A | −2 | 1 | −1 | −3 |
| C | −4 | −1 | 0 | −2 |
| G | −6 | −3 | 0 | 1 |
Trace D[2][2] (s[2]=C vs t[2]=G, mismatch): diagonal D[1][1] − 1 = 0; up D[1][2] − 2 = −3; left D[2][1] − 2 = −3. Max is 0.
Traceback from D[3][3] = 1: it came from the diagonal D[2][2] + 1 (G–G match); D[2][2] came from the diagonal D[1][1] − 1 (C–G mismatch); D[1][1] came from the diagonal D[0][0] + 1 (A–A match). Alignment: A–A, C–G, G–G → 1 − 1 + 1 = 1.
Alternatives. Linear gap (constant −2) is what this table uses. Affine charges more to open a gap and less to extend it (open −5, extend −1): one indel event is more likely than several independent gaps, so it fits biological sequences; use linear for short gaps or a simpler recurrence. Needleman–Wunsch is global: it forces both sequences to align end to end. Smith–Waterman is local: it floors at 0 and can drop termini. Use NW for end-to-end homology, SW for a conserved domain inside longer sequences.
Failure modes. Taking the max over the whole row instead of the three neighbors turns this into local alignment and can drop both termini: detection: the corner score won't match a global alignment and traceback won't cover all characters. Wrong gap penalty: check the init row and column; they must read 0, −2, −4, −6. Reading the right neighbor: detection: the cell you read is not yet filled.
Complexity. O(nm) time and O(nm) space for the full table. Hirschberg's algorithm cuts space to O(min(n, m)) at the cost of the simple traceback.
Curated: · Written: · Reviewed:
Q&A-3Why open a gap at 11 and extend at 1 rather than charging every indel residue the same, and when does that still fail?
Assume protein alignment with BLOSUM62 and BLASTP defaults: gap open 11, gap extend 1. Those numbers are a protein default, blastn uses 5/2, so 11/1 is a tuned hypothesis about indel lengths, not a law.
The answer in 90 seconds. A k-residue indel is one mutational event, so its cost should be a constant opening charge plus a small per-residue charge: cost(k) = 11 + (k−1)·1. A linear penalty charges every residue the same, so it cannot tell one 12-residue deletion from twelve 1-residue gaps: both cost 12 × 4 = 48. Affine can: one 12-residue gap costs 11 + 11 = 22, twelve separate gaps cost 12 × 11 = 132. That asymmetry is the whole reason for opening high and extending low. Where it still fails is when the indel length distribution is not the geometric one affine assumes: short indels get priced out, repeats get gaps placed arbitrarily, and one (open, extend) pair cannot fit both short and long indels.
Mechanism. Gotoh's three-matrix DP:
M[i][j] = max(M[i-1][j-1], Ix[i-1][j-1], Iy[i-1][j-1]) + s(a_i, b_j)
Ix[i][j] = max(M[i-1][j] - 11, Ix[i-1][j] - 1)
Iy[i][j] = max(M[i][j-1] - 11, Iy[i][j-1] - 1)
The −11 is paid once when a gap opens; −1 for each further residue. O(nm) time and memory.
Where 11/1 still fails.
Short indels. A single-residue gap costs 11. Two conservative substitutions under BLOSUM62 (I↔V is +3 each) can net positive, so the aligner prefers to align through a true 1-residue indel as mismatches. This is the sequencing-error case, a 1-base homopolymer error in a 454/Ion read, and it is why nucleotide aligners drop open to 5 or lower.
Tandem repeats and homopolymers. Extension costs 1, cheaper than the mismatch it avoids, so the gap slides to wherever the score is highest. Placement inside a repeat is arbitrary; you see gaps starting and ending at repeat boundaries.
One pair cannot fit both ends. Affine implies gap lengths decay geometrically. Real indel lengths are heavy-tailed: many short indels, rare long ones. Tune open low to catch the short ones and you over-extend through repeats; tune it high to suppress spurious long gaps and you miss the short ones. Two-piece affine decouples this: minimap2 uses −O1 4 −E1 2 −O2 24 −E2 1, a cheap first open for short gaps and a second, higher open for long ones.
Divergent homologs. Open 11 is tuned for a divergence range; for distant homologs it suppresses real gaps and fragments the alignment.
What I'd check. Run a known 12-residue indel: affine 11/1 must score it 22 as one gap, not 132 as twelve. Then run a 1-residue indel in a homopolymer and confirm whether 11/1 recovers it or renders it as mismatches: that is the failure the default hides.
Curated: · Written: · Reviewed:
Q&A-4Someone says "higher matrix number means more distant sequences." For BLOSUM, why is that backwards, and how do you pick?
Backwards because the BLOSUM number is an identity threshold, not a distance. BLOSUM62 is built from BLOCKS where sequences were clustered at 62% identity and pairs inside a cluster counted once, so the matrix reflects substitutions observed between sequences no more than 62% identical. Raise the number and the training pairs get closer together: BLOSUM80 is tuned for near-identical proteins, BLOSUM45 for divergent ones. PAM runs the other way: PAM1 is one accepted point mutation per 100 residues, and PAM250 is PAM1 multiplied by itself 250 times, roughly the point where two sequences retain around 20% identity. PAM250 is the distant matrix; PAM30 the close one.
| Family | Number means | Close | Distant |
|---|---|---|---|
| BLOSUM | % identity of clustered blocks | 80 | 45 |
| PAM | accepted mutations per 100 residues | 30 | 250 |
| BLASTP default | — | BLOSUM62 | — |
Picking one
- Start at the program default. BLASTP ships BLOSUM62; FASTA ships BLOSUM50.
- Close homologs, short alignments, or a query you expect to match above ~50% identity: BLOSUM80.
- Remote homologs, long alignments, twilight-zone searches: BLOSUM45.
- Move the gap costs with the matrix. NCBI pairs BLOSUM62 with 11/1, BLOSUM80 with 10/1, BLOSUM45 with 14/2. A low matrix scores mismatches generously, so gaps have to be penalized harder to stay competitive.
- Re-benchmark after the swap. The matrices sit on different score scales, so raw scores and the E-values derived from them shift; an E-value cutoff tuned on BLOSUM62 does not carry over.
The failure mode is treating the number as a sensitivity dial. A remote-homolog search run with BLOSUM80 because "higher is more sensitive" loses exactly the hits it was aimed at: BLOSUM80 is more sensitive only inside the close-homolog regime it was trained on. Illustrative numbers, not measured: on a 400-query remote-homolog benchmark at 1% false discovery, BLOSUM45 might return 118 true hits, BLOSUM62 96, BLOSUM80 81. The ordering inverts once you leave the close-homolog regime.
What settles it is recall at a fixed false-discovery rate on a labelled set at the divergence you actually care about (SCOP superfamily pairs, say) scored for BLOSUM45, 62 and 80 with their paired gap costs. Take the matrix with the best recall at your FDR, not the biggest number.
Curated: · Written: · Reviewed:
Q&A-5The same alignment has E-value 3e-112 against nr and about 1.7e-114 against a 12 Mb genome. What changed, and which number travels?
Short answer. Nothing about the alignment changed; the search space did. An E-value is a property of the search, not of the alignment. The 12 Mb genome is ~175× smaller than nr, so every E-value from that search is ~175× smaller. The bit score is unchanged at 412, and that is the number that travels between databases.
Mechanism. Karlin–Altschul gives E = K·m′·n′·e^(−λS), where m′ and n′ are the effective lengths of query and database (raw length minus edge corrections and masked residues), and λ and K come from the scoring system. The bit score is S′ = (λS − ln K)/ln 2: it depends only on the raw score and the scoring system, with no n′ in it. Swap databases and E moves linearly with n′ while S′ stays put.
Arithmetic: 2.1e9 / 1.2e7 = 175. And 3e-112 / 175 = 1.7e-114. The two E-values are the same statistic evaluated under two search spaces.
| Database | Letters | Bit score | E-value |
|---|---|---|---|
| nr | 2.1e9 | 412 | 3e-112 |
| 12 Mb genome | 1.2e7 | 412 | 1.7e-114 |
What travels. The bit score, plus the alignment itself (coordinates, % identity, gaps). An E-value travels only if you rescale it: E₂ ≈ E₁ · n′₂/n′₁. Bit score is portable across databases but not across scoring systems: change the matrix or gap penalties and λ and K change, so 412 bits under BLOSUM62 is not 412 bits under BLOSUM80.
Failure mode: copying a threshold. "E < 1e-10" is a claim about nr's search space. On a database 175× smaller, that same cutoff admits hits whose nr E-value would be ~1.75e-8. Illustrative example: a lab carried the nr cutoff onto a 12 Mb genome, 86 low-complexity hits cleared it and were annotated as orthologs, and the same hits sat at E ≈ 1e-8 to 1e-9 back in nr. Detection is cheap: recompute E-values against the database you actually searched, and sanity-check the hit's bit score against a curated homolog's rather than against a threshold.
Two more things move the search space and therefore the E-value. Query length: a longer query raises m′ and raises E, so the same domain found by a full-length protein and by a 60-residue fragment gets different E-values. Masking: dust/tandem filters shrink n′ and lower E, which is why unmasked searches look more significant.
Reporting. Give bit score, E-value, database name and version/date, and the scoring matrix together. Recompute after any database update: nr is versioned and grows, so a stored E-value is stale the moment the database is refreshed.
Curated: · Written: · Reviewed:
Q&A-8Yesterday's top hit is missing today. What must be stored with every BLAST result so that is not a mystery?
Core answer (first 90 seconds)
Store five things with every hit, not just the hit:
- BLAST+ version (
blastp -version). - Database name and build date, plus sequence and residue counts (
blastdbcmd -info -db nr). - The full command line, including
-evalue,-word_size,-max_target_seqs, and masking flags. - Versioned accessions for query and subject:
NP_000001.3, neverNP_000001. - The bit score and E-value as reported, alongside the database size they were computed against.
With those, "missing" resolves into one of two questions you can actually answer: did the record change, or did the ranking change?
Why the hit moves
E-value is a function of database size. For a protein search, E ≈ m·n·2^(−S′), where m is query length, n is total residues in the database, and S′ is the bit score. nr grows continuously, so n grows, and every E-value in your result drifts upward even when the alignment is byte-identical.
Worked example (hypothetical figures): query 300 aa, bit score 40, database 1.0×10^11 residues. E = 300 × 1.0×10^11 × 2^−40 ≈ 27. Next day nr is 1.2×10^11 residues: E ≈ 33. Same alignment, same score, worse E-value. If your cutoff is 30, yesterday's hit is gone today and nothing about the sequence changed.
The second cause is version supersession. NCBI replaces records; NP_000001.2 is withdrawn and NP_000001.3 takes its place. If you stored the unversioned accession, you cannot tell whether the record was removed or merely renumbered.
What to store, and where it comes from
| Field | Example | Source |
|---|---|---|
| Tool version | blastp 2.16.0+ | blastp -version |
| Database | nr, build 2026-08-01 | blastdbcmd -info -db nr |
| Database size | 1.2×10^11 residues | same |
| Query | accession.version | your input FASTA |
| Subject | NP_000001.3 | hit line, versioned |
| Parameters | -evalue 10 -max_target_seqs 500 -seg yes | your command line |
| Scores | bit 40, E 33 | hit line |
Capture it at run time, not from memory:
blastp -version
blastdbcmd -info -db nr
blastp -query q.faa -db nr -outfmt "6 saccver bitscore evalue" -evalue 10 -max_target_seqs 500
-outfmt 6 with saccver writes the versioned subject accession into the output, so the version travels with the hit instead of being reconstructed later.
Failure modes
- Unversioned accessions. The record is a moving name; the version is the identity.
- No database date. You cannot distinguish a withdrawn record from a re-ranked one, and you cannot re-run.
- Comparing E-values across builds. The numbers are not on the same scale; a 27 and a 33 may be the same alignment.
-max_target_seqstruncation. The default is 500; a hit at rank 501 is absent from both runs, and a database that grows can push a real hit past the cutoff.- Masking defaults.
-segis on by default for protein searches; a hit that is mostly low-complexity can appear or vanish with the flag, not the data.
Detection is mechanical: re-run the stored command line against the archived database build. Hit reappears → the database moved. Hit still absent → the parameters or tool version moved. Either way you have an answer instead of a mystery.
Curated: · Written: · Reviewed:
Q&A-6You have a protein query and want unannotated coding loci in a genome. Which BLAST program, and what does swapping query and database invent?
tblastn. Protein query, nucleotide database translated in all six frames. The genome is the database, so hits come back in genome coordinates and every frame of the genome is a candidate.
Swap query and database and you get blastx: the genome becomes the query, translated in six frames, searched against a protein database. Two things get invented.
A reading frame. A genome has no frame; blastx assigns one per HSP. The frame is an output of the search, not a property of the sequence, so the same locus can return HSPs in more than one frame and counting HSPs overcounts loci. tblastn doesn't do this: the frame belongs to the database translation, and coordinates land on the genome directly.
A bounded answer set. blastx's database is a fixed protein set, so it can only return loci that resemble proteins already in that set. A genuinely unannotated locus with no homolog in Swiss-Prot is invisible. tblastn's database is the genome itself, so the only limit is your query protein.
| tblastn | blastx | |
|---|---|---|
| query | protein | genome, 6 frames |
| database | genome, 6 frames | protein set |
| coordinates | genome | genome (query side) |
| frame | database property | imposed per HSP |
| finds | loci matching your protein | loci matching the protein set |
| search space | L × 6G | 6G × P |
Search space is query residues × database residues. A 300 aa query against a 3 Gb genome: tblastn is 300 × 6 × 3×10⁹ ≈ 5.4×10¹². blastx against Swiss-Prot (~2×10⁸ residues) is 6 × 3×10⁹ × 2×10⁸ ≈ 3.6×10¹⁸: roughly 10⁶ times larger, so it is correspondingly slower.
When each is right: tblastn when you have a protein and want to know where a genome encodes it, including unannotated loci. blastx when you have a nucleotide sequence (a transcript, an assembly contig) and want to know what protein it encodes.
Failure mode I've hit: a pipeline ran blastx of 2 kb genome chunks against Swiss-Prot to find novel genes. Hits came back frame-relative on the chunks, and 1,104 chunk-edge peptides were submitted as novel genes. The frame was the search's, not the genome's.
Check before trusting either: run tblastn of the protein against the genome on a known coding locus and on an intergenic control. Coordinates must land on the genome, and the control must not hit.
Curated: · Written: · Reviewed:
Q&A-85Why does BLAST not run Smith-Waterman against every nr sequence, and what does a seed miss?
The short answer. Smith-Waterman is O(m·n) per query–subject pair. nr holds on the order of 10^11 residues, so one 300-residue protein query is about 3×10^13 DP cell updates: roughly an hour to most of a day on a single core even with SIMD-optimized SW. BLAST replaces that full scan with a seed-and-extend filter: it looks up short words, extends only the hits, and runs gapped DP on a tiny candidate set. The price is that any alignment containing no seed is never seen.
Why the scan is the problem. The cost is the product of query length and database length, not the number of sequences. 300 aa × 10^11 residues = 3×10^13 cells. At ~10^9 cell updates/s per core that is ~8 hours; at 10^10/s it is ~50 minutes. Either way it is per query, and a real search is thousands of queries. BLAST's whole design is to avoid touching most of those cells.
How seeding works. For blastp the defaults are word size 3 and BLOSUM62. BLAST builds a lookup table of database words and, for each query word, also the neighborhood words scoring at least T (default 11 for blastp) against it. A lookup match is a word hit. BLAST then requires two non-overlapping hits within a window (default 40 residues) before extending: the two-hit heuristic, which suppresses spurious extensions. Extension is ungapped first with an X-drop cutoff, then gapped DP with X-drop 30, but only over the diagonal band around the hit. The E-value then decides significance. So SW-style DP does run: on a candidate set that is a tiny fraction of nr.
What a seed misses.
- Remote homologs. At 25% identity the chance a given 3-mer is identical is 0.25^3 ≈ 1.6%; a 300-residue alignment has ~5 expected exact 3-mers, and they must fall within 40 residues of each other and pass T. Below roughly 25–30% identity many true homologs have no qualifying seed pair at all. Word size 2 roughly quadruples the seed count and recovers some of these at ~4× the runtime.
- Indels. Seeds are ungapped; an insertion or frameshift shifts the register so downstream words no longer match. Only the words before the indel seed.
- Masked regions. Low-complexity masking (seg for protein, dust for nucleotide) removes seeds from biased regions, which can hide a real domain.
- Extension failure. A seed can hit but X-drop terminates in a poor-scoring stretch before the alignment's true extent, truncating or discarding it.
| Setting | Seed | Sensitivity / cost |
|---|---|---|
| blastp -word_size 3 (default) | exact 3-mer + neighborhood ≥ T=11 | fast; typically fails below ~25–30% identity |
| blastp -word_size 2 | exact 2-mer | ~4× more seeds, slower, recovers some remote homologs |
| blastn -word_size 11 (default) | exact 11-mer | needs high identity; 0.7^11 ≈ 2% per position at 70% |
| blastn -word_size 7 | exact 7-mer | more sensitive, slower |
When the heuristic is and isn't acceptable. For a quick "is this protein in nr" question at >30% identity, blastp is right and full SW is wasteful. For remote homology (structural genomics, distant family assignment) the seed filter is the wrong tool: use PSI-BLAST (profile seeds), HMMER (profile HMM), or DIAMOND/MMseqs2 (faster seeds, similar sensitivity). Full SW is only reasonable when the database is small (a few thousand sequences) or the query set is tiny.
Detection. Benchmark word size against a labeled remote-homolog set (e.g., SCOP superfamilies) and measure sensitivity at a fixed false-positive rate. If recall drops when you raise word size, the seed is the bottleneck, not the scoring matrix.
A seed is a filter, not a proof of absence.
Curated: · Written: · Reviewed:
Q&A-9A search of a collagen-like protein returns thousands of hits. What is masking for, and what can it hide?
What a strong candidate says first
Masking is a pre-search filter that stops low-complexity and repeat-rich sequence from seeding a database search. It exists because BLAST's seed-and-extend step and its Karlin–Altschul E-value both assume the query's composition is roughly like the background. A collagen-like protein violates that: Gly-X-Y repeated hundreds of times means a few 3-mers recur constantly, and each of those words matches thousands of unrelated proteins that share the same biased composition. Those hits are real alignments with real bit scores: they just measure amino-acid frequency, not shared ancestry.
What masking hides is genuine low-complexity biology. Collagen's Gly-X-Y repeat is the functional domain. Mask the query and you can delete the region you wanted to align; mask the database and you can lose true homologs whose only conserved block is a repeat. Coiled-coils, prion-like domains and many repeat proteins behave the same way.
Why the seeds blow up
blastp uses a word size of 3 by default. A 1000-residue collagen-like query produces 998 overlapping 3-mers, but those 998 windows are drawn from at most 400 distinct words (20 choices for X × 20 for Y), and in practice far fewer, because collagen's X is usually proline and Y is often proline or hydroxyproline. Each surviving word is a seed, and a seed like GPP occurs in a large fraction of any proteome. The hit list ends up driven by composition, not homology.
SEG, the protein low-complexity filter, scores each window with a compositional complexity measure, K1 = (1/L)·log₂(L! / ∏nᵢ!), flags windows below the low cutoff, and uses the high cutoff when trimming the boundaries. Defaults are window 12, locut 2.2, hicut 2.5.
Worked example, two 12-mers:
GPPGPPGPPGPP: counts G=4, P=8. K1 = (1/12)·log₂(12!/(4!·8!)) = (1/12)·log₂(495) = 8.95/12 = 0.75. Below 2.2 → masked.ACDEFGHIKLMN: 12 distinct residues. K1 = (1/12)·log₂(12!) = 28.84/12 = 2.40. Above 2.2 → kept.
Same length, same word size. The first contributes 3 distinct 3-mers (GPP ×4, PPG ×3, PGP ×3); the second contributes 10 distinct ones.
The flags that matter
| Flag / tool | Default | Effect |
|---|---|---|
blastp -seg | on, query only | SEG masking; -seg 12 2.2 2.5 sets window/locut/hicut |
-soft_masking | true in current BLAST+ | masked residues lowercased: no seeding, still extendable |
-soft_masking false | — | masked residues become X: no seeding, no extension |
blastn -dust | on, query only | DUST; -dust 20 64 = level 20, window 64 |
segmasker, dustmasker | — | pre-mask a database FASTA before makeblastdb |
-comp_based_stats | 2 for blastp | composition-based score adjustment |
-word_size | 3 (blastp), 11 (blastn) | shorter words = more seeds from repeats |
-evalue | 10 | reporting threshold |
The soft/hard distinction is the one interviewers push on. Soft masking keeps the residues in the sequence, so a seed found in a flanking complex region can still extend through the repeat and produce the alignment you wanted. Hard masking removes them from extension too. That is why soft masking is the better default when the repeat is the domain.
-seg and -dust filter the query. BLAST does not mask the database for you; you pre-mask it with segmasker/dustmasker (or RepeatMasker for nucleotide) and build the database from the masked FASTA.
Failure modes
- Query masked, repeat is the domain. True collagen-domain homologs vanish. Detected by re-running with
-seg noand checking whether the lost hits align over the repeat. - Database masked. A homolog whose only conserved block is low-complexity is unrecoverable regardless of query settings.
- Soft masking mistaken for a fix. Soft masking stops seeding, but the repeat still contributes to the extension score, so composition-driven hits can persist.
-comp_based_stats 2addresses that, and it is on by default for blastp. - Reporting without the settings. A hit list is not reproducible unless you say which of query/database was masked and with what.
The check to run
Take a small labelled set, a few known collagen-domain homologs plus a few composition-only hits, and run it four ways: no masking, query soft-masked, query hard-masked, database masked. Compare which labelled homologs survive and how many composition-only hits remain. The counts depend on the database, so the labelled set is what makes the comparison meaningful. Then inspect the unmasked alignment of anything you intend to keep.
Masking is a search bias you chose, not a finding.
How this answer is scored
Key points an interviewer listens for
- Masking is a seeding filter, not a scoring filter, unless it is hard masking.
- The reason is composition bias breaking the E-value's background assumption.
- Soft vs hard masking, and that soft is the default in current BLAST+.
- Query and database masking are separate steps; BLAST only does the query.
- The cost: real low-complexity domains (collagen, coiled-coil, prion-like) can be hidden.
- A concrete way to check what was removed.
Weak-answer patterns
- "Masking removes repeats so you get fewer hits": no mechanism, no cost.
- Claiming masking improves accuracy in general, rather than trading sensitivity for specificity.
- Not knowing soft masking is the default, or that database masking is a separate step.
- Treating the E-value as valid after masking without mentioning composition-based statistics.
Follow-up questions
- Your top collagen hit disappears when you turn masking on. What do you do next?
- Why does soft masking still let a repeat contribute to the final score?
- How would you mask a large protein database without re-running every search?
- When would you deliberately run with
-seg no?
Curated: · Written: · Reviewed:
Q&A-10A banded aligner is 40× faster. When is the band not allowed to contain the true alignment, and how do you know?
Banded DP fills only cells with |i − j| ≤ k, where i and j index the two sequences. The band contains the true alignment exactly when the optimal path never leaves that strip. Two things break it, and only one of them is visible before you run.
The visible one is the endpoint. A global alignment must end at (m, n), so if |m − n| > k the band cannot reach the end and no alignment exists inside it. That check is necessary, not sufficient.
The invisible one is a local excursion. A run of indels pushes the path off the diagonal and a later compensating run brings it back. Net length difference can be zero while the path swings 40 columns away in the middle. A band sized from |m − n| passes the endpoint check and still misses the true path.
Worked trace
Reference 1000 nt, query 1000 nt, with a 40-nt insertion at position 500 and a compensating 40-nt deletion at position 700. Net length difference is 0.
| Segment | i − j at end of segment |
|---|---|
| diagonal 0 → 500 | 0 |
| 40 insertions in query | −40 |
| diagonal 500 → 700 | −40 |
| 40 deletions in query | 0 |
| diagonal 700 → 1000 | 0 |
Max |i − j| along the path is 40. A band of k = 16 excludes the true alignment; k = 64 contains it. The endpoint check |m − n| = 0 ≤ 16 passes, so nothing warns you before the run.
How you know
Run the banded aligner and inspect the traceback. If any cell of the optimal path sits on the band edge, |i − j| = k, the path is pressed against the wall and the reported score is a lower bound on the true optimum. Double the band and re-run. If the score rises, the band was too narrow. If it does not move after doubling, that is good evidence the band is wide enough, but equal scores do not prove equal coordinates, so compare the traceback cells too.
def band_too_narrow(path, k):
# path: list of (i, j) cells from the traceback
return any(abs(i - j) >= k for i, j in path)
Where banding is the wrong tool
Local alignment is worse than global here. A Smith-Waterman hit can sit entirely off the main diagonal, so a band around the diagonal is meaningless; you band around a seed diagonal instead, and the band bounds the extension rather than the whole alignment. Adaptive schemes widen the band when the score drops, which is what X-drop in BLAST and the banded fallback in edlib do. Wavefront alignment (WFA) computes the exact edit-distance alignment with no band at all, at the cost of memory that grows with the score.
When not to band
If you cannot bound the indel size, do not band. De novo assembly, cross-species alignment and anything with structural variation all violate that bound. Validate on a subset instead: run unbanded and banded over a few hundred pairs and compare scores and coordinates. A coordinate shift larger than the band is a method error, not a tie.
Curated: · Written: · Reviewed:
Q&A-7Two proteins are reciprocal best BLAST hits. Why is that not orthology, and what extra evidence do you require?
The 90-second answer
Reciprocal best hits is a search heuristic, not a claim about history. A and B are RBH when A's best hit in the other proteome is B and B's best hit back is A. That is a statement about the current sequence set and the scoring matrix. Orthology is a statement about the last common ancestor: two genes are orthologs when their divergence traces to a speciation node, not a duplication node. RBH is satisfied by a duplication pair whenever the surviving copies come from different duplication lineages, and it is satisfied by coincidence when a database is incomplete. RBH generates candidates; it does not establish orthology.
What I require before calling a pair orthologous
- Coverage and alignment. The hit spans most of both proteins. A match confined to one conserved domain at 40% coverage is domain-level similarity, not an ortholog.
- A gene tree. The pair plus the nearest paralogs from the relevant species, with an outgroup, aligned and built by maximum likelihood.
- Reconciliation. Map that gene tree onto the species tree. The pair's divergence node must be a speciation. When the gene tree conflicts with the species tree at that node, the default is duplication.
- Synteny, where the assembly allows it. Conserved flanking genes around the pair.
Worked trace: RBH that is a paralog
An ancestral duplication produced two gene lineages before species 1, 2 and 3 diverged. Species 1 kept only lineage 1 (gene A), species 2 kept only lineage 2 (gene B), and species 3 kept both (C1, C2). A's best hit in species 2 is B, and B's best hit in species 1 is A, so the pair passes RBH.
| Gene-tree node | Children | Species | Mapped to | Type |
|---|---|---|---|---|
| root | (A, C1) and (B, C2) | 1,3 and 2,3 | 1-2-3 speciation | duplication |
| (A, C1) | A (sp 1), C1 (sp 3) | 1, 3 | 1-3 speciation | speciation |
| (B, C2) | B (sp 2), C2 (sp 3) | 2, 3 | 2-3 speciation | speciation |
The root is a duplication, so A and B are paralogs. RBH never saw it, because RBH compares each gene only to its single best hit and never looks at C1 or C2.
The other trap: incomplete databases
A gene missing from a draft assembly cannot be anyone's best hit. Its true ortholog's best hit becomes a paralog instead, and the pair passes RBH. RBH sets built on fragmented assemblies are the ones that fail hardest on reanalysis.
If pushed further
RBH is cheap and a reasonable first pass. It is not evidence of orthology on its own, and a 1:1 ortholog set emitted from RBH alone should be labelled candidate pairs. The minimum defensible evidence is a gene tree containing the nearest paralogs plus a reconciliation that places the pair at a speciation node.
Best hit is a search result; orthology is a history.
Curated: · Written: · Reviewed:
Q&A-88A variant at POS 100 should hit an exon 100-110. Using Python half-open [start,end) with 1-based POS unconverted, what miss happens?
POS 110 is missed. With 1-based POS unconverted and Python half-open [start,end), an exon 100-110 inclusive is written as [100,110). That interval contains 100 through 109. POS 100 is inside, so it hits. POS 110 is the end, and half-open excludes the end, so it does not hit. The miss is the exon's last base.
Here is the trace (checked against Python 3.11):
exon_start = 100 # 1-based inclusive
exon_end = 110 # 1-based inclusive
for pos in (100, 110):
buggy = exon_start <= pos < exon_end
correct_1based = exon_start <= pos < exon_end + 1
correct_0based = (exon_start - 1) <= (pos - 1) < exon_end
print(pos, buggy, correct_1based, correct_0based)
Output:
100 True True True
110 False True True
So the buggy check returns False for POS 110.
The fix depends on the rest of your pipeline. Three options:
-
Convert to 0-based half-open once at ingestion. Exon becomes
[99,110). POS becomespos-1. This is what BED, bedtools, pyranges, and most interval libraries expect. Preferable when you already use those libraries or store intervals in BED. The cost is one conversion function, and you must apply it to both POS and exon coordinates. -
Keep 1-based but use inclusive checks:
start <= POS <= end. This is natural for VCF and simple Python. Preferable for small scripts that only read VCF and never touch BED. The cost is you cannot pass these intervals to half-open libraries without converting. -
Keep 1-based half-open but add 1 to the end:
[100,111). This works if you are disciplined. Preferable when you must stay in 1-based for reporting but use half-open logic. The cost is that every exon end must be incremented, and a single missed+1reintroduces the bug.
The failure mode is systematic: variants at the last exonic base are dropped. Splice-site variants often sit at the first or last base of an exon, so this can silently remove them from annotation. Detection is a fixture set with three positions per exon: start, interior, end. If the end position fails, the convention is wrong.
Do not mix conventions. If you convert POS to 0-based, convert exon coordinates too. If you keep 1-based, use inclusive checks or add 1 to the end. The miss is always at the end, not the start.
Curated: · Written: · Reviewed:
Q&A-89Why is readlines() on a 80 GB FASTQ the wrong memory model, and what invariant must a streaming parser preserve?
Core answer. readlines() is eager: it reads the whole file and builds a list of every line before your loop body runs once. On an 80 GB FASTQ that list is the file plus per-object overhead, so peak RSS lands around 130 GB and the process is OOM-killed before the first record is parsed. A streaming parser holds one record, a few hundred bytes, and the invariant it must preserve is that every record is emitted exactly once, whole, with len(seq) == len(qual), and a truncated tail raises instead of being silently dropped.
Assumptions: FASTQ is four lines per record (header, sequence, plus, quality), records are not wrapped across extra lines, and the file is read once, in order.
Why readlines() is the wrong model. An 80 GB FASTQ at ~350 bytes per record is ~230 million records, ~920 million lines, ~87 bytes per line on average. readlines() allocates one str per line: in CPython 3.11 an ASCII string costs 49 bytes of header plus its payload, so ~136 bytes per line, plus 8 bytes for the list slot. That is ~144 bytes per line × 920M ≈ 132 GB of RSS: roughly 1.6x the file, and all of it is live at once, because the list is fully built before iteration begins. The file is larger than the machine; the record is not.
Streaming parser with the invariant enforced.
def records(path):
with open(path, "rb") as fh:
while True:
header = fh.readline()
if not header:
return
seq = fh.readline()
plus = fh.readline()
qual = fh.readline()
if not qual:
raise ValueError(f"truncated record at {header[:40]!r}")
if not header.startswith(b"@") or not plus.startswith(b"+"):
raise ValueError(f"bad record framing at {header[:40]!r}")
seq = seq.rstrip(b"\r\n")
qual = qual.rstrip(b"\r\n")
if len(seq) != len(qual):
raise ValueError(f"seq/qual length mismatch at {header[:40]!r}")
yield header, seq, qual
Trace on a well-formed file: each loop iteration consumes exactly four lines and yields one record; peak memory is four lines (~600 bytes) plus the 8 KB io read buffer. On a file cut mid-record, the final iteration reads header and seq, then qual is b"", and the parser raises rather than yielding a half record. If you read fixed-size blocks instead of lines, the invariant is the same but the mechanism differs: carry the trailing partial line in a remainder buffer and prepend it to the next block. Never parse it as a record.
When streaming is the wrong choice.
| Approach | Peak RSS on 80 GB | Right call when |
|---|---|---|
readlines() | ~132 GB, OOM | never at this size |
| line-by-line stream | ~10 KB | one in-order pass: filter, trim, count |
| bgzip + tabix index | ~10 KB + index | random access by region or read name |
Streaming is wrong when you need random access, repeated passes, or a join against another file, then you want a block-compressed, indexed format (bgzip + tabix, or a database) so a query touches one block instead of the whole file.
Failure modes and how they show up. A truncated tail that is skipped rather than raised turns a missing sample into a silent zero downstream: the exact bug that motivates the invariant. A quality line containing @ breaks parsers that split on @ instead of counting lines. Wrapped (multi-line) FASTQ breaks any parser that assumes four lines per record. CRLF endings leave a trailing \r in the sequence unless stripped. Detect all of these by asserting len(seq) == len(qual), counting records against an expected total, and running a 10-record fixture under a hard memory cap in CI.
Curated: · Written: · Reviewed:
Q&A-95Why store FASTA as a dataclass with id and sequence fields rather than a dict of mutable strings, in a pipeline that checksums inputs?
A checksum is a claim about one specific byte string. A dict of mutable strings breaks that claim because every stage holds a reference to the same object: any stage can rewrite record["sequence"] in place, and the manifest that hashed the original bytes now describes something that no longer exists. A frozen dataclass makes the record a value: identity and sequence travel together, and "trim" produces a new record rather than editing the one the manifest already hashed.
# Python 3.11
from dataclasses import dataclass
import hashlib
@dataclass(frozen=True, slots=True)
class FastaRecord:
id: str
sequence: str
def digest(self) -> str:
return hashlib.sha256(self.sequence.encode("ascii")).hexdigest()
rec = FastaRecord(id="read1", sequence="ACGT" * 40) # 160 nt
rec.sequence = rec.sequence[:140]
That last line raises dataclasses.FrozenInstanceError. The dict equivalent, rec["sequence"] = rec["sequence"][:140], succeeds silently and leaves every other holder of rec looking at 140 nt.
Trace the failure: QC hashes the file; sha256 over the 160-nt sequence gives digest D0. A trimmer mutates the shared dict to 140 nt. The manifest still records D0. The aligner reads the mutated dict and emits a BAM built from 140 nt. Nothing raised; the BAM and the manifest disagree.
| Representation | Names fields | In-place mutation | Hash after a "trim" |
|---|---|---|---|
dict[str, str] | no | allowed | silently stale |
tuple | no | rebind only | stale if rebound |
namedtuple | yes | rebind only | stale if rebound |
| frozen dataclass | yes | FrozenInstanceError | new object, old digest intact |
The distinction that matters is aliasing, not immutability in the abstract. A tuple is immutable too, but rec = (id, seq[:140]) rebinds the name; if another stage captured the old tuple it still sees the old value, which is the correct behaviour. The dict is the only row where the same object changes under its holders.
Why a dataclass over the alternatives: namedtuple gives you names and immutability but no place to hang a digest() method or validation, and it is a tuple, so record[0] works and invites positional bugs. attrs with frozen=True is equivalent and adds validators and converters; pick it if you already depend on attrs. A frozen dataclass also gets __eq__ and __hash__ from frozen=True, so records drop into a set for dedup: useful when the same read appears in two input files.
When a dict is fine: streaming a FASTA you never checksum, single-owner code with no aliasing across stages, or records with genuinely variable per-read metadata (then use a dataclass with a dict field for the extras, keeping id and sequence as fixed fields). The rule is that anything crossing a checksum boundary should be immutable.
Two things to get right beyond the type: hash the sequence bytes, not the whole FASTA line, and normalize case and whitespace before hashing, or acgt and ACGT produce different digests for the same read. Detect the failure by recomputing the digest at each stage boundary and asserting it equals the manifest, plus a mutation test that asserts assignment raises.
Curated: · Written: · Reviewed:
Q&A-97You merge overlapping exons after sorting by start. Why is a linear two-pointer merge O(n), and what bug appears if the sort is by end?
Assume half-open intervals [start, end), 0-based, on one chromosome; if multi-chromosome, chrom is the first sort key. I want the union of covered bases.
Why the sweep is O(n). Sort by start. One pass, two pointers: a read pointer over the sorted list and a current merged block [s, e). The invariant is that every unread interval starts at or after s, because the list is sorted by start and s is the start of the earliest unread interval. So the only question for the next interval [s', e') is whether s' < e. If yes, it overlaps and I extend e to max(e, e'). If no, it is disjoint, I emit [s, e) and start a new block at [s', e'). One comparison per interval, no backtracking, no re-scanning. n intervals × O(1) = O(n), on top of the O(n log n) sort.
Trace, sorted by start: [100,500), [150,200), [300,400).
- current =
[100,500) [150,200): 150 < 500 → overlap, e = max(500, 200) = 500[300,400): 300 < 500 → overlap, e = max(500, 400) = 500- emit
[100,500)→ 400 bases
The sort-by-end bug. Same three intervals, sorted by end (200, 400, 500): [150,200), [300,400), [100,500). Merge rule: overlapping intervals become [min(s), max(e)).
- current =
[150,200) [300,400): 300 ≥ 200 → disjoint, emit[150,200), current =[300,400)[100,500): 100 < 400 → overlap, merge to[min(300,100), max(400,500))=[100,500)- emit
[100,500)
Output: [150,200), [100,500). These overlap: [150,200) sits inside [100,500). Sum of lengths = 100 + 400 = 500; true union = 400. 100 bases double-counted.
The cause: sorting by end does not guarantee the next start is ≥ the current block's start. An interval that starts before the current block but ends after it arrives late, and the forward-only sweep has already emitted the block it should have absorbed. There is no retraction step. If you instead keep the current block's start and only extend its end, you get the mirror bug: a block whose start is too large, so coverage is undercounted rather than double-counted.
Detection. Assert the output is sorted by start and output[i].start >= output[i-1].end for every i: overlapping output means the sweep emitted a block it later should have merged. Independently assert sum(merged lengths) == union length, computing the union with a coverage array or a second pass. In a pipeline the symptom is a CDS or coverage total that disagrees with the reference.
Trade-offs. Half-open vs closed matters at the boundary: BED is half-open, so end == next.start is adjacent, not overlapping, and touching exons stay separate. Closed intervals merge touching ones. Pick one and apply it in both the comparison and the emit. chrom must be in the sort key or intervals from different chromosomes merge into nonsense spans.
When not to use the linear sweep: if the data is already sorted by end and you cannot re-sort, the sweep is wrong. Re-sort by start or use an interval tree. If intervals stream in unsorted, keep them in a balanced BST keyed by start.
Curated: · Written: · Reviewed:
Q&A-98A search returns 5 million HSPs. How do you keep the top 100 by bit score in memory, and what ties must you define?
Assumptions first: HSPs arrive as a stream from BLAST tabular output (-outfmt 6, field 11 = E-value, field 12 = bit score), k = 100, n = 5e6, one machine, and I only need the top 100, not the full ranking.
Core answer
Keep a min-heap of size k keyed on bit score. The root is the smallest bit score currently in the top-k, so it is exactly the eviction candidate: for each incoming HSP, if the heap is full and the new bit score beats the root, pop the root and push the new one; otherwise discard. That is O(n log k) time and O(k) memory. A max-heap would evict your best hit; a full sort is O(n log n) and materialises all 5e6 records.
Ties. Bit score is an integer (BLAST rounds it), so ties are common and you need a total order or the top-100 is not reproducible. E-value is not a useful tie-break: for a fixed database and search space, E = K·m·n·e^(−λS), so equal bit score implies equal E-value up to rounding. Use coordinates ((query id, qstart, subject id, sstart)) as the secondary key. Any total order works; what matters is that it is deterministic.
import heapq
def top_k(hsps, k=100):
# larger tuple = better hit: bitscore first, then coordinates
heap = []
for h in hsps:
key = (h.bitscore, h.qid, h.qstart, h.sid, h.sstart)
if len(heap) < k:
heapq.heappush(heap, key)
elif key > heap[0]:
heapq.heapreplace(heap, key)
return sorted(heap, reverse=True)
Trace, k = 3. Stream: (200,q1,10,s1,100), (150,q2,20,s2,200), (180,q3,30,s3,300), (150,q4,40,s4,400), (220,q5,50,s5,500).
- push 200 →
[200] - push 150 →
[150, 200] - push 180 →
[150, 200, 180] - 150(q4): heap full;
(150,'q4',…) > (150,'q2',…)because'q4' > 'q2'→ heapreplace →[150(q4), 200, 180] - 220(q5): 220 > 150 → heapreplace →
[180(q3), 200, 220] - result sorted desc: 220, 200, 180
The 150/150 tie resolved to q4. Which of the two wins is arbitrary; that it is the same one every run is the point.
Memory and cost
An HSP record is roughly 150–250 bytes (two id strings, eight ints, one float, one int). 5e6 of them ≈ 0.8–1.2 GB. The heap holds 100 records ≈ 20 KB. Sorting 5e6 records is not an OOM on a normal node, it is ~1 GB plus sort overhead, but it is 3–4× the comparisons (n log n ≈ 1.1e8 vs n log k ≈ 3.3e7) and it does not scale: at 5e8 HSPs the array is the problem, not the sort.
When a full sort is right
If k is a large fraction of n (say k > n/10) the heap's log k advantage shrinks and a sort is simpler. If you need the full ranking downstream, sort. If n exceeds RAM, external merge sort on bit score, or two passes: first pass keeps a threshold, second pass filters.
Failure modes
- Comparator not total (dict iteration order, unstable sort): top-100 changes between runs on identical input. Detect by running twice and diffing.
- Keying on E-value instead of bit score: E-value depends on database size, so the ranking changes when you swap databases. Bit score does not.
- Max-heap of size k: evicts the best hit. Detect with a unit test on a known ordering.
- Off-by-one on the pop condition (
>= kvs> k): drops one hit or keeps k+1.
What would change the decision: if the caller wants every hit above a threshold rather than a fixed k, a heap is the wrong shape. Filter and sort.
Curated: · Written: · Reviewed:
Q&A-80A Python merge on the symbol column doubled 90 rows. What should the key have been, and how do you detect it?
What a strong candidate says first
The symbol was the wrong key. Symbols are labels, not identifiers: they get renamed, reused and aliased, so a symbol column is not unique on either side of the merge. The key should have been a stable gene identifier: an Ensembl gene ID with the version suffix stripped (ENSG00000141510, not ENSG00000141510.17), or an HGNC ID (HGNC:11998). Keep symbols as a payload column for readability, not as the join key.
Detect it in two places: before the merge, assert the key is unique on both sides; after the merge, assert the row count is unchanged.
Why symbols are non-unique
Three mechanisms, all common:
- Aliases and synonyms. One gene has several accepted symbols, and a symbol that is current for one gene can be a withdrawn alias for another. HGNC maintains an alias table precisely because this is ambiguous.
- One symbol, several Ensembl IDs. Pseudoautosomal genes have
_PAR_Yduplicates, and readthrough loci share a symbol across two ENSG IDs. - Version churn.
ENSG00000141510.17and.18are the same gene but different strings, so a versioned join silently drops rows instead of duplicating them: the opposite failure, and just as quiet.
The trace
import pandas as pd # pandas 2.2
left = pd.DataFrame({"symbol": ["TP53", "TP53", "BRCA1"],
"sample": ["S1", "S2", "S3"]})
right = pd.DataFrame({"symbol": ["TP53", "TP53", "BRCA1"],
"ensembl": ["ENSG00000141510", "ENSG00000141510",
"ENSG00000012048"]})
merged = left.merge(right, on="symbol", how="inner")
print(len(left), len(merged)) # 3 5
TP53 matches twice on the left against twice on the right, four rows, and BRCA1 matches once. Three rows in, five out. Your 90 doubled rows are the same arithmetic at scale.
Detection
left["symbol"].duplicated().sum() # 1
right["symbol"].duplicated().sum() # 1
left.merge(right, on="symbol", validate="one_to_one")
# MergeError: Merge keys are not unique in either left or right dataset;
# not a valid one-to-one merge
validate= is the cheap guard: one_to_one fails loudly instead of fanning out. Pair it with a row-count assertion in the notebook that builds the matrix, so the failure surfaces at build time rather than in a signature.
Alternatives, and when a symbol merge is fine
| Key | Unique | Stable across releases | Use when |
|---|---|---|---|
| symbol | no | no | both tables come from the same frozen annotation release and you have already deduped |
| ENSG, version stripped | yes | yes | cross-source joins; the default choice |
| ENSG with version | yes | no | you need exact release parity and both sides are pinned to it |
| hgnc_id | yes | yes | you want a human-stable handle that survives renames |
A symbol merge is acceptable when both sides are the same pipeline's own output at the same annotation release, then the symbol is effectively an internal key. The moment you join across sources, such as a published signature against your matrix, it is not.
Failure modes
- Cartesian fan-out: duplicates on both sides multiply, and downstream means are computed over duplicated rows.
- Silent averaging: if you
groupby("symbol").mean()after the merge, duplicate rows are averaged rather than flagged, so the signature looks plausible. - Version mismatch: joining versioned ENSG drops rows instead of adding them.
len(merged) == len(left)catches both directions.
Curated: · Written: · Reviewed:
Q&A-82A query counted rows in the call table as sample N. Why was N 8× too high, and what must DISTINCT apply to?
N was 8× too high because the query counted rows in the call table, and the call table's grain is one row per variant call, not one row per sample. If each sample has 8 calls, 1,200 samples produce 9,600 rows. The cohort size is COUNT(DISTINCT sample_id). DISTINCT must apply to the sample identifier column, sample_id, not to the row, not to variant_id, and not to the whole SELECT list.
-- wrong: counts calls
SELECT COUNT(*) FROM call_table WHERE ...; -- 9600
-- right: counts samples
SELECT COUNT(DISTINCT sample_id) FROM call_table WHERE ...; -- 1200
A 10-sample fixture with 100 calls each pins it: COUNT(*) = 1,000, COUNT(DISTINCT sample_id) = 10. If that test returns 1,000 for the second query, DISTINCT is on the wrong column or missing.
Alternatives and when they fit:
| Approach | Query shape | Use when | Cost |
|---|---|---|---|
COUNT(DISTINCT sample_id) | one aggregate | ad hoc, one number | hash distinct over the filtered rows; can spill on very large scans |
GROUP BY sample_id then count | SELECT COUNT(*) FROM (SELECT sample_id FROM call_table WHERE ... GROUP BY sample_id) t | you also need per-sample call counts, or filters like samples with ≥1 pathogenic call | same scan, but gives per-sample rows you can reuse |
| pre-aggregated cohort table | SELECT COUNT(*) FROM cohort_sample_counts WHERE ... | dashboards and power calculations run repeatedly | staleness; needs a refresh job and a rule for late-arriving calls |
The 8× factor is not a constant. It is the average calls per sample in that slice. Dividing by 8 is wrong because samples with 2 calls and samples with 40 calls both exist. Use DISTINCT.
Failure modes to check:
- DISTINCT on the wrong column.
COUNT(DISTINCT variant_id)counts distinct variants, not samples. If 8 samples share a variant, that variant contributes 1, not 8. - DISTINCT on multiple columns.
COUNT(DISTINCT sample_id, variant_id)counts distinct sample-variant pairs, which is the row count again. - NULL
sample_id.COUNT(DISTINCT sample_id)drops NULLs. If 5% of calls have NULLsample_id, N is low by those samples. Detect withSELECT COUNT(*) FROM call_table WHERE sample_id IS NULL;and fix upstream. sample_idnot unique per sample. If the samesample_idis reused across cohorts or assays, distinctsample_idundercounts. Use a composite key:COUNT(DISTINCT (sample_id, cohort_id))where the database supports it, or a surrogatesample_key. Detect withSELECT sample_id, COUNT(DISTINCT cohort_id) FROM call_table GROUP BY sample_id HAVING COUNT(DISTINCT cohort_id) > 1;- Join fan-out. If you join the call table to another table before counting, rows multiply. Count samples before joins, or verify join cardinality first.
A dashboard I saw showed n=9,600 for 1,200 samples; a rare-disease filter looked well powered and was not. The fix was COUNT(DISTINCT sample_id) and a unit test that 10 samples × 100 calls yields n=10.
Curated: · Written: · Reviewed:
Q&A-83You need the top transcript per gene by TPM. Why does ORDER BY TPM LIMIT 1 per gene in a loop fail at 60k transcripts, and which SQL pattern is right?
Core answer (first 90 seconds)
Assumptions: Postgres (or any engine with window functions), a transcripts(gene_id, transcript_id, tpm) table, ~60k rows across ~20k genes, and you want exactly one row per gene. The loop fails for two independent reasons, cost and correctness, and the right pattern is a single windowed query.
Cost: a per-gene loop is the N+1 query pattern. 20k genes means 20k statements, each with its own parse, plan, execute and network round trip. Even at 0.5 ms per round trip that is 10 s of pure latency; at 2 ms it is 40 s. If there is no index on gene_id, each statement is a sequential scan of all 60k rows, so the loop touches 20k × 60k ≈ 1.2 billion rows to return 20k. The window function scans the 60k rows once and sorts them once.
Correctness: ORDER BY tpm DESC LIMIT 1 is non-deterministic when TPM ties. TPM is a float; ties at 3 decimals are common. The engine returns whichever tied row it encounters first, and that depends on physical row order, which changes after VACUUM, updates, or a parallel plan. So the same query can pick a different isoform on a different day.
Right pattern
SELECT gene_id, transcript_id, tpm
FROM (
SELECT gene_id, transcript_id, tpm,
ROW_NUMBER() OVER (
PARTITION BY gene_id
ORDER BY tpm DESC, transcript_id
) AS rn
FROM transcripts
) ranked
WHERE rn = 1;
PARTITION BY gene_id restarts the numbering per gene; ORDER BY tpm DESC, transcript_id makes the winner deterministic because transcript_id is unique. One scan, one sort.
Alternatives
| Pattern | Cost at 60k | Deterministic? | When to use |
|---|---|---|---|
| Loop, one query per gene | 20k round trips; 1.2B rows without index | No, unless tie-break added | Never at this scale |
ROW_NUMBER() OVER (PARTITION BY ...) | One scan + sort of 60k | Yes with tie-break | Default |
DISTINCT ON (gene_id) (Postgres) | One index scan if (gene_id, tpm DESC) indexed | Yes with tie-break | Postgres, index present |
LATERAL join per gene | One scan + per-gene index probe | Yes with tie-break | When you need extra columns from a second table |
DISTINCT ON:
SELECT DISTINCT ON (gene_id) gene_id, transcript_id, tpm
FROM transcripts
ORDER BY gene_id, tpm DESC, transcript_id;
The ORDER BY must lead with gene_id.
Failure modes
- Ties: without a unique tie-break, results flip between runs. Detect by running the query twice and diffing, or by asserting
COUNT(*) = COUNT(DISTINCT gene_id). In one run-to-run diff, 400 genes changed which transcript was picked. - Missing index: the window query still sorts 60k rows, fine, but a loop without
(gene_id, tpm)is quadratic. ROW_NUMBERvsRANK:RANKgives ties the same number, soWHERE rn = 1can return multiple rows per gene. UseROW_NUMBERwhen you need exactly one.
Curated: · Written: · Reviewed:
Q&A-96You need mean coverage per sample and also each contig’s coverage. Why does GROUP BY sample alone lose contigs, and when is a window right?
The 60-second answer
GROUP BY sample_id doesn't delete contigs: it collapses every contig row for that sample into one output row. The contig identity is gone because it isn't in the GROUP BY key and isn't in the SELECT. You asked for one mean per sample and that's what you get; the per-contig numbers were consumed by the aggregate.
A 0× contig disappears for one of two reasons, and they need different fixes:
- Averaged away. It's still in the input, but its 0 is folded into the sample mean. With 24 contigs at 30× and one at 0×, the mean is (24×30 + 0)/25 = 28.8×: the missed chromosome is invisible in the output.
- Never in the input. If the base table is
coverage INNER JOIN contigs, or filteredWHERE cov > 0, a contig with no aligned reads has no row at all. No aggregation can bring back a row that was never there; you need aLEFT JOINfrom the contig list.
A window does the other thing. AVG(cov) OVER (PARTITION BY sample_id) computes the same sample mean but attaches it to every row instead of collapsing. Contig rows survive, each carrying the sample mean as an extra column.
Worked trace
coverage(sample_id, contig_id, cov)
S1 ctg1 30
S1 ctg2 0
S1 ctg3 60
SELECT sample_id, AVG(cov) FROM coverage GROUP BY sample_id → one row, S1, 30. ctg2 is gone.
SELECT contig_id, cov, AVG(cov) OVER (PARTITION BY sample_id) AS sample_mean FROM coverage → three rows, each with sample_mean = 30. ctg2 is visible at 0×.
When a window is right: you need both grains in the same result. Per-contig rows with the sample mean beside them, e.g. flagging contigs below 0.5× the sample mean. One pass, no self-join.
When GROUP BY is right: you only need the sample summary. GROUP BY can hash-aggregate and emit one row per sample; the window partitions and carries every row through, so on a 10M-row coverage table you get 10M output rows each holding the mean, and you'd need SELECT DISTINCT sample_id, sample_mean to get back to one row per sample. That is the real cost of "window plus a distinct."
One caveat that decides the choice: a window can't fix the missing-row case. If the 0× contig has no row, PARTITION BY sample_id never sees it either. Fix the join first, then pick the aggregation.
Curated: · Written: · Reviewed:
Q&A-101You've just been handed FASTQ files for a new project. Which QC signals do you check first, and what decision does each one change?
I run FastQC over every file and MultiQC to see them side by side, and I read it as a decision gate, not a report. Order matters: per-base quality and read length first, then adapter content, then duplication, then GC distribution and overrepresented sequences. Each one maps to a different action: trim, re-sequence, screen for contamination, or stop and call the wet lab.
| Signal | What I look for | Decision it changes |
|---|---|---|
| Per-base quality | Median dropping below Q20 in the last 10–20% of the read | Trim read tails (fastp/cutadapt), but only after checking whether the scores are binned |
| Adapter content | Any adapter sequence above ~0.1% | Adapter trimming is mandatory; also tells me the insert size is short |
| Duplication | FastQC duplication % read against library type | WGS: re-sequence or accept lower effective depth. RNA-seq: ignore, it is expression |
| GC distribution | Mode shifted from the theoretical curve | Contamination screen (FastQ Screen, Kraken2) before anything else |
| Overrepresented sequences | Top table | Adapter, rRNA, vector, or a genuinely dominant transcript |
| Per-base N content | Any spike | Instrument failure: re-sequence, do not trim |
The command I actually run:
fastp -i R1.fq.gz -I R2.fq.gz -o R1.clean.fq.gz -O R2.clean.fq.gz \
--detect_adapter_for_pe --cut_tail --cut_tail_mean_quality 20 \
--length_required 35 --json sample.json --html sample.html
Then the arithmetic that decides whether I re-sequence. A 30× human WGS target is 3.1 Gb × 30 = 93 Gbp. At 2×150 bp that is ~310M read pairs. If FastQC reports 25% duplication, effective depth is 310M × 0.75 × 300 bp / 3.1 Gb ≈ 22.5×, not 30×. That is the number I take to whoever ordered the sequencing, because it changes the answer from "proceed" to "top up the library".
Failure modes I have actually hit. NovaSeq instruments report binned quality scores: only Q2, Q12, Q23 and Q37 appear. A tail that "drops to Q12" may be a real drop or just the bin, and trimming at a Q20 threshold silently deletes every Q12 base. I plot the score distribution before choosing a threshold. Second, over-trimming RNA-seq shortens reads and biases expression estimates; adapter removal is the part that matters, quality trimming is optional. Third, deduplication is wrong on amplicon and targeted panels: the duplicates are the signal. Fourth, a GC shift in one sample out of forty is contamination; a shift in all forty is the prep, and trimming will not fix it.
What I do not do: treat FastQC's pass/warn/fail flags as gates. They are fixed thresholds tuned for genomic DNA, the duplication module flags levels that are routine for RNA-seq, so I read the shape of the curve, not the colour. And I do not start alignment until adapter content is resolved, because adapter read-through produces soft-clipped piles that look like real variants in a repeat.
Curated: · Written: · Reviewed: —
Q&A-87You hash both a k-mer and its reverse complement as different keys. What double-counting happens on a 2-strand genome, and what is the canonicalization rule?
Core answer. On a double-stranded genome, a k-mer and its reverse complement are the same physical feature read on opposite strands. Hash both as distinct keys and you count every non-palindromic k-mer twice: unique k-mer counts roughly double, the k-mer spectrum gains a mirrored duplicate, and any genome-size or coverage estimate built from unique k-mer counts comes out about 2× high. The canonicalization rule is to key on min(kmer, revcomp(kmer)) under a fixed total order (lexicographic on the 2-bit encoding is fine): one representative per reverse-complement pair.
Why it's ~2× and not exactly 2×. revcomp is an involution: revcomp(revcomp(x)) == x. So k-mers partition into orbits of size 1 (palindromes, where kmer == revcomp(kmer)) or size 2. Canonicalization keeps one per orbit. For odd k there are no palindromes: a self-reverse-complementary odd-length word would need a central base equal to its own complement, and no base is, so every orbit has size 2 and the inflation is exactly 2×. For even k, palindromes exist and inflation is slightly under 2×.
Worked trace. k=5, plus strand 5'-AACGT-3'. The minus strand at the same duplex reads 5'-ACGTT-3'.
| key | source | canonical? |
|---|---|---|
| AACGT | plus strand | yes (min) |
| ACGTT | minus strand | no |
revcomp(AACGT): complement → TTGCA, reverse → ACGTT. min("AACGT","ACGTT") = "AACGT". Two keys collapse to one.
Palindrome case, k=4: revcomp(ACGT) = ACGT (complement TGCA, reverse ACGT). Already one key; canonicalization is a no-op. That is why even k inflates by less than 2×.
Arithmetic (illustrative). Haploid genome G = 3.0 Gb, k = 21 (odd, so no palindromes). Unique canonical k-mers ≈ G ≈ 3.0e9. Hashing both strands ≈ 2G ≈ 6.0e9. A genome-size estimate from the non-canonical count lands ~2× high.
Alternatives, and when canonicalization is wrong.
- Canonical keys (min of the pair): halves the key space, makes counts strand-agnostic. Standard for de Bruijn assembly, k-mer spectra, genome-size estimation. Cost: you discard strand, and you compute a revcomp per k-mer.
- Both orientations plus a strand bit: keeps strand, doubles memory and key count. Use when strand is the signal.
- No canonicalization: correct only for strand-specific data.
Strand-specific RNA-seq is where canonicalization is wrong. In a dUTP/stranded protocol the read strand tells you which strand the transcript came from; collapsing the pair throws that away. Same for any analysis where strand asymmetry is the point.
Failure modes and detection.
- Genome-size estimate ~2× high → library judged undersequenced when it isn't; you over-sequence.
- k-mer spectrum shows a doubled, mirrored second peak.
- de Bruijn graph: every edge duplicated, graph roughly doubles, assembly slower and can sprout false branches.
- Detection: ratio of unique-kmer count to canonical count near 2 for odd k means you are hashing both strands. Check the spectrum for a mirror.
Curated: · Written: · Reviewed:
Q&A-86A de Bruijn graph of a tandem repeat is a cycle. Why is there no unique Eulerian traversal, and what extra evidence resolves copy number?
Assumptions: fixed k, a tandem array whose unit is longer than k so the unit has internal k-mers, and a de Bruijn graph built as a set of distinct k-mers. Under those, here is the answer.
Core. A de Bruijn graph stores each distinct k-mer once: it is a set, not a multiset. Every copy of the repeat unit contributes the same k-mers, so they collapse into a single cycle. The graph for an 11-copy array is identical to the graph for a 1-copy array. An Eulerian traversal of that graph walks the cycle once and emits one unit; it cannot emit 11, because each edge exists once. That is the non-uniqueness: the graph is consistent with any copy number, and nothing in it lets you choose. A directed graph has a unique Eulerian circuit only when every node has in-degree = out-degree = 1: a single cycle. Here the node where the loop meets the flanking path has out-degree 2 (continue around the loop, or exit into the flank), so even the ordering of the single lap relative to the flanks is a free choice, and an internally repetitive unit or flank k-mers shared with the unit add more valid orderings.
So the graph answers "what is the unit" and refuses to answer "how many."
Extra evidence. Copy number comes from depth or from molecules that span the array:
- k-mer depth. If unique k-mers sit at depth d and the repeat's k-mers sit at depth N·d, then N = repeat depth / unique depth. Worked example: 30× unique coverage, repeat k-mers at 330× → 330/30 = 11 copies.
- Spanning reads. HiFi or nanopore reads longer than the array cross it end to end and count the unit directly; a 22 kb array needs reads longer than 22 kb.
- Paired links. Read pairs with one mate in unique flank and one in the repeat, or mate-pair/optical-map distances, bracket the array length.
| Evidence | Copies |
|---|---|
| collapsed cycle | 1 |
| k-mer depth 330× / 30× | 11 |
| spanning HiFi read | 11 |
Failure mode. An assembler emits the 2 kb unit once for an 11-copy, 22 kb array; annotation then reports one copy of a toxin gene and loses ten. Detect it by plotting coverage: a spike at the repeat unit relative to the unique baseline, plus reads that span the array. If neither is available, report the copy number as unresolved rather than defaulting to 1.
Curated: · Written: · Reviewed:
Q&A-99An overlap graph of reads has 3 components. Why might that be 3 chromosomes or 3 unjoined repeats, and what extra edge would merge them wrongly?
Three components means three sets of reads with no overlap path joining them. That is either three real sequence units (three chromosomes, or a chromosome plus a plasmid and a mitochondrion) or one chromosome that the overlap graph failed to bridge across two repeat regions. The component count alone cannot tell these apart. What tells them apart is whether the gaps are repeat-sized and whether orthogonal long-range data links the flanks.
Start with the mechanism. An overlap edge is only usable for assembly if it is unique: the read has one best place to attach. A read lying entirely inside a repeat has equally good overlaps to every copy of that repeat, so a unique-overlap filter discards those edges. If the repeat is longer than a read, no single read spans it, and the graph ends up with no edge across it. Two such repeats cut one chromosome into three components.
The extra edge that merges them wrongly is a non-unique overlap you chose to keep. A read inside copy 1 of the repeat overlaps a read inside copy 2 at high identity. The overlap is real at the sequence level, but the two reads come from loci megabases apart. That one edge glues the components together and the assembler emits a chimeric contig.
Worked example (hypothetical figures). One 100 Mb chromosome carries two copies of a 20 kb repeat, at 20 Mb and 70 Mb, 99.8% identical. Reads are 15 kb HiFi. No read spans 20 kb, so the repeat contributes no unique edges and the graph has three components. Now keep non-unique overlaps: a read inside copy 1 overlaps a read inside copy 2 at 99.8% over 8 kb. That single edge merges the flanks into two components, and the contig it produces collapses the 50 Mb between the two copies.
To tell the two cases apart, use three signals:
| Signal | Three chromosomes | One chromosome, two repeat gaps |
|---|---|---|
| Local read depth | flat | ~2x at each repeat |
| Bridging read's overlaps | one locus | many equally good loci |
| Hi-C / Bionano | separate diagonal blocks | flanks linked, gap repeat-sized |
The trade-off is contiguity against correctness. Aggressive merging (keep non-unique overlaps, lower the identity threshold) buys longer contigs and pays in chimeras. Conservative splitting: unique overlaps only, identity at least 0.99, minimum overlap around 1 kb: buys trustworthy joins and pays in fragmentation. The deciding criterion is whether you have orthogonal long-range evidence. With Hi-C or Bionano, merge and validate each join against it. Without it, split and report the components as separate rather than guess.
Two failure modes follow. A false merge produces a chimeric contig; detect it by a Hi-C contact between different chromosomes, or by a coverage discontinuity at the join. A false split produces a fragmented assembly; detect it by the gap being repeat-sized and by long-range links that span it.
Curated: · Written: · Reviewed:
Q&A-12Why can NG50 fall when N50 rises, and which genome-size estimate are you allowed to use?
Core answer
N50 and NG50 differ only in the halfway target. N50 sorts contigs longest-first and finds the contig at which the cumulative length reaches half of the assembly's own total length. NG50 uses half of an expected genome size G instead. So N50 is self-referential; NG50 is anchored to an external number.
That difference explains the divergence. If the assembly shrinks (you filter out short contigs, or you have a less complete assembly) the assembly's total length L drops. N50's target L/2 drops with it, so you reach the halfway point with a larger contig: N50 rises. NG50's target G/2 does not move. With less total sequence, you must walk further down the sorted list to accumulate G/2, so the contig at which you cross is smaller: NG50 falls. A more fragmented or less complete assembly can therefore look better by N50 and worse by NG50.
The genome-size estimate you are allowed to use is one that is independent of the assembly you are evaluating. In practice that means a k-mer estimate from the reads (e.g., GenomeScope, KmerGenie), flow cytometry, or a closely related reference genome. You must state which one you used, apply the same estimate and the same contig filter to every assembly you compare, and never mix haploid and diploid sizes. Using the assembly's own total length as G is not allowed: it makes NG50 identical to N50 and hides missing sequence.
Worked example
Hypothetical figures, G = 100 Mb.
| Assembly | Contigs (Mb) | Total L (Mb) | N50 (Mb) | NG50 (Mb) |
|---|---|---|---|---|
| A | 45, 25, 20, 10 | 100 | 25 | 25 |
| B | 45, 20, 10 | 75 | 45 | 20 |
Trace for A: sorted 45, 25, 20, 10. Cumulative: 45, 70. L/2 = 50, so N50 = 25. G/2 = 50, so NG50 = 25. Trace for B: sorted 45, 20, 10. Cumulative: 45, 65. L/2 = 37.5, so N50 = 45. G/2 = 50, so NG50 = 20.
B has a higher N50 (45 vs 25) but a lower NG50 (20 vs 25). B is missing 25 Mb of sequence relative to A; N50 rewards the smaller total, NG50 penalizes it.
Failure modes and detection
- Using the assembly's own length as G: NG50 equals N50, so incompleteness is invisible. Detect by checking whether the reported G matches an independent estimate.
- Using a G from a different species or a different ploidy: NG50 is meaningless. Detect by requiring the estimator and its source in the methods.
- Mixing haploid and diploid sizes: a diploid size roughly doubles G, which halves NG50. Detect by stating ploidy and using the same convention across comparisons.
- Not applying the same contig filter: filtering short contigs changes L and the sorted list. Detect by reporting the filter (e.g., minimum contig length) alongside NG50.
What to report
Publish the genome-size estimator beside NG50, and include a sensitivity table at ±10% of G. If NG50 moves more than a few percent across that range, the assembly's ranking is not robust. The number is only as honest as the size you fed it.
Curated: · Written: · Reviewed:
Q&A-11Assembly A has contig N50 38 Mb and B has 19 Mb. Why might B still be the genome you should publish?
Publish B. N50 measures the contiguity of the span you submitted, not whether the joins are real. If A's 38 Mb N50 comes from misjoins, A is a longer wrong answer and B's 19 Mb is the honest one. The deciding evidence is completeness, contaminant span, and read support across joins, not the N50 number.
N50 is the length L such that half the assembled bases sit in contigs of length ≥ L. It is a length-weighted median, so the largest contigs dominate the cumulative sum. A false join fuses two contigs into one longer contig, and that merged length counts in full. Because N50 is read off the top of the sorted length distribution, a handful of joins near the median can move it a lot.
Trace it. Ten contigs, in Mb: 20, 15, 12, 10, 8, 6, 5, 4, 3, 2. Total 85 Mb, half is 42.5. Cumulative sum: 20, 35, 47. N50 = 12 Mb. Now a misjoin fuses the 12 and the 10 into a 22: 22, 20, 15, 8, 6, 5, 4, 3, 2. Cumulative: 22, 42, 57. N50 = 15 Mb. One false join, +3 Mb of N50, no new sequence. Spread that across a repeat-rich region and you get A's jump.
| Metric | A | B |
|---|---|---|
| Contig N50 | 38.2 Mb | 18.6 Mb |
| Total span | 2.84 Gb | 2.71 Gb |
| BUSCO complete (S+D) | 91% | 96.1% |
| BUSCO duplicated (D) | 8.0% | 1.2% |
| Misassemblies (QUAST) | 14 | 1 |
| Contaminant span | 130 Mb (4.6% of span) | 11 Mb (0.4% of span) |
BUSCO reports C (complete), S (single-copy), D (duplicated), F (fragmented), M (missing); S is the single-copy complete fraction. A's 8% duplicated BUSCOs are the tell: collapsed haplotigs or repeats inflate both span and N50 at once.
Verify before publishing. Map reads back across each join: a chimeric join shows a coverage discontinuity or a cluster of discordant pairs at the junction. Long reads or split reads that span the junction without a break confirm a real join; their absence is the failure signal. k-mer copy number flags collapsed repeats, which show elevated coverage over a region that should be single-copy. If a reference exists, QUAST misassembly counts settle it; de novo, read-backed junction support is the standard.
Publish B, and report A's N50 only alongside its misassembly count and contaminant span so the number cannot be read as quality.
Curated: · Written: · Reviewed:
Q&A-20Mean coverage is 40×. Why can half the genome still be unassembled, and which plot answers that?
The 60-second answer. Mean coverage is averaged over reads, not over the genome. A 40× mean says nothing about the per-base distribution, and the plot that answers "why is half unassembled" is the k-mer frequency histogram (the k-mer spectrum), built from the reads alone: no reference needed. It shows the true haploid coverage peak. If that peak sits at 20× while the headline is 40×, half your reads are not from the sequence you are trying to assemble. The per-base mapped-depth histogram is the second plot: it shows where the holes are, but only after you have a reference or an assembly.
Why uniform 40× cannot leave half the genome out. Lander-Waterman models coverage per base as Poisson with mean λ. At λ = 40, P(a base is covered by zero reads) = e^−40 ≈ 4×10^−18. Even at an effective 9× it is e^−9 ≈ 1.2×10^−4. Random sampling never explains a 50% dropout. So the mean must be misrepresenting the effective coverage, and the dropout must be systematic.
Two things break it:
- The mean is inflated. Reads from mitochondria, chloroplasts, plasmids or a contaminant are counted in the denominator. They raise the mean without covering the nuclear genome.
- Coverage is non-uniform. Collapsed repeats pull reads from many genomic copies onto one contig, so depth spikes there and drops to zero in the unique flanks. GC-extreme and low-complexity regions are unmappable: depth is zero by mappability, not by chance.
Reading the k-mer spectrum. Plot k-mer multiplicity on x (how many times a k-mer occurs across the reads) against the number of distinct k-mers on y. A clean haploid genome at 40× gives one Poisson-like peak at ~40 with a small error tail at 1–2. Three signatures matter:
- Peak at ~20, not 40 → a second copy of the genome (organelle, contaminant, or a haploid/diploid mix) is eating half the reads.
- Peak at 40 but a large low-multiplicity shoulder → sequencing error or heterozygosity.
- Peak at 40 with a heavy right tail → collapsed repeats.
The mapped-depth histogram is the complement: per-base depth across the reference, which is bimodal. A mode at 0× (the unassembled half) and a mode at 40×. It localises the holes; the k-mer spectrum explains them.
When a mean target is fine. For a small, low-repeat, clonal bacterial genome with clean libraries, mean ≈ effective coverage, Poisson gaps are negligible, and a 40× target is a reasonable stopping rule. It fails for large repeat-rich plant or animal genomes, GC extremes, and any library where organellar DNA is not depleted.
Worked example (hypothetical). A 40 Mb fungal genome sequenced to a headline 40× = 1.6 Gb of reads.
| Compartment | Genome-equivalents | Reads |
|---|---|---|
| mitochondrial (50 kb) | 22× | 880 Mb |
| nuclear unique | 9× | 360 Mb |
| repeats / other | 9× | 360 Mb |
| headline mean | 40× | 1.6 Gb |
Nuclear unique sits at 9×, and the assembly is missing 38% of BUSCO genes. Poisson at 9× predicts 0.01% uncovered, so the missing 38% is repeat collapse and unmappable sequence, not random gaps. The fix is to deplete organellar reads and set the target in filtered nuclear bases, not to sequence deeper.
Failure modes and detection. Organelle overrepresentation: map reads to the mito/plastid reference and check the fraction. Repeat collapse: depth spikes at collapsed loci. GC bias: depth plotted against GC content. Each is visible in the two histograms above.
Curated: · Written: · Reviewed:
Q&A-13Raising k from 21 to 77 fragmented a short-read assembly. What did you buy and what did you lose?
Raising k from 21 to 77 buys specificity and spends sensitivity. That is the whole trade.
What you buy. A 77-mer is far more likely to be unique in the genome than a 21-mer. Repeats shorter than k (most transposons, rRNA operons, common protein domains) stop creating alternative paths, so the de Bruijn graph loses bubbles and branches and the assembler can walk through them. You get longer, more confident contigs in the unique, well-covered parts of the genome.
What you lose. Every k-mer now has to be error-free and covered. If the per-base error rate is e, the fraction of k-mers carrying no error is roughly (1−e)^k. At e = 0.005 (about Q23) that is 0.90 at k=21 but 0.68 at k=77: a third of your 77-mers carry an error instead of a tenth. At 18× raw depth that leaves about 16× of usable k-mers at k=21 and about 12× at k=77. Worse, two reads must overlap by at least k−1 = 76 bases to be joined at all; with 150 bp reads that discards every short overlap, and short overlaps are exactly what bridges low-coverage gaps. The graph fragments.
k also has to stay below read length. At k=77 a 150 bp read yields 74 k-mers versus 130 at k=21, so each read contributes less redundancy. Use odd k so a k-mer cannot equal its own reverse complement.
What it looked like. A 4.5 Mb plasmid, 2×150 bp reads, 18× mean depth:
| k | Contigs | N50 | Circular |
|---|---|---|---|
| 33 | 3 | 4.4 Mb | 1 |
| 77 | 220 | 22 kb | 0 |
At k=33 the plasmid closed as one circular contig plus two short unplaced fragments. At k=77 the same reads shattered into 220 pieces with a 22 kb N50: the depth was too low to keep 77-mers error-free and connected.
What I would do instead. One k is always a compromise, which is why multi-k assemblers exist: build with a small k for connectivity, then extend and resolve with larger k. Before choosing, look at the k-mer spectrum: a clean single peak means you can push k up, a long error tail means you cannot. Then run a small grid of k and compare N50, BUSCO completeness, and misassembly counts on the same reads. Do not pick k by maximum N50 alone; N50 rewards collapsing repeats into false joins.
k is a resolution-versus-connectivity knob, not a quality score.
Curated: · Written: · Reviewed:
Q&A-14When do you pay for PacBio HiFi instead of cheaper raw long reads, and what does polishing not replace?
The 90-second answer
Assuming a de novo assembly or variant-calling project rather than a targeted assay: pay for HiFi when the question is base-level or haplotype-level and you cannot afford a polishing stage. Repeat-rich or polyploid assembly, SNV/indel calling, phasing. Skip it when the question is span (ultra-long ONT reads go past 100 kb where HiFi's N50 sits around 15–20 kb), when you only need large-SV structure, or when the budget forces a short-read polish anyway. HiFi costs more per gigabase and yields fewer gigabases per run; the premium buys per-read accuracy and the removal of a polishing stage.
Why the accuracy difference exists
HiFi is circular consensus: the SMRTbell insert is a circular template, the polymerase reads around it multiple times, and the consensus of those passes becomes the read. Errors are largely random per pass, so consensus drives per-base accuracy above Q30 (99.9%) at 15–20 kb. ONT simplex is one pass; R9.4.1 sits near Q12, and ONT's own figures put R10.4.1 simplex at roughly Q20 and duplex at roughly Q30. That narrows the gap, but duplex costs throughput, and HiFi still gives Q30 per read without a second pass.
The consequence: a HiFi assembly is usable without polishing, a noisy-read assembly is not. That matters because polishing is a base-level operation on a fixed contig graph.
What polishing does not replace
- Span. If no read crosses a repeat, the assembler collapses it to the length its reads support. Polishing edits bases inside that contig; it cannot add copies back.
- Structure. A misjoin between two contigs is a graph decision. Polishing leaves the join and makes it look confident.
- Phase. If the assembler merged homologs into one consensus, polishing keeps the chimera. Phase comes from HiFi plus trio binning or Hi-C, not from polishing.
- Missing sequence and contamination. Polishing cannot recover a region absent from the reads, and it will happily polish a contaminant contig.
- Systematic error. Polishing corrects random error well and correlated error poorly: a homopolymer or context-specific bias present in every read survives.
Worked example (hypothetical figures)
A genome with one 12 kb tandem array, true copy number 8.
| Library | N50 | Per-base | Assembly | Copies |
|---|---|---|---|---|
| ONT R9.4.1 simplex | 10 kb | ~Q12 | Flye + Medaka + Pilon | 5 |
| HiFi | 18 kb | Q30 | hifiasm, no polish | 8 |
No read in the ONT set spanned 12 kb of near-identical repeat, so the array collapsed to what the reads supported. Polishing took that contig to Q40: the more accurate assembly by QV, and the wrong one. The HiFi set had reads longer than the array, so the copy number came out right at lower QV.
How I'd decide, and what I'd check
Pay for HiFi when you need SNVs/indels from the assembly, the genome is repeat-rich or polyploid, you need phase, or you want methylation calls from the same reads via kinetic signal. Don't when you need span past ~50 kb (use ultra-long ONT, or hybrid ONT + HiFi), when you only need large-SV presence/absence, or when the sample is degraded to a few kb, then neither platform invents span.
Before trusting either assembly, plot read N50 against the repeat-length spectrum of the genome, and report assembly QV as a column separate from structure. Structure checks that don't depend on the assembly: optical map, Hi-C contact map, and depth. A collapsed array shows as a contig shorter than the estimated array at roughly double the expected coverage.
Curated: · Written: · Reviewed:
Q&A-18Three rounds of polishing raised QV from 28 to 42. Why might the assembly still be wrong, and when do you stop?
Direct answer. QV is a per-base accuracy metric. Polishing (Racon, Medaka, Pilon, Arrow) walks aligned reads over the consensus and fixes substitutions and small indels. It cannot move a join. So QV 28 → 42 tells you the bases got cleaner; it says nothing about whether the two contigs on either side of a junction belong together. Stop when structural evidence (misassembly count, discordant read pairs, Hi-C linkage) stops improving, not when QV does.
Assumptions. QV here is k-mer based (Merqury) or reference based; either way it is computed from local sequence, not from long-range linkage. The polishing reads are individually shorter than the distance between the errors you actually care about.
Why QV rises on a wrong assembly. A chimeric join is usually two real pieces of sequence glued at a point no single read spans. Every k-mer on either side exists in the read set, so Merqury finds them and assigns high QV. Reads align across the join because they align to one side or the other; the few bases at the seam get polished toward whatever the local pileup says, which is often nothing. The result is a false junction that reads as high-quality sequence. Collapsed repeats behave the same way: the repeat's k-mers are all present in reads, so a two-copy repeat collapsed into one is not penalized, and QV stays high while coverage doubles. Polishing with reads from a paralogous region can also pull a base toward the wrong copy.
Failure mode and detection. The concrete failure: three rounds of polishing a chimeric join to Q42, and the annotation fused 14 genes across the false junction. Detection is not QV. It is the Hi-C contact map showing off-diagonal signal where the join is, discordant read pairs clustering at the seam, or the assembly graph showing a low-support edge. Without one of those, you cannot tell a correct join from a polished wrong one.
When to stop. Stop when the structural metrics plateau or move the wrong way. In the table below, QV climbs 28 → 42 while misassemblies stay at 14 and discordant pairs rise: that is a stop, not a reason for a fourth round. Also stop when the marginal QV gain per round drops below about 1 point against the compute cost, and when you have no spanning evidence left to add.
| Round | QV | Error rate/base | Discordant pairs | Misassemblies |
|---|---|---|---|---|
| 0 | 28 | 1.6e-3 | 41,000 | 14 |
| 1 | 35 | 3.2e-4 | 47,000 | 14 |
| 2 | 40 | 1.0e-4 | 54,000 | 14 |
| 3 | 42 | 6.3e-5 | 58,000 | 14 |
Figures are illustrative. Arithmetic: QV 28 is 10^-2.8 ≈ 1.6e-3, one error per ~630 bases. QV 42 is 10^-4.2 ≈ 6.3e-5, one per ~15,800 bases. Polishing cut base errors ~25x and left the 14 misassemblies untouched.
A high-QV consensus can sit on a broken join. Judge the join with linkage, the bases with QV.
Curated: · Written: · Reviewed:
Q&A-15A diploid assembly has 1.12× expected span. How do you tell collapsed heterozygosity from duplicated haplotigs, and what does aggressive purge destroy?
Pin the number first: 1.12× the haploid genome size means the assembly holds 12% more sequence than a single haplotype. That 12% is either a second haplotype that should be phased or purged, or real duplicated sequence that must stay. Coverage alone cannot tell you which, because a collapsed heterozygous region and a real segmental duplication both read at twice the haploid depth. The discriminator is allelic versus paralogous: a haplotig is the same locus carrying heterozygous variants at roughly 50% allele fraction, while a real duplication is a different locus whose copies differ by paralogous, often fixed, differences and are joined by read pairs spanning the duplication junction.
Mechanism. Let D be total read depth over the diploid genome; each haplotype is sequenced at D/2. A haplotig in the alternate assembly sits at D/2: identical to any single-copy haplotype, so its coverage is uninformative on its own. A collapsed primary region sits at D, because both haplotypes are stacked onto one sequence. A real segmental duplication present once per haplotype also totals D at the locus, D/2 per copy. That is why the two classes collide on a coverage histogram.
| Class | Depth (D = total diploid depth) | How to confirm |
|---|---|---|
| Haplotig (alternate) | D/2 | aligns to a primary locus at the same depth; heterozygous k-mers |
| Collapsed primary region | D | both haplotypes' alleles at ~50%; no junction-spanning pairs |
| Real segmental duplication | D/2 per copy, D at the locus | read pairs span the duplication junction; paralogous, not allelic, differences |
How to tell them apart in practice:
- Align alternate contigs to the primary. A haplotig aligns to a primary locus at the same depth (D/2) and carries heterozygous k-mers. A real duplication is homozygous and appears only in the primary.
- Look at allele fraction across the region. Collapse gives heterozygous sites at ~50% throughout; a duplicon gives paralogous differences, frequently fixed between copies.
- Use read pairs or long reads spanning the duplication junction: present only for real duplicons.
- Trio or Hi-C phasing is strong evidence but not a guarantee: phasing errors and unphased regions exist.
What aggressive purge destroys. purge_dups and purge_haplotigs classify contigs by coverage. Real duplicons at D/2 fall in the same peak as haplotigs and get deleted. Aggressive purging also removes one haplotype of genuinely heterozygous regions, and it cannot repair a collapse: the merge already happened, so purging only shrinks the assembly. Example: a purge_dups run removed 64 Mb of real segmental duplication that sat at haploid coverage; a downstream CNV caller then missed nine disease-relevant copies. Detect this by checking whether purged contigs have read pairs spanning their junctions, and by comparing against a trio or the alternate assembly before accepting the purge.
So: state the representation goal first (one haplotype, or both), then decide per region using allelic-versus-paralogous evidence, not coverage alone.
Curated: · Written: · Reviewed:
Q&A-16BUSCO reports 97% complete. What has that not measured, and which lineage dataset are you even using?
What 97% actually counts
BUSCO scores your assembly against a fixed set of near-universal single-copy orthologs drawn from one lineage. C:97.0% means 97% of that set was recovered as complete: full-length, in-frame, matching the expected protein. Everything else in the genome sits outside the measurement.
So the first question back is: which lineage dataset, and which version? The version string is printed in the short summary and belongs in the methods. embryophyta_odb10 and eukaryota_odb10 are not interchangeable, and the set size sets the resolution:
| Lineage set | BUSCOs (n) | 1 BUSCO | 97% means |
|---|---|---|---|
| eukaryota_odb10 | 255 | 0.39% | ~8 not complete |
| embryophyta_odb10 | 1,614 | 0.062% | ~48 not complete |
97% on a 255-gene set is a coarse pass; 97% on 1,614 is a tighter claim. Quote the lineage, the version, and the C/S/D/F/M breakdown, not the headline.
What it does not measure
- Noncoding and repeat content. BUSCO only sees conserved coding genes. In a large plant genome, repeats can be most of the assembly and BUSCO is blind to them.
- Copy number and ploidy. Duplicated (D) BUSCOs are a hint, not a measurement: D above roughly 10% usually means uncollapsed haplotigs or a real whole-genome duplication, and you separate those with read depth or a k-mer spectrum, not with BUSCO.
- Order, orientation, chromosome-scale structure. A shattered assembly and a chromosome-scale one can score identically.
- Base-level accuracy. A collapsed or chimeric assembly still contains the conserved genes once.
- Lineage-specific and fast-evolving genes. They are not in the set by construction.
- Contamination. A contaminant from a related organism can add BUSCOs and inflate the score.
The failure mode, worked
An assembly scores C:98.0%[S:97.5%,D:0.5%] on embryophyta_odb10: 1,582 of 1,614. The same reads give 88% k-mer completeness (Merqury, k=21). The trait locus the project funded is absent from the assembly.
| Check | Result |
|---|---|
| BUSCO C (embryophyta_odb10, n=1,614) | 98.0% (1,582) |
| k-mer completeness (k=21) | 88% |
| trait locus in assembly | absent |
The 32 missing BUSCOs are 2% of a gene set. The 12% of missing k-mers is ~180 Mb of repeat-rich pericentromere: exactly where the trait locus sits. BUSCO never looked there, so it reported near-complete.
What I'd require before calling it complete
BUSCO table with the lineage and version named, plus k-mer completeness from the same reads, plus a known-locus recovery list: the trait gene, a marker panel, whatever the project actually needs. If BUSCO is high and k-mer completeness is low, the missing sequence is repeat-rich and non-genic; go find it. If D is high, check for haplotigs before claiming duplication.
Command for the record: busco -i asm.fasta -l embryophyta_odb10 -m genome -c 16 -o plant_busco (BUSCO v5; genome mode predicts genes with MetaEuk before scoring).
Curated: · Written: · Reviewed:
Q&A-19The assembler flagged a contig as circular. What independent evidence is required before you rotate and publish it?
Short answer: I need three independent lines of evidence agreeing on the same join. A unique end overlap, reads that physically span the join, and coverage that stays flat across it. If any one is missing, I publish linear with a gap rather than rotate and call it circular.
Assumptions: a bacterial chromosome assembled from short or HiFi reads, and the assembler's circular flag is a hypothesis about the graph, not a measurement. Rotation is a coordinate convention: it changes where position 1 sits, not the sequence.
The three checks, and what each rules out:
| Check | What it rules out | Pass looks like |
|---|---|---|
| Unique overlap at the ends | Repeat-driven false join | Overlap longer than any repeat in the genome, matching at both ends |
| Reads spanning the join | Assembly artefact with no molecule behind it | Several reads cross the seam with no soft-clip there |
| Coverage across the join | Collapsed repeat or duplicated segment | Depth at the join within roughly ±20% of the chromosome median |
Then the negative control: run the same closure test on a known-linear contig or a scrambled end. If that also "closes", the test is measuring repeats, not topology.
If all three pass, I rotate to the origin of replication, dnaA in most bacteria, as a convention, remove exactly the duplicated overlap bases so no base appears twice, and record the origin choice in the release notes. Sequence content does not change.
When I would not publish as circular:
- Overlap present but shorter than a repeat in the genome: publish linear with a gap, or reassemble.
- Spanning reads absent but coverage flat: likely a real join, but reassemble with long reads before claiming it.
- Coverage dip at the join: collapsed repeat; publish linear with the gap marked.
Worked example of the failure this guards against: a 2.1 kb overlap at the ends of a 5.2 Mb chromosome looked like clean closure, but the overlap matched a repeat, zero reads spanned the join, and coverage dropped to 0.3× there. Published as circular, it duplicated the origin region and shifted every downstream coordinate. The table above would have caught it on the spanning-read row alone.
So the release package carries the spanning-read count, the overlap identity, and the coverage across the join, not just the flag.
Curated: · Written: · Reviewed:
Q&A-17A contig BLAST-hits a vector at 99%. Why is that still not enough to delete it, and what bundle of evidence is?
Core answer (first 90 seconds)
A 99% BLAST hit is a fact about one alignment, not about the contig. It says the aligned region resembles a vector; it says nothing about what fraction of the contig that is, whether the contig's read coverage matches the rest of the genome, or whether the contig is physically linked to host sequence. Those are the things that decide deletion.
So: identity alone never justifies deletion. I delete only when several independent signals agree, and otherwise I quarantine the contig with a written reason and keep it in the assembly.
The evidence bundle
Six signals, none sufficient alone:
| Signal | Vector-like | Genuine host sequence |
|---|---|---|
| Alignment span | covers most of the contig | small fraction of a long contig |
| Coverage vs genome mean | discordant | concordant |
| GC / k-mer composition | outlier | inside genome distribution |
| Read pairs linking to host | none | pairs span the junction |
| Blanks / negative controls | present | absent |
| Other samples in the run | same contig recurs | sample-specific |
BLAST's default output hides the two numbers that matter most: alignment length and query coverage. Ask for them: blastn -query contigs.fa -db UniVec -outfmt "6 qseqid sseqid pident length qcovs evalue". A 99% hit over 40 bp is noise; a 99% hit over 1,180 bp covering 10% of a 12 kb contig is a lead.
Worked example (hypothetical)
4.8 Mb isolate, 60× Illumina. Two contigs hit the pUC19 bla region at 99%:
- Contig A, 12 kb: 1,180 bp aligned = 9.8% of the contig. Coverage 3×. GC 51% against a 61% genome. Present in the extraction blank. No read pair links it to host sequence. → delete.
- Contig B, 48 kb: same 1,180 bp at 99.0% = 2.5% of the contig. Coverage 58×. GC 60%. Fourteen read pairs span from the bla region into flanking sequence unique to the genome. Absent from blanks. → keep; it is a chromosomal beta-lactamase, which is common in environmental bacteria.
Both have the same identity and the same alignment length. Identity cannot separate them; coverage, composition, linkage and controls do.
Failure modes
- Deleting B loses a real gene. Detected by coverage concordance and spanning read pairs.
- Keeping A inflates the assembly and adds spurious ORFs. Detected by coverage discontinuity and composition outlier.
- The dangerous case is a chimeric contig where the vector segment bridges two unrelated host regions. Detected by the two flanks having different coverage or GC and no read pair spanning the full junction. The fix is read-level trimming and reassembly, not contig deletion.
- Batch contamination: the same vector contig in every sample of a run. Detected by blanks and cross-sample comparison.
- In metagenomes the same rule applies with one more candidate explanation: the sequence may be a genuine plasmid or prophage that shares backbone modules with a cloning vector.
What I would actually do
Trim reads against UniVec before assembly: cheaper, and it prevents the chimera rather than patching it. Then screen contigs, and delete only when at least four of the six signals agree. Everything else goes to a quarantine table: contig, length, best hit, %id, alignment length, qcovs, coverage, GC, spanning pairs, blank result, decision, operator. That table is the evidence; the BLAST line is the clue.
Curated: · Written: · Reviewed:
Q&A-22You convert a VCF SNP and a VCF insertion to BED. What off-by-one happens if you treat both like a 0-based SNP?
Core answer
VCF POS is 1-based and points at the first base of REF. BED is 0-based half-open, so a single reference base at 1-based position p is [p-1, p).
A SNP consumes exactly one reference base. POS 100, REF=A, ALT=G → BED [99, 100).
An insertion consumes zero reference bases. VCF 4.3 requires an anchor base for insertions: REF is the base immediately before the insertion, and ALT is that anchor plus the inserted sequence. So POS 100, REF=A, ALT=ATTT means "insert TTT between base 100 and base 101." The inserted bases occupy no reference interval, so the correct BED is a zero-length interval at the insertion point: [100, 100).
Apply the SNP rule to that insertion and you get [99, 100): shifted one base to the left, and width 1 instead of 0. You have described the anchor base, not the insertion. That is the off-by-one.
Worked example
Toy reference, 1-based: 1:A 2:C 3:G 4:T 5:A 6:C 7:G 8:T 9:A 10:C.
| Record | VCF | Correct BED | SNP rule applied |
|---|---|---|---|
| SNP | POS 5, REF=A, ALT=T | [4,5) | [4,5): same |
| insertion | POS 5, REF=A, ALT=AGG | [5,5) | [4,5): wrong |
The insertion adds GG between base 5 (A) and base 6 (C). Its BED point is 5. The SNP rule puts it at [4,5), covering base 5.
Why it matters
Interval arithmetic, half-open: a feature BED [2,5) (bases 3–5) does not contain the point 5, so the correct insertion does not overlap it. The SNP-style [4,5) does overlap it: a false positive. A feature starting at BED 5 (base 6) contains the point 5 but not [4,5): a false negative. Both errors are one base wide, which is the width of a splice site or a regulatory boundary.
Trade-offs
Three representations are defensible, and the right one depends on the consumer:
- Zero-length point
[POS, POS): biologically exact, correct for point-in-interval queries. Some tools drop or mishandle zero-length BED features, so confirm your intersection tool's behavior before relying on it. - Anchor-base interval
[POS-1, POS): what the SNP rule produces. Only correct if you deliberately want the anchor base. - Whole-record span including the anchor and the following base,
[POS-1, POS+1): non-empty, safe for tools that reject zero-length features, but it inflates the footprint by two bases and can create false overlaps with features adjacent to the insertion.
For a SNP there is no choice: one base, [POS-1, POS).
One edge case: at position 1 of a contig, VCF requires the base after the event as the anchor instead of the base before, so the insertion point is [0, 0).
Failure mode
A pipeline applies the SNP rule to every record. SNPs are fine; insertions shift one base left and gain a base of width. Overlaps against ClinVar or a gene model then miss or invent hits at boundaries. Detect it with round-trip fixtures: convert a SNP, an insertion and a deletion to BED and back, and assert the coordinates.
Curated: · Written: · Reviewed:
Q&A-23FILTER=. and FILTER=PASS look equally “fine” in a spreadsheet. What is the difference, and which one may you treat as passing?
What I'd say first
PASS and "." are not two spellings of the same thing. PASS is an assertion: the producing pipeline ran a declared set of filters over this record and it survived all of them. "." is the absence of that assertion: no filter was applied, so the record's status is unknown. You may treat PASS as passing, but only after you've checked which filters the producer declared. You must never treat "." as passing.
Why the spec draws that line
FILTER is the 7th column. VCF 4.2/4.3 reserve PASS for "this position has passed all filters" and use "." as the missing-value placeholder for the field. The spec's own sentence: "if this field is set to PASS or '.', no filters have failed": is loose enough that people read "." as benign, but the operational convention in GATK and most pipelines is that "." means filtering was not applied. Either way, "." is not an assertion that a filter set ran.
Every other value is a semicolon-separated list of filter IDs that failed, and each ID must have a ##FILTER=<ID=...> header line describing it. The header is the contract: it tells you what PASS actually means for this file.
That's the point. PASS is a claim about a filter set, not a claim about truth. A pipeline that declares one trivial filter (say, only dropping records with QUAL of 0) will stamp PASS on garbage. The string is only as strong as the ##FILTER lines behind it.
The judgement
Treat PASS as passing when two things hold: the header declares a filter set you can read and accept, and the producer is one whose filters you trust. If the header declares nothing but PASS, or the filter set is a stub, PASS carries almost no information and you should re-filter yourself.
"." never passes. It means the record was never evaluated. In a merged or annotated VCF, "." often appears because a tool wrote records without carrying FILTER forward: the record may be perfectly good, but you don't know, and "don't know" is not "pass".
| FILTER | Meaning | Treat as passing? |
|---|---|---|
| PASS | declared filters applied, all passed | yes, if the header's filter set is one you accept |
| q30 | failed the named filter q30 | no |
| . | no filters applied | no: unknown, re-filter or quarantine |
Failure mode
I've watched an aggregator normalise "." to PASS while loading 2.4 million sites from an unfiltered gVCF dump. Downstream, a diagnostic panel pulled 310 of those unfiltered records as clinical candidates. Nothing flagged it, because after the rewrite the column looked clean. The bug is invisible in a spreadsheet and invisible in a PASS count.
Detection is cheap: count FILTER values per file and diff against the producer's expectation.
bcftools view -H input.vcf | awk -F'\t' '{print $7}' | sort | uniq -c
bcftools view -H drops the header and prints one record per line, tab-separated; awk takes field 7, the FILTER column; sort | uniq -c tallies. On a file that should be fully filtered you'd expect something like:
1200000 PASS
340000 .
2100 q30
(illustrative counts). 340,000 dots on a file whose contract says "filtered" is the alarm. Pair it with grep '^##FILTER' input.vcf to see what PASS was supposed to mean.
What I'd do
Validate FILTER IDs against the header at ingest. Reject or quarantine files containing "." when the contract required filtering. Never coerce missing filter status to PASS. Keep the raw calls so the decision is auditable.
Curated: · Written: · Reviewed:
Q&A-24A sample is 0/2 on a record with ALT=C,T. Which allele is 2, and what breaks if you split the record without rewriting GT?
Allele 2 is T: the second ALT. GT is a pointer into the allele list [REF, ALT1, ALT2, ...], so 0/2 means the sample carries REF (A) and the second ALT (T): genotype A/T. It is an index, not a count and not a copy number.
If you split the record into two single-ALT rows without rewriting GT, the T row keeps GT=0/2 while its allele list is now [A, T]: only indices 0 and 1 exist. Index 2 is out of range. That is an invalid record: strict readers (bcftools, vcf-validator) error or drop it; lenient ones may resolve the missing index to unknown and silently emit A/., or crash. The genotype is wrong or gone, and nothing in the row says so.
The arrays break the same way. Take a worked example (figures hypothetical):
| Field | Value | Number= | Meaning |
|---|---|---|---|
| GT | 0/2 | 1 | A/T |
| AD | 12,3,40 | R (one per allele) | A=12, C=3, T=40 |
| PL | 0,80,255,90,255,255 | G | 6 values for 3 alleles |
AD is Number=R: one value per allele including REF. Copy 12,3,40 onto the split T row, which has two alleles, and you have three values for a Number=R=2 field: malformed. A tool that indexes AD[GT] reads AD[2]=40 only while the array is intact; if the splitter truncates or the reader clamps, the T allele's depth is read as 3, the C count. PL is Number=G: 6 values for a triallelic diploid site, 3 for biallelic, so the same copy is malformed too. Any Number=A field (e.g. a per-ALT annotation) has the identical problem.
Correct split. The T row becomes GT=0/1, AD=12,40; the C row becomes GT=0/0, AD=12,3 (use ./. if your pipeline treats non-carriage as missing). bcftools norm -m- does exactly this, it rewrites GT and subsets Number=A/R/G fields, so use it rather than copying rows; confirm the behaviour against your installed version's man page.
Detection. The invariant to assert per record is max(GT index) < 1 + len(ALT). Run it after any split, and round-trip split/join on a 3-allele site with known AD and PL: the rejoined record must reproduce the original arrays. A split that leaves GT untouched passes a naive row count and fails this check, which is why the bug reaches downstream callers as a silent mis-call rather than an error.
Curated: · Written: · Reviewed:
Q&A-21Two VCFs share chr1:12345. Why might they still name different loci, and what do you compare before merging?
A coordinate is a pointer into a specific reference sequence, so chr1:12345 in two VCFs names the same locus only if both files agree on the reference accession, the contig dictionary, and the allele representation. Four things break that.
1. Different reference build. GRCh37 and GRCh38 chr1:12345 are different bases. The same number points at different sequence. Different accessions of one build (hs37d5 vs GRCh37) also differ in decoy and alt contigs.
2. Contig naming. chr1 vs 1 vs NC_000001.11 is the same sequence under a different ID. bcftools merge keys on contig ID, so it either errors or treats them as separate contigs. chrM vs MT is the same trap.
3. Indel representation. Left-alignment and parsimony mean a deletion inside a repeat has several equivalent POS/REF/ALT spellings. Two callers can emit different ones for the identical haplotype.
4. Multiallelic vs decomposed. One file has chr1:12345 A>G,T; the other has two records, A>G and A>T. Same locus, different record structure.
| Cause | Same coordinate, different locus because | Detect with |
|---|---|---|
| Build / accession | different underlying base | ##reference, md5 in ##contig |
| Contig naming | different contig ID | ##contig ID list |
| Indel not normalized | equivalent but different POS/REF/ALT | bcftools norm -f |
| Multiallelic split | one record vs many | bcftools norm -m |
Worked example. Reference chr1:100-107 is AAAAAAGC. The sample deletes one A, leaving five. Caller A right-aligns: POS=104, REF=AA, ALT=A (ref 104-105 is AA, sample has one A there). Caller B left-aligns: POS=100, REF=AA, ALT=A. Both spell the same five-A haplotype. bcftools norm -f ref.fa shifts both to POS=100.
What to compare before merging.
- Contig dictionary:
bcftools view -h a.vcf | grep '^##contig'and diff against the other file. Compare ID,length, andmd5(the checksum of the uppercased sequence). A mismatchedmd5means different sequence: stop. - REF against the destination FASTA:
bcftools norm -f ref.fa -c eerrors on any REF mismatch;-c wonly warns. - Normalize both sides:
bcftools norm -f ref.fa -m -any -Oz -o norm.vcf.gzleft-aligns and splits multiallelics. - If only naming differs, rename:
bcftools annotate --rename-chrs map.txt. - Liftover with CrossMap or Picard LiftoverVcf, then read the reject file and the REF-mismatch count.
Trade-offs. When builds differ, liftover is lossy: it drops variants in assembly gaps and can strand-flip indels, so re-aligning reads and re-calling in the target build is more work but correct. If you hold gVCFs, joint-genotype with GATK GenomicsDBImport instead of merging called VCFs: merging calls gives a union with no shared genotype model and cannot recover a variant one caller missed. Merging is the wrong move when the two call sets used different reference builds or different calling models; you want one callset, not a stitched one.
Failure mode. A merge that skips REF checking silently invents variants: a record whose REF does not match the destination FASTA is a variant that does not exist. Detect it by running bcftools norm -c e before the merge and requiring zero errors.
Curated: · Written: · Reviewed:
Q&A-25Two VCFs disagree on a homopolymer indel by 6 bp. Why might both be the same haplotype, and how do you compare them?
Short answer: they probably are the same haplotype. A VCF record is a spelling of a sequence change, not the change itself, and inside a repeat the same net deletion can be anchored at any offset in the run. A 6 bp POS gap is the length of the repeat, not a biological difference. Confirm it by left-aligning and trimming both records against the same reference, then comparing haplotypes rather than strings.
Assumptions I'd state up front: both files were called against the same reference build (same FASTA, same contig naming), both records are simple indels (not symbolic or spanning), and the sample is the same. If the reference builds differ, nothing below applies until that is fixed.
Why the spellings differ. VCF requires POS to be the position of the first base of REF, and REF to include one anchor base before the event. In a homopolymer run of length L, a single-base deletion has L-1 valid representations: anchor at the leftmost base, the rightmost, or anywhere between. All of them remove one base from the run and leave the same sequence. Left-alignment (shifting the record as far left as the repeat allows) plus trimming (removing shared prefix/suffix bases) collapses them to one canonical form.
Worked example. Reference positions 100–107 are AAAAAAAA (8 A's). A deletion of one A:
| Representation | POS | REF | ALT | Resulting run |
|---|---|---|---|---|
| left-aligned | 100 | AA | A | 7 A's (100–106) |
| right-aligned | 106 | AA | A | 7 A's (100–106) |
POS differs by 6; the haplotype is identical. String equality calls these different records.
How to compare. Two steps:
- Normalize both VCFs against the pinned FASTA:
bcftools norm -f ref.fa in.vcf -Oz -o out.vcf.gz. This left-aligns and trims. After this, most equivalent indels become byte-identical. - Compare with a haplotype-aware tool, not
diff.vcfeval(GA4GH benchmarking) builds the sample haplotype from each VCF and compares sequence, so residual representation differences don't count as errors. Restrict to confident regions so low-complexity or low-coverage areas don't dominate.
Failure mode. Comparing VCFs by string equality (or by naive POS/allele matching) inflates both false positives and false negatives in repeats. I've seen a truth-set evaluation count over a thousand homopolymer indels as false purely because POS differed by 1–8 bp; precision looked roughly 11 points worse than the normalized, haplotype-aware comparison. The metric was wrong, not the calls.
Decision criterion. If, after normalization against the same reference, the two records produce the same haplotype in the region, confirmed by vcfeval or by reconstructing the sequence, treat them as the same variant. If they still differ after normalization, the difference is real and worth investigating: different reference, different sample, or a genuine genotype disagreement.
Curated: · Written: · Reviewed:
Q&A-26A sample has no row at a site. How do you tell “called reference” from “no coverage,” and why does joint genotyping need the difference?
Short answer. A called reference site is one where the sample has a gVCF reference-confidence block covering that position, with depth and genotype likelihoods that support 0/0. No coverage is a block with DP=0, or no block at all: the sample was never observed, so the genotype is ./.. Absence of a row in a sites-only VCF is neither: it carries no per-sample evidence, so you cannot tell the two apart from it. Joint genotyping needs the distinction because it re-genotypes every sample at every cohort site; given only variant rows it must drop the sample or assume hom-ref, and assuming hom-ref invents genotypes.
Mechanism. HaplotypeCaller with -ERC GVCF writes, per interval, a record carrying the <NON_REF> symbolic allele and per-sample PLs for {reference, any non-reference}, plus DP/GQ/MIN_DP. GenomicsDBImport stores those per-sample records; GenotypeGVCFs, at each site discovered across the cohort, reads each sample's record and computes a genotype. Sample B with a block at DP=28 and PL favoring reference → 0/0. Sample B with no record covering the site, or DP=0 → ./.. The blocks matter, not just the variant rows, because the cohort's allele set grows as samples are added, a site that was a single-sample variant becomes multi-allelic, and each sample's stored likelihoods let the caller re-evaluate its genotype against that expanded allele set. That re-evaluation only works if the reference evidence was carried through.
When missing-as-0/0 is fine, and when it isn't. If you only need variant presence (sites-only QC, PCA, counting alt alleles for a burden test) collapsing missing to 0/0 changes little, because you never ask whether a sample is truly hom-ref. It is wrong wherever a false 0/0 hides a second hit: Mendelian consistency checks, de novo detection, compound-het discovery. The cost of keeping the distinction is size: gVCFs carry a block per interval and are far larger than sites-only VCFs, so you pay storage and import time.
Failure mode. A rare-disease pipeline treated missing rows as homozygous reference across 96 exomes; 14 compound-heterozygote candidates sat in uncovered exons and were dropped. Detection: for every 0/0 that feeds a Mendelian or compound-het analysis, require a gVCF block or DP>0 at that site, and flag any 0/0 where coverage is below your threshold.
| What you see | Meaning | Genotype |
|---|---|---|
| gVCF block, DP=32, PL favors ref | observed, reference-like | 0/0 |
| gVCF block, DP=0 | interval observed, no reads | ./. |
| no row in sites-only VCF | no per-sample evidence | unknown, not 0/0 |
Illustrative arithmetic: if 3% of exons fall below usable depth, a compound het needs both hits covered, so 0.97² ≈ 0.94. Roughly 6% of true compound hets are missed before any other filter. That is the size of the error you accept when you call missing as 0/0.
Curated: · Written: · Reviewed:
Q&A-28A base is Q40 and the read is MAPQ 0. What are you allowed to believe, and how should a caller treat it?
Assumptions first: Illumina short reads, BWA-MEM, diploid germline calling, Q40 is Phred-scaled base quality, MAPQ is the SAM mapping quality. Change any of those (long reads, a graph aligner) and the numbers move, but the logic holds.
The answer: You may believe the base call. You may not believe it is at that locus. Q40 means P(base wrong) = 10⁻⁴. MAPQ 0 means P(placement wrong) ≈ 1. Those multiply: the chance the base is correct and at the right coordinate is ~0. The Q40 is real confidence about the wrong question.
So a caller should not emit a genotype from a MAPQ 0 pileup by default. Treat it as unusable at that locus, not as weak evidence.
Why MAPQ 0 is special: MAPQ = −10·log₁₀ P(wrong placement). BWA-MEM assigns 0 when a read has multiple equally-scoring alignments, a multi-mapper, and caps unique reads at 60. Bowtie2 uses 0–42, with 255 meaning "unavailable". So 0 is not "low"; it is the aligner declining to choose.
| MAPQ | P(wrong placement) | Joint with a Q40 base |
|---|---|---|
| 0 | ~1 | ~0 |
| 20 | 10⁻² | ~0.98 |
| 30 | 10⁻³ | ~0.999 |
| 60 | 10⁻⁶ | ~0.9999 |
How callers actually treat it, and the trade-off:
- GATK HaplotypeCaller defaults
--min-mapping-quality 20, so MAPQ 0 reads are dropped before calling. - FreeBayes defaults
--min-mapping-quality 1: it keeps MAPQ 0 reads, but they contribute almost nothing to the likelihood. samtools/bcftools mpileupdefaults-q 0, i.e. no MAPQ filter at all; you have to add it.
The cost of filtering is sensitivity. In segmental duplications a real homozygous alt can sit entirely on multi-mapping reads, and a hard MAPQ filter will call it reference. That is the deciding criterion: is the region uniquely anchored elsewhere? If a read pair has one mate placed uniquely and the other in the repeat, the pair can rescue the call. If not, filtering is correct and the variant is simply not callable from short reads.
Failure mode to name: pseudogene/paralog collapse. PMS2 vs PMS2CL, CYP2D6 vs CYP2D7, SMN1 vs SMN2, GBA vs GBAP1. A Q40 base from the pseudogene placed at the gene locus looks like a confident pathogenic het. Detection: overlay a mappability track (GEM, or the UCSC 100-mer mappability track) on the callset and check whether the variant sits in a segdup.
What I'd do: filter MAPQ < 20 for the primary callset, keep MAPQ 0 reads in a separate pass, and require orthogonal evidence (a long read, or a uniquely-anchored pair) before reporting anything in a repeat. Never promote a MAPQ 0 pileup to a genotype on base quality alone.
Curated: · Written: · Reviewed:
Q&A-29Why is HaplotypeCaller’s diploid model the wrong default for a 30% purity tumor, and what must the model include instead?
Core answer
HaplotypeCaller's genotyper is a diploid model: it scores 0/0, 0/1, 1/1 and puts the heterozygous likelihood peak at allele fraction 0.5. That prior is right for inherited variation and wrong for a tumor, where the observed allele fraction is a function of purity, local copy number, and the fraction of tumor cells carrying the mutation. At 30% purity a clonal heterozygous somatic SNV in a diploid region sits near 15% VAF, and a subclonal one at half that. The diploid model has no component there, so it scores those sites as hom-ref or as error and drops them. What the model must include instead: tumor purity, local copy number/ploidy, cancer-cell fraction (subclonality), and normal/contamination admixture. A somatic or joint tumor-normal model, not a germline one.
Why the prior breaks: arithmetic
Expected VAF for a mutation present in m copies of a region with total copy number CNt, in a sample of purity p, carried by fraction c of tumor cells:
VAF = c · p · m / (p · CNt + (1 − p) · 2)
Clonal het, diploid region, p = 0.30, m = 1, CNt = 2, c = 1:
0.30 / (0.60 + 1.40) = 0.15
Subclonal, c = 0.5:
0.15 / 2 = 0.075 ≈ 8%
So 8% AF is a real variant, not noise. HaplotypeCaller's het likelihood at 0.5 gives that site a near-zero genotype likelihood; it either never enters an active region or fails the diploid QUAL. Lowering GQ thresholds doesn't fix it: you're moving the cutoff on the wrong likelihood.
What the model must include
- Purity p, estimated not assumed: ContEst from germline het sites, or ASCAT/PureCN/FACETS from allele fractions plus coverage.
- Local copy number CNt, so expected VAF is computed per locus, not globally.
- Cancer-cell fraction c: a mixture over subclones, not a single genotype.
- Contamination/normal admixture (1 − p) and a sequencing-error term, so low-VAF calls aren't all artifacts.
Alternatives and when to use them
| Caller | Model | Needs | Use when |
|---|---|---|---|
| HaplotypeCaller | diploid germline | — | germline only |
| Mutect2 | somatic, tumor-normal or tumor-only | matched normal, PoN, germline resource | standard somatic SNV/indel |
| Strelka2 somatic | somatic, tumor-normal | matched normal | high-precision somatic |
| PureCN/FACETS + caller | joint purity/CN/CCF | panel or matched normal | CN-altered, low-purity tumors |
Tumor-only Mutect2 without a PoN and germline resource trades sensitivity for false positives from CHIP and artifacts.
Failure modes and detection
Missed subclonal SNVs: detect by plotting sensitivity against VAF on a mix-in truth set (synthetic dilutions or a reference cell line at known purity). The diploid curve collapses below ~15% VAF while a somatic model holds. False positives: FFPE deamination and orientation bias inflate low-VAF calls; check the orientation-bias filter and read-position distribution. Operational cost: matched-normal sequencing, PoN construction, and purity/CN estimation add compute and a reference panel to maintain.
Validation
Don't accept a caller on reputation. Build a dilution series at 30%, 20%, 10% purity, spike known variants, and measure sensitivity and precision per VAF bin. Diploid GQ is not the x-axis; VAF and purity are.
Curated: · Written: · Reviewed:
Q&A-31You have 12 WGS. Why might VQSR be the wrong filter, and what do you use instead?
Core answer (first 90 seconds)
With 12 WGS I would not run VQSR. I'd apply GATK's hard-filter thresholds and validate them against a truth sample. VQSR fits a Gaussian mixture model to the cohort's own variant annotations, and 12 genomes does not give it enough variants to fit that model reliably.
Why VQSR fails at n=12
VQSR (GATK 4.x, VariantRecalibrator) builds a GMM per annotation (QD, FS, MQ, MQRankSum, ReadPosRankSum, SOR) using truth sets (HapMap 3.3, Omni 2.5, 1000G phase 1 high-confidence) as positive training and the rest of the cohort as negative. Two things break here:
- Too few variants per annotation. GATK's guidance puts the practical floor at roughly 30 exomes; a dozen genomes is below it. The mixture components are estimated from a thin sample, so the model overfits this assay's annotation distributions.
- Thin positive training set. Few of 12 samples' variants overlap the truth sets, so the positive class is sparse and the fit is driven by the negative class.
The consequence: the tranche (99.9, 99.0) no longer maps to a stable sensitivity. It is a number fit to noise. Copying a tranche from a 1000G-scale run is worse, because the annotation distributions differ by chemistry, read length and caller.
What to use instead
GATK's hard-filter thresholds, applied with VariantFiltration:
| Annotation | SNP threshold | Indel threshold |
|---|---|---|
| QD | < 2.0 | < 2.0 |
| FS | > 60.0 | > 200.0 |
| MQ | < 40.0 | — |
| MQRankSum | < -12.5 | — |
| ReadPosRankSum | < -8.0 | < -20.0 |
| SOR | > 3.0 | > 10.0 |
gatk VariantFiltration \
-R ref.fa -V cohort.vcf.gz -O cohort.filtered.vcf.gz \
--filter-expression "QD < 2.0" --filter-name "QD2" \
--filter-expression "FS > 60.0" --filter-name "FS60" \
--filter-expression "MQ < 40.0" --filter-name "MQ40" \
--filter-expression "MQRankSum < -12.5" --filter-name "MQRankSum-12.5" \
--filter-expression "ReadPosRankSum < -8.0" --filter-name "ReadPosRankSum-8" \
--filter-expression "SOR > 3.0" --filter-name "SOR3"
These are GATK Best Practices defaults, not tuned to your assay: treat them as a starting point and check them.
Validation
Run both on a truth sample (GIAB HG002, or a sample with a validated truth VCF) and compare recall and precision on singletons, where the difference bites. Hypothetical: if hard filters give 0.91 singleton recall and VQSR 0.58 on this cohort, hard filters win, but that is a number you measure, not assume.
When VQSR is still right
Hundreds of samples, or a cohort large enough that the GMM is well-conditioned. The cost of hard filters is that you lose the adaptive tranche: thresholds are fixed and assay-dependent, so a shifted chemistry can silently pass bad variants or drop good ones. That is why you validate.
Curated: · Written: · Reviewed:
Q&A-30Your caller’s genome-wide F1 is 0.999. Why might it still be unsafe in a repeat, and where are you allowed to count FP and FN?
Direct answer. A genome-wide F1 of 0.999 is a claim about the whole genome, but the benchmark only asserts truth inside its confident regions: about 90% of GRCh38 for GIAB v4.2.1. So the number is undefined outside those regions, and it is an average that can hide a stratum where the caller is bad. It is unsafe in a repeat because the three things that change between runs (the truth set, the sample, the batch) each move the number, and one scalar gives you no way to see which one moved.
Where you may count FP and FN. Only inside the intersection of the benchmark's confident regions and your callable regions, with the same sample, the same reference build, and the same truth-set version. Outside confident regions an unlabeled variant is not a false positive: it is unknown, and counting it as FP is the most common way to manufacture a bad number. Same for no-call or low-coverage stretches: exclude them from both sides or you are scoring coverage, not accuracy.
Why a repeat can fail.
- Truth-set update: v3.3.2 to v4.2.1 changed both the region BED and the variant set. A caller tuned to v3.3.2 can drop several points on v4.2.1 with no code change. Detect by hashing the benchmark VCF and BED and pinning the version in the report.
- Sample swap: HG002 versus HG003. Detect with a genotype-concordance fingerprint (somalier, verifyBamID) before you trust any F1.
- Batch effect: different library prep or coverage shifts calls in GC-rich and repeat regions. Detect by per-batch F1 drift and coverage-matched strata.
- Build mismatch: GRCh37 truth against a GRCh38 call set scores as garbage. Detect by checking contig names and lengths.
How stratification works. Intersect callable with confident, then intersect that with annotation BEDs: MHC, segmental duplications, tandem repeats, low-mappability. Run hap.py or vcfeval with the stratification file so each stratum gets its own TP, FP, FN and F1.
Trade-offs. Genome-wide F1 is fine as a single regression gate when your deployment is also genome-wide and the truth set matches. It is the wrong gate when the product targets specific regions (MHC typing, a clinical panel) because the average is dominated by easy regions. Stratified reporting costs more compute and invites cherry-picking, so publish all strata with counts, not the best one.
Artefact. Illustrative counts:
| Stratum | Truth variants | TP | FP+FN | F1 |
|---|---|---|---|---|
| Confident, genome-wide | 4,000,000 | ~3,996,000 | ~8,000 | 0.999 |
| MHC | 8,500 | 8,100 | 3,800 | 0.81 |
MHC is 0.2% of truth variants but about 47% of the error budget. The genome-wide 0.999 is real and useless for MHC typing.
Curated: · Written: · Reviewed:
Q&A-27You archived 40 TB as CRAM. What must travel with the files, and what fails if the FASTA moved?
The 90-second answer
CRAM stores reads as differences from a reference, so the reference is part of the archive, not a tool you happen to have lying around. Four things travel with the 40 TB: the CRAM files, their .crai indexes, the exact reference FASTA plus its .fai, and checksums. Sha256 of the FASTA, and the per-slice MD5s that are already inside every CRAM. Or you embed the reference in the CRAM and ship one file.
If the FASTA moved but its bytes are unchanged, nothing fails: point -T at the new path, or let an M5-keyed cache resolve it. If the bytes changed (one patched contig, one swapped base) every slice overlapping that region fails to decode. htslib computes the MD5 of the reference bases for each slice (uppercased, whitespace stripped) and compares it to the M5 stored in the slice; a mismatch aborts the decode. If the FASTA is gone and there is no cache, the decode fails outright.
A "GRCh38" label is not enough, because GRCh38 ships in several flavours: with or without decoys, with or without alt contigs, with different contig naming. The M5 is what pins the exact bytes.
Why the M5 is the thing that saves you
The CRAM header carries a reference URI, but it is a hint: often a path on a machine that no longer exists. The authoritative link is the per-slice M5. That is what lets a tool fetch the right sequence from a checksum-keyed cache, and what turns a wrong reference into a loud failure instead of a quiet one.
Three ways to carry the reference
| Approach | What you ship | Applies when | Cost |
|---|---|---|---|
| Embed in CRAM | CRAM only | cold archive, no registry access, custom or patched reference | +~3.1 GB uncompressed per file ≈ 0.008% of 40 TB |
| External FASTA | FASTA + .fai + sha256 | you need the reference for other tools too | must stay byte-identical forever |
M5 cache / REF_PATH | CRAM + a REF_PATH | reference is a public standard build | depends on the registry staying up; useless for a patched reference |
For 40 TB, embedding is the cheap answer: 3.1 GB is 0.008% of the archive, and it removes the failure mode entirely. The external-FASTA route is fine if you also keep the sha256 and re-verify it before every restore.
What actually fails
- Renamed FASTA, same bytes: fine, as long as you resolve it by content or point at the new path.
- Patched FASTA: slices over the patched contig fail, the rest decode. This partial failure is the dangerous one: a spot check that decodes three files can pass while 1% of the archive is unreadable.
- Missing
.crai: you can still stream-decode the whole file, but no region queries. - Missing
.fai: the reference cannot be randomly accessed, so decode fails. - Skipping the check: htslib's
ignore_md5option turns the mismatch into silently wrong bases. Never use it on a restore.
The drill
sha256sum -c GRCh38.fa.sha256 # reference is what you think it is
samtools view -T GRCh38.fa -h s1.cram | head # decodes
samtools view -T GRCh38_patched.fa s1.cram # aborts: MD5 reference mismatch
Decode a random 1% of the archive against the checksummed reference, not the files you happened to test last time. Checked against CRAM 3.0 and htslib/samtools 1.x.
Curated: · Written: · Reviewed:
Q&A-32A VCF BND looks like A[chr2:10[. What information lives in the brackets, and what does interval-only parsing lose?
In A[chr2:10[, the brackets carry two things: the mate's address and the join's orientation. Read it as: the local sequence up to and including the base A continues into the piece of chr2 that extends to the right of chr2:10. A is the reference base at the breakend on the local chromosome; chr2:10 is the mate breakend; the two [ characters fix which side of the mate is retained and which strand the join runs on.
The four legal forms (VCF 4.3):
| ALT | Meaning |
|---|---|
t[p[ | piece extending to the right of p is joined after t |
t]p] | reverse-complemented piece extending to the left of p is joined after t |
]p]t | piece extending to the left of p is joined before t |
[p[t | reverse-complemented piece extending to the right of p is joined before t |
A[chr2:10[ is row 1. Its mate record at chr2:10 names chr1:100 (the local position) and describes the same junction from the other end; the two records are one join seen from each side.
What interval-only parsing loses:
- Orientation.
[vs], and whether the bracket sits before or aftert, select one of four joins. Strip them and the join is gone. - Mate linkage. The mate's coordinates live in the ALT. Drop the ALT and you have an unpaired coordinate, not a rearrangement.
- Event type. The same two coordinates are a deletion or an inversion depending only on the bracket.
- The local base
t, needed to reconstruct the junction base-exactly. - Imprecision.
CIPOS/CIENDwiden the breakend; a point coordinate overstates certainty.
Worked example: same coordinates, different event:
chr1 100 . A A[chr1:5000[ # deletion: 100 joined rightward to 5000
chr1 100 . A A]chr1:5000] # inversion: 100 joined to revcomp of 100-5000
An interval-only parser reads chr1:100 and chr1:5000 in both rows and reports the same thing, usually a ~4.9 kb deletion. Row 1 is a deletion; row 2 is an inversion of the intervening segment. Only the bracket differs.
Failure mode: a parser that ignores ALT and spans min/max POS per chromosome fabricates intervals. Given a balanced t(1;2) with breakends at chr1:100 (mate chr2:10) and a second pair at chr1:40,000,000 (mate chr2:50,000,000), min/max on chr1 yields a ~40 Mb "deletion" and drops both chr2 partners. A CNV that does not exist. Detect it by round-tripping: every BND must survive ingest as two oriented joins with mates intact.
Curated: · Written: · Reviewed:
Q&A-35Why are TPM values the wrong matrix for DESeq2, and which unit belongs in the likelihood?
Short answer. DESeq2 fits a negative-binomial GLM whose response is a count of fragments. The unit in that likelihood is expected fragment counts: non-negative integers, with the mean modeled as s_j * q_ij (a per-sample size factor times the expected count). TPM is not a count: it is (c_i / L_i) rescaled so each sample sums to 1e6. Length and library size are already divided out, so TPM is a different quantity and cannot be the response.
Why the count unit is load-bearing. Two parts of the model depend on it:
- Size factors. DESeq2 estimates
s_jby median-of-ratios against a pseudo-reference, computed from counts. TPM is already library-normalized, so those ratios sit near 1 and the size factors collapse toward 1: you lose composition correction between samples. - Dispersion. The NB variance is
mu + alpha * mu^2, andalphais estimated from the mean-variance relationship of counts. Dividing by gene length rescales each gene's mean by a different factor, so two genes of equal expression but different length land at different means with a variance that is no longer NB. The dispersion estimate is then driven by length rather than by sampling noise.
Arithmetic. For gene i in sample j: TPM_ij = (c_ij / L_i) / sum_k (c_kj / L_k) * 1e6. A 2 kb gene and a 4 kb gene with the same 100 counts get TPMs differing by 2x. The denominator also depends on every other gene's length, so TPM is not proportional to counts across samples, and that proportionality is exactly what DESeq2 assumes.
What to use instead. Raw fragment counts, or tximport's estimated counts with DESeq2's offset (DESeqDataSetFromTximport). If a downstream tool can only take a matrix, countsFromAbundance="lengthScaledTPM" gives length-corrected counts. Use TPM only for within-sample description, "gene X is 3% of this library", never for between-sample DE.
Failure mode. Rounding TPM to integers does not restore counts; it invents pseudo-counts whose scale is arbitrary. TPM×10 and TPM×1000 give different dispersions and different gene lists on the same data. The tell is a DE list enriched for long or short genes. A worked example (hypothetical figures):
| Input | FDR<0.05 genes |
|---|---|
| raw counts | 40 |
| rounded TPM×10 | 400 |
| validation set | 40 |
Assumptions. This is gene-level DE with DESeq2 on RNA-seq. Isoform-level work, or tools that model TPM directly (e.g. limma-voom on log-CPM), is a different question with a different likelihood.
Curated: · Written: · Reviewed:
Q&A-36One globin gene ate 40% of reads in the treated group. Why does dividing by total counts create fake DE elsewhere, and what estimator do you use?
The short answer
Total-count normalization assumes total RNA per cell is roughly constant across samples, so library size tracks a common quantity. A globin surge breaks that assumption. Reads are a fixed pie: if globin takes 40% of the treated library, every other gene's share of the pie falls even though its absolute expression is unchanged. Dividing by the total converts that mechanical deflation into apparent log-fold changes. Use a size factor estimated from a robust subset of genes, median-of-ratios (DESeq2) or TMM (edgeR), never the column sum.
Why the fake DE appears
Hypothetical two-sample example, 20M reads each, gene X at 1000 reads in untreated. Globin is 2% of untreated reads and 40% of treated. Gene X's true expression is unchanged, so its treated count is 1000 × (1 − 0.40)/(1 − 0.02) = 612.
Total counts: 612/1000 = 0.61 → log2 = −0.71. Reported as 1.5-fold repression; nothing happened.
Median-of-ratios: geometric mean of gene X across the two samples = sqrt(1000 × 612) = 782. Ratios 1000/782 = 1.28 and 612/782 = 0.78. Taking the median of those ratios over all genes gives size factors s_U = 1.28, s_T = 0.78. Normalized: 1000/1.28 = 781, 612/0.78 = 785. log2FC ≈ 0. The estimator recovered the truth because it never used the total.
The estimators
- DESeq2 median-of-ratios (
estimateSizeFactors): per gene, geometric mean across samples; per sample, the ratio of that sample's count to the geometric mean; size factor = median ratio over genes. Genes with a zero in any sample are excluded from the geometric mean. - edgeR TMM (
calcNormFactors(method="TMM")): log-fold-changes against a reference sample, trimmed by abundance and by M-value. DefaultslogratioTrim = 0.3,sumTrim = 0.05,doWeighting = TRUE.
library(DESeq2)
dds <- DESeqDataSetFromMatrix(counts, colData, ~condition)
dds <- estimateSizeFactors(dds)
sizeFactors(dds)
library(edgeR)
y <- DGEList(counts = counts, group = condition)
y <- calcNormFactors(y, method = "TMM")
y$samples$norm.factors
Where they fail
Both assume most genes are not DE. If a treatment genuinely shifts the whole transcriptome, that real global effect gets absorbed into the size factor and disappears. Symptom: size factors consistently far from 1 in one group with no single gene explaining it.
Above roughly half the library on one gene, the informative gene set is too small and noisy for either estimator. Fix it upstream: globin depletion at library prep, in-silico removal of globin reads before normalization, or ERCC spike-ins with a spike-in-based size factor.
TPM and RPKM carry the same total-based flaw; don't use them for DE.
How I'd check it
Plot size factor against total counts per sample. If they diverge, composition bias is present. Then put the size-factor table next to the top genes' fraction of reads, and sanity-check the surviving hits against qPCR.
Curated: · Written: · Reviewed:
Q&A-34Every treated sample is on plate A and every control on plate B. Can DESeq2 unconfound treatment, and what do you do instead?
No. If treatment and plate are the same column of the design, the contrast is not identifiable: the two effects are perfectly collinear and no likelihood can separate them. DESeq2 will not do it, and neither will edgeR or limma. The fix is a design decision made before sequencing, not a software setting.
Why it is not identifiable. Write the design matrix for ~ condition + plate with 8 treated on A and 8 control on B:
| Plate | Treated | Control |
|---|---|---|
| A | 8 | 0 |
| B | 0 | 8 |
The condition column and the plate column are the same vector. The model matrix has three columns (intercept, condition, plate) and rank 2, so there is no unique least-squares solution: any split of the observed difference between "treatment effect" and "plate effect" fits the data equally well.
# R 4.4.1, DESeq2 1.44.0
coldata <- data.frame(
condition = factor(c(rep("treat", 8), rep("ctrl", 8))),
plate = factor(c(rep("A", 8), rep("B", 8)))
)
mm <- model.matrix(~ condition + plate, coldata)
qr(mm)$rank # 2
ncol(mm) # 3 -> rank-deficient
The trap is the silent run. DESeq2 only knows the columns you name in the design formula. Fit ~ condition and it never sees plate, so it fits cleanly and returns a confident gene list:
dds <- DESeqDataSetFromMatrix(counts, coldata, design = ~ condition + plate)
dds <- DESeq(dds)
# Error: the model matrix is not full rank, so the model cannot be fit as specified.
design(dds) <- ~ condition # plate omitted
dds <- DESeq(dds) # fits, returns results — all of it confounded
The error is the good outcome. The run that succeeds is the one that produces a publishable-looking table of plate effects labelled as treatment effects. In one illustrative case of this shape (8 vs 8, fully confounded), the naive ~ condition run returned 1,842 genes at FDR < 0.05; a later balanced rerun of the same comparison reproduced 61 of them. Treat those numbers as an illustration of the failure mode, not a measured constant.
What to do instead.
- Fix the layout before sequencing. Split each condition across both plates (4/4 and 4/4). Condition and plate are then no longer collinear and
~ condition + plateis identifiable. This is the only real fix. - If samples are paired by donor, use
~ donor + condition. The donor term absorbs the plate effect and the contrast becomes identifiable, but only if every donor contributes both conditions. - If the data are already confounded and cannot be re-run, estimate the unwanted factor with surrogate-variable methods: RUVseq's
RUVg(needs negative-control genes such as spike-ins or housekeeping genes) orRUVs(needs replicate samples within each group). Both add estimated factors to the design. Without those anchors there is nothing to estimate the factor from and the method degenerates. limma::removeBatchEffectis for plots only. It corrects the expression matrix for PCA and heatmaps; running a DE test on the corrected matrix understates variance and is not valid.- Report it as a hypothesis. A confounded comparison can generate a candidate list for a follow-up experiment. It cannot support a claim about treatment.
Detection. Before any DE run, check qr(model.matrix(design, coldata))$rank against ncol(...). If rank is short, stop the pipeline. After a run, look at the VST/rlog PCA: if plate is PC1 and separates perfectly, the design is suspect regardless of what the results table says. Batch-driven hits cluster in pathways (ribosomal, mitochondrial, stress-response genes) which is a symptom, not proof.
Curated: · Written: · Reviewed:
Q&A-33You sequenced the same library three times. Why does that not let you claim a population expression difference?
The 60-second answer
Three runs of one library are technical replicates. They sample lane-to-lane and flowcell-to-flowcell variation in the sequencing chemistry, not the variation between the animals or donors your claim is about. The reads are correlated, they trace back to the same molecules, so they are not independent draws from the population you want to generalise to. A differential-expression test needs a denominator: the spread among independent biological units. With one unit per group you have zero biological residual degrees of freedom, so there is nothing valid to divide by. Any p-value you get is computed against technical variance, which is far smaller than biological variance, so it is inflated and the "hits" are lane noise.
Why the test breaks
DESeq2 and edgeR estimate per-gene dispersion from the spread among replicates, then use it to set the variance of the log-fold-change estimate. Feed them three lanes of one library and the dispersion they estimate is technical. With a single sample per group there are no replicates at all, and DESeq2 falls back to a fixed dispersion instead of estimating one; edgeR needs a prior or a fixed common dispersion. Either way you have substituted a guess for the quantity the test is supposed to measure. This is pseudoreplication: subsamples treated as independent units inflate the effective n.
Worked example (hypothetical figures)
Take log2 expression for one gene. Technical SD between lanes of the same library: 0.05. Biological SD between mice: 0.5. Observed difference: 0.2 log2.
tech_sd <- 0.05; bio_sd <- 0.5; delta <- 0.2
# A: 3 lanes of one mouse vs 1 mouse
se_a <- tech_sd/sqrt(3); t_a <- delta/se_a; p_a <- 2*pt(-abs(t_a), df=2)
# B: 3 mice vs 3 mice
se_b <- bio_sd/sqrt(3); t_b <- delta/se_b; p_b <- 2*pt(-abs(t_b), df=4)
round(c(se_a, t_a, p_a, se_b, t_b, p_b), 4)
# 0.0289 6.9282 0.0202 0.2887 0.6928 0.5266
| Design | SE | t | df | p |
|---|---|---|---|---|
| 3 lanes of one mouse vs 1 mouse | 0.029 | 6.93 | 2 | 0.02 |
| 3 mice vs 3 mice | 0.289 | 0.69 | 4 | 0.53 |
The denominator is off by roughly 10x, so the p-value is off by orders of magnitude. That is the whole failure.
Depth vs replication
Depth and replication buy different things. Depth reduces technical (approximately Poisson) noise and lets you see lowly expressed transcripts; biological replicates reduce biological variance and give you the degrees of freedom to test. For a between-group population claim, replication wins once each sample is sequenced deeply enough to observe the genes you care about. ENCODE's guideline for gene expression is 30M paired-end reads per sample; DE power studies put the knee lower, around 10-15M reads (Liu et al., Bioinformatics 2014). Deep sequencing of few samples is the right call when the question is about one individual: isoform discovery, allele-specific expression, rare transcripts within a sample. It is the wrong call when the question is "does this differ between groups."
How it shows up
The tell is in the sample sheet: if the replicate rows share a library ID, or a flowcell/lane column varies while the subject column does not, you have technical replicates. Check that the experimental-unit column has as many distinct values as the design matrix has rows per group.
Self-assessment
Key points the interviewer wants:
- Technical replicates are not independent draws from the population; the unit of inference is the biological subject.
- The test's denominator is biological variance; with one subject per group there is no biological residual df.
- Pseudoreplication inflates effective n and shrinks p-values.
- Fix: add biological replicates; keep technical replicates only to control lane/flowcell effects.
What separates a strong answer:
- Names the mechanism (dispersion estimation) rather than just saying "you need replicates."
- Quantifies the gap between technical and biological variance.
- Distinguishes when deep sequencing of few samples is correct (within-individual questions).
Weak-answer patterns:
- "Three replicates is enough": conflates technical with biological.
- "Just use a paired test": pairing does not create independent units.
- "Increase depth to fix it": depth reduces technical noise, not biological variance.
Follow-up questions:
- How many biological replicates do you need for 80% power at a given effect size?
- What does DESeq2 do when you give it n=1 per group?
- How would you detect pseudoreplication in a published dataset?
- When is a technical replicate the right unit of analysis?
Curated: · Written: · Reviewed:
Q&A-37A stranded library counted with the opposite convention lost 70% of assigned reads. What did strandedness decide, and how do you catch it?
What strandedness decided
Strandedness decides which strand of a fragment is allowed to support a feature. featureCounts assigns a read to a gene only if the read's strand matches the gene's strand under the -s setting: -s 1 forward, -s 2 reverse, -s 0 ignore. Set the wrong one and every read whose true gene sits on the other strand has no feature to land on, so it falls into "unassigned", not misassigned, just dropped. That is the 70% loss: the library was reverse-stranded (dUTP) and the counter was told forward.
Why dUTP inverts orientation
In the Illumina dUTP protocol, second-strand synthesis incorporates dUTP and that strand is not amplified, so only the first strand, the reverse complement of the original mRNA, is sequenced. Reads therefore map to the opposite strand from the transcript. That is why dUTP libraries need -s 2, while older non-dUTP protocols need -s 1. The convention is a property of the library prep, not of the aligner or the FASTQ.
Worked example (hypothetical)
20M read pairs, 1,200 gene pairs with overlapping antisense transcription:
| featureCounts -s | Assigned | Unassigned | What happens |
|---|---|---|---|
| 2 (correct) | 85% | 15% | reads land on their gene |
| 1 (opposite) | 25% | 75% | ~70% of assigned reads lost |
| 0 (unstranded) | 92% | 8% | keeps everything, including antisense noise |
Note the asymmetry: the wrong stranded setting loses reads; unstranded keeps them but also counts the antisense partner, inflating overlapping loci.
How to catch it
Run RSeQC infer_experiment.py on a BAM subset (it samples 200,000 reads by default). It prints two fractions: reads explained by 1++,1--,2+-,2-+ versus 1+-,1-+,2++,2--. One near 1.0 means stranded: high on the first pattern is forward (-s 1), high on the second is reverse (-s 2); both near 0.5 means unstranded. Cross-check that against the sample sheet or protocol metadata and fail the run on disagreement. Salmon's -l A auto-detects the same thing, but per-sample inference beats a global guess.
Trade-offs
-s 0 is the safe default when you genuinely don't know the prep, it never loses reads, but it cannot disambiguate overlapping sense/antisense features and doubles the effective noise at those loci. -s 1/-s 2 buys that specificity at the cost of a hard failure if wrong. Auto-detection is convenient but can be fooled by low-complexity or heavily antisense libraries, so treat it as a check, not the source of truth.
Failure modes
The expensive one is mixing conventions across samples in a single count matrix: half the samples lose ~70% of their signal, and the resulting "batch effect" looks like biology in the DE table. Detect it by running inference per sample and comparing the fraction pattern across the cohort, not just against the sheet. Second, antisense-overlapping loci (lncRNAs, readthrough transcripts) are where a wrong or unstranded setting does the most damage: check those genes' counts before trusting the matrix.
Curated: · Written: · Reviewed:
Q&A-39A tumor RNA-seq study shows immune genes up. How do you tell regulation inside cells from more immune cells in the mix?
Short answer: bulk counts are per-cell expression multiplied by cell-type proportion, so an immune signature can rise because immune cells got more numerous, because each cell transcribed more, or both. Separate them by estimating composition and testing whether the signal survives adjustment; only a cellular assay (sorted cells, scRNA-seq, spatial) settles the within-cell claim.
What I'd say in the first two minutes
- Check the signature's own markers. If the "IFN signature" is driven by IFNG, CXCL10 or HLA-DRA, those genes are themselves induced by activation, so they cannot double as abundance markers. Use markers that don't respond to the stimulus: CD3E for T cells, CD68 for macrophages, EPCAM for epithelium.
- Estimate composition. Deconvolution (CIBERSORTx, quanTIseq, BayesPrism) or matched scRNA-seq gives per-sample fractions. Compare fractions between groups before interpreting any signature.
- Adjust and re-test. Divide or regress out the estimated fraction and see what is left of the signature.
- Confirm cellularly. Sorted-population RNA-seq, per-cell DE from scRNA-seq, or spatial transcriptomics.
Worked example (hypothetical numbers)
Bulk IFN-stimulated genes read 8-fold up in tumor versus adjacent normal. Deconvolution says T cells went from 12% to 24% of the mix (2.0x), and within-T-cell IFN induction is 1.3x.
| Component | Fold |
|---|---|
| T-cell fraction 12% → 24% | 2.0 |
| within-T-cell IFN induction | 1.3 |
| composition × regulation | 2.6 |
| observed bulk | 8.0 |
| residual (8.0 / 2.6) | ~3.1 |
T cells explain at most 2.6x of the 8x. The remaining ~3x has to come from another compartment (NK cells, macrophages, a stromal response) and until that is attributed, "IFN pathway activated in tumor cells" is not supported. The bulk number is real; the interpretation is not.
Where this goes wrong
- Blood-derived signature matrices applied to tumor. CIBERSORT's LM22 contains 22 leukocyte subsets and no fibroblasts, no tumor-associated macrophage states, no exhausted T cells, so fractions come back biased. Use a tissue-matched reference or build one from scRNA-seq of the same tissue.
- Activation-responsive genes used as abundance markers. IFNG, GZMB and HLA-DRA all move with stimulation, so a "T-cell fraction" built from them is partly the effect you are trying to measure.
- Pseudobulk versus single-cell mismatch. A signature defined on pseudobulk of sorted cells may not transfer to whole-tissue deconvolution.
- Ignoring tumor purity. In a low-purity sample everything looks immune-rich.
What I'd ask for before believing a within-cell claim
- A composition table next to the signature, per sample.
- The same signature tested after composition adjustment.
- A cellular assay: sorted T cells, per-cell DE from scRNA-seq, or spatial co-localization of the transcript with a cell-type marker.
More of a cell type is not the same as turning a gene on.
Curated: · Written: · Reviewed:
Q&A-38A gene has adjusted p=0.04 at FDR 5%. Why is that not a 4% chance the gene is false, and what else belongs on the plot?
Short answer. padj = 0.04 says: reject every gene with padj ≤ 0.05 and at most 5% of that rejected set is expected to be null. It is a statement about the list. It is not P(this gene is null | data) = 0.04.
Why, mechanically. Benjamini–Hochberg sorts the m raw p-values, p(1) ≤ … ≤ p(m), and sets
padj(i) = min over k ≥ i of ( p(k) · m / k )
with the running minimum enforcing monotonicity. Worked example, hypothetical: m = 20,000 genes tested. A gene has raw p = 0.001 and is the 500th smallest. padj = 0.001 × 20,000 / 500 = 0.04. The multiplier 20,000/500 = 40 is the multiplicity correction, and it depends on the gene's rank, not on the gene. If 500 genes pass at padj ≤ 0.05, expected false discoveries ≤ 0.05 × 500 = 25. Which 25 is unknown; this gene is one of 500 candidates, not a 4% coin flip.
The gap to a posterior is the null proportion π0. BH controls FDR at α·π0 ≤ α, so with π0 = 0.9 the realised FDR is nearer 4.5%. A posterior P(null | data) needs π0 and a null distribution; BH uses neither. Storey's q-value estimates π0 from the p-value histogram and buys a little power; Efron's local fdr gives a per-gene posterior but is model-dependent.
When FDR is the wrong measure. If one false positive is expensive (a clinical variant, a confirmatory target) control FWER instead: Bonferroni (padj = min(1, p·m)) or Holm, which is uniformly more powerful. FWER bounds P(any false positive) ≤ α, which is what the "this gene is wrong" intuition actually wants, at the cost of a much shorter list. If you want a per-gene probability, you need a posterior or local-fdr model and a stated prior. The deciding criterion is the cost of one false positive against the cost of a missed true positive, and whether the list is exploratory or confirmatory.
What belongs on the plot. A volcano: x = log2 fold change, y = −log10(padj). Horizontal line at padj = 0.05, vertical lines at |log2FC| = 1. Label the top genes by padj. Point size or colour by mean normalized count, so low-count noise is visible. Use shrunken LFCs (DESeq2 lfcShrink with apeglm, or edgeR's glmTreat): raw LFCs overstate effects for low counts. Pair it with an MA plot (log2 mean normalized count vs log2FC) to show the abundance trend. DESeq2, edgeR and limma-voom all default to BH for padj.
Failure mode. A volcano on raw p with no effect-size threshold yields a long tail of tiny-fold-change genes that replicate poorly. Detect it on the MA plot: fold changes fanning out at low counts. Validate hits independently, orthogonal assay or held-out cohort, because the set-level guarantee does not transfer to a new sample.
Curated: · Written: · Reviewed:
Q&A-40Why must count matrices keep Ensembl IDs with versions instead of joining on HGNC symbols?
Core answer
The join key has to identify the gene model, not the gene's label. In Ensembl that is the stable ID plus its version: ENSG followed by 11 digits, then . and the model version, e.g. ENSG00000141510.18. The ID is stable across releases; the version increments when the gene model changes. Symbols are attributes (HGNC-approved names that get renamed, aliased and reused) and they are not unique in the annotation files you actually join against.
So keep the versioned gene_id as the key, carry gene_name as a display column, and record the GTF release the matrix was quantified against.
Why symbols fail as keys
Two independent problems, both silent:
- Renames. HGNC renames genes and annotation sources lag. A count matrix quantified against release 110 can carry
C1orf106where release 113 carriesINAVA(HGNC renamed C1orf106 to INAVA in 2018). An inner join on symbol returns zero rows for that gene: it disappears, with no error. - Duplicates.
gene_namein a GTF is not guaranteed unique. If two features share a symbol, an inner join on symbol fans out: one count row becomes two, and any downstream sum double-counts.
The stable ID has neither problem: one ID maps to one gene model.
Worked example (IDs illustrative)
import pandas as pd # pandas 2.x
counts = pd.DataFrame({
"gene_id": ["ENSG00000123456.5"],
"gene_name": ["C1orf106"], # symbol as written in the release-110 GTF
"count": [120],
})
annot = pd.DataFrame({ # pulled from release 113
"gene_id": ["ENSG00000123456.5", "ENSG00000654321.2"],
"gene_name": ["INAVA", "INAVA"],
})
counts.merge(annot, on="gene_id") # 1 row, count 120
counts.merge(annot, on="gene_name") # 0 rows — gene dropped
counts.assign(gene_name="INAVA").merge(annot, on="gene_name") # 2 rows, 120 twice
Trace: the ID join matches the single ENSG00000123456.5 row, so one row out with count 120. The symbol join finds no C1orf106 in the annotation, so zero rows and the gene is gone. If the symbol had matched, the duplicate INAVA in the annotation would return two rows and the count would be counted twice (240).
| Join key | Rows in | Rows out | Result |
|---|---|---|---|
gene_id (versioned) | 1 | 1 | correct |
gene_name (renamed) | 1 | 0 | gene silently dropped |
gene_name (duplicated) | 1 | 2 | count double-counted |
Why the version, not just the ID
The version pins the model. The same ENSG can appear at different versions in different releases because the exon structure or sequence changed. Join a release-110 matrix to release-113 annotation on ID alone and you silently mix models. Join on ID+version when you need exact model identity, which is what reproducibility demands; join on ID alone only when you accept model drift and want to carry counts forward.
Trade-offs and when symbols are fine
Symbol joins are acceptable for quick, single-release, single-source exploration where both sides come from the same GTF, and even then they are fragile. The cost of versioned IDs is churn: a version bump breaks exact-match joins across releases, and retired or merged IDs need a mapping table to follow. Budget for both: a retired-ID mapping and a symbol column for display.
Detection
Assert the join is 1:1. Row count unchanged, every feature mapping to exactly one annotation row. Store the GTF release and version in the matrix metadata and check it matches the annotation. A row-count drop or a fan-out is the signal.
Curated: · Written: · Reviewed:
Q&A-90You drop genes with mean count below 10 after looking at which ones were significant. Why is that invalid, and when is a filter allowed?
Assume bulk RNA-seq, per-gene differential expression, and FDR control at 5%. The invalid move is choosing the filter after seeing the test results. If you run the test, notice that low-mean genes are mostly non-significant, then drop mean count <10 and report the remaining p-values, the filter is a function of the test outcome. Under the null, p-values are uniform; after conditioning on a filter selected from those same p-values, they are not. BH-FDR no longer controls. A filter is allowed when its statistic is independent of the test statistic under the null. Then you can predeclare a mean-count cutoff, or use DESeq2's independent filtering, which chooses the cutoff that maximizes rejections using mean normalized counts. That tuning is legitimate because the filter statistic is independent of the test under the null. The distinction is independent statistic vs test outcome, not tuning vs no tuning.
Mechanically, independent filtering works because the filter removes genes that carry little information for the test without changing the null distribution of the p-values you keep. DESeq2's results() does this by default: it uses the mean of normalized counts as the filter statistic and selects the threshold that maximizes the number of rejections after BH adjustment. That is a data-dependent choice, but it is valid because the mean count is independent of the test statistic under the null. A predeclared mean-count cutoff is the simpler special case: state it before unblinding, apply it, and report it.
Variance and dispersion filters are the common trap. They are not generally independent of the test under the null: under the null, variance is a function of mean and dispersion, and the test statistic depends on those. Filtering on variance can remove genes with high variance that would have been true positives, and it can distort the p-value distribution. If you use one, you need a proof or simulation showing independence for your design, not an assumption.
| Approach | Cutoff chosen from | Valid? | Why |
|---|---|---|---|
| Predeclared mean count >=10 | Protocol before p-values | Yes | Mean count is independent of the test under the null |
| DESeq2 independent filtering | Mean normalized count, maximizing rejections | Yes | Same independence; optimization is over an independent statistic |
| Drop mean<10 after seeing significance | The p-values themselves | No | Filter is a function of the test outcome; null p-values are no longer uniform |
| Variance/dispersion filter | Usually after seeing spread | Usually no | Variance is not independent of the test under the null; can bias p-values |
Failure mode: FDR inflation and failed validation. Hypothetical: 20,000 genes, 2,000 true DE. If you raise the mean-count cutoff until 200 genes remain "significant", you may enrich for genes that were already extreme by chance; qPCR of 20 such genes might confirm 2. A predeclared mean>=10 filter might leave 40 genes and confirm 16 of 20. The numbers are illustrative, but the pattern is the point: selection on the outcome buys a shorter list, not a more accurate one.
What to do: state the filter statistic and cutoff in the analysis plan before unblinding. If you use DESeq2, leave independentFiltering=TRUE and report the filter threshold. If you must filter on variance, validate the independence assumption by simulation under your null. Otherwise, do not filter after seeing p-values.
Curated: · Written: · Reviewed:
Q&A-102Your differential-expression table has 20,000 genes and 1,800 hits at raw p < 0.05. Why is that number expected by chance, what does BH FDR do to it, and when would you use a family-wise method instead?
Twenty thousand independent tests at α = 0.05 give 20,000 × 0.05 = 1,000 false positives under the global null, before any biology. So 1,800 raw hits is 800 above the null floor, and 800 is an upper bound, because it assumes every non-null gene was detected. The number to report is not 1,800.
Benjamini–Hochberg controls the expected proportion of false discoveries among rejections. Sort p-values ascending, find the largest k with p(k) ≤ (k/m)·q. At q = 0.05 and m = 20,000 the threshold is a ramp: 2.5e-6 at k = 1, 5e-4 at k = 200, 4.5e-3 at k = 1,800, and 0.05 only at k = m. A hypothetical p-value distribution makes the effect concrete:
| rank k | p(k) | BH line (k/20000)×0.05 | reject |
|---|---|---|---|
| 1 | 1e-9 | 2.5e-6 | yes |
| 200 | 3.0e-4 | 5.0e-4 | yes |
| 201 | 5.1e-4 | 5.0e-4 | no |
| 1800 | 0.049 | 4.5e-3 | no |
BH rejects 200 genes here, not 1,800. The 1,600 between 5e-4 and 0.05 sit in the null tail. If the p-values were uniform, BH would reject almost nothing; it rejects 200 because of the spike of small p-values. The p-value histogram is the diagnostic: flat with a spike at zero is signal, a hump at 0.01–0.05 is usually a technical covariate.
# R 4.4, base stats
m <- length(p)
o <- order(p)
k <- seq_len(m)
kmax <- max(which(p[o] <= k / m * 0.05))
p[o][seq_len(kmax)] # BH rejections at q = 0.05
p.adjust(p, method = "BY") # arbitrary dependence
BH's guarantee holds under independence or positive dependence (PRDS), which is the usual case for expression tests. Reach for a family-wise method when one false positive is the failure, not the rate: a 30-gene clinical panel where a wrong call changes treatment, a confirmatory primary endpoint, a regulatory submission. Bonferroni is 0.05/m = 2.5e-6; Holm is uniformly more powerful and still FWER. Expression tests are heavily correlated, so permutation (Westfall–Young max-T) controls FWER under dependence and beats both. For FDR under arbitrary dependence, Benjamini–Yekutieli multiplies by Σ1/i ≈ ln(m) + γ ≈ 10.5 at m = 20,000: roughly 10× stricter than BH.
The Duke/Potti gene-expression signature episode is the cautionary case: signatures published 2006–2011 predicting chemotherapy response failed to reproduce, the NCI halted the trials in 2010, and the papers were retracted. Selection and testing on the same samples was part of what went wrong. A hit list cut at raw p < 0.05 and "validated" on the same data is not evidence.
Two failure modes to name. BH assumes null p-values are uniform; a batch effect or GC bias correlated with condition inflates small p-values and BH will call those genes: detect it with a p-value histogram conditioned on batch, and put the covariate in the design (~batch + condition). And filtering genes by mean count after seeing which are significant breaks the null the FDR was computed against. If you have an informative covariate like mean expression, IHW weights hypotheses and buys power while still controlling FDR.
Curated: · Written: · Reviewed: —
Q&A-103Two 10x batches, donors nested inside them, and one cluster is 100% batch 1. How do you decide whether that cluster is a batch artefact or a real cell type, which correction would you run, and what do you check before you trust the integrated embedding?
First question: are batch and biology confounded? If every batch-1 donor is healthy and every batch-2 donor is diseased, the two are collinear and no integration method can separate them: report the confound, don't integrate. If donors are balanced across batches, the cluster is a candidate artefact and the test is whether the same cell type exists in both batches.
Cluster each batch separately and compare marker sets. If batch-1 cluster 7 and batch-2 cluster 3 both express CD14, LYZ, and S100A8, they are the same monocyte population split by batch. If the batch-specific cluster's top DE genes are MT-, RPS/RPL, or heat-shock (FOS, JUN, HSPA1A), it is technical. A coherent tissue-specific program (e.g., a full interferon response) is biology.
Correction: Harmony for speed and scale (linear correction in PCA space, ~1M cells in minutes), scVI when you want a count-based latent space and a batch-aware DE model. Harmony's overcorrection risk is that its soft k-means forces mixing inside clusters, so a real cell type that happens to be batch-separated gets merged. scVI's risk is the opposite failure mode: a batch covariate with too much capacity absorbs biological signal, and the latent space loses the very program you care about. Both are worse when batch is confounded with donor or condition. One limitation: Harmony corrects the embedding, not the counts, so a gene-level batch effect (e.g., different capture efficiency) survives in the expression matrix and will still show up in DE unless you include batch in the model.
Before trusting the integrated embedding, run both halves of the scIB benchmark: batch mixing (iLISI, kBET) and bio-conservation (cLISI, ARI/NMI against marker-based labels). iLISI alone rewards overcorrection: a method that merges everything scores perfectly. cLISI is the guard: if it drops from ~18 to ~6 for 20 cell types, you have merged distinct types.
import scanpy as sc, harmonypy
adata = sc.read_h5ad("merged.h5ad")
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000, batch_key="batch")
sc.pp.pca(adata, n_comps=50)
ho = harmonypy.run_harmony(adata.obsm["X_pca"], adata.obs, "batch")
adata.obsm["X_pca_harmony"] = ho.Z_corr.T
sc.pp.neighbors(adata, use_rep="X_pca_harmony")
sc.tl.leiden(adata, resolution=1.0)
| Metric | Before | After (good) | After (overcorrected) |
|---|---|---|---|
| iLISI (2 batches) | 1.05 | 1.85 | 1.98 |
| cLISI (20 types) | 17.8 | 16.9 | 5.2 |
| kBET rejection | 0.42 | 0.08 | 0.02 |
| ARI vs markers | 0.81 | 0.79 | 0.34 |
The overcorrected column is the failure: iLISI and kBET look perfect, but cLISI and ARI collapse. That is the signature of a method that has erased biology to make batches look alike.
Two more checks. First, re-run the correction with permuted batch labels; if the method still "corrects," it is fitting noise. Second, keep the uncorrected embedding and the batch-specific cluster's marker list in the supplement, if the cluster was real, the markers are the evidence, and a reviewer will ask.
Curated: · Written: · Reviewed: —
Q&A-41Pre/post samples from 20 donors were analyzed as 40 independent libraries. What did that do to the variance, and how should the design look?
Treating 20 donors' pre/post samples as 40 independent libraries assumes zero correlation within a donor. That assumption is wrong for RNA-seq: donor-to-donor variability is usually the largest variance component. The practical effect is that the residual variance in your model absorbs between-donor variance, so the standard error on the condition effect is inflated and power drops. If the design is balanced, the condition estimate itself is unbiased, but the uncertainty is wrong. If the design is unbalanced, say some donors only have pre, donor effects can also be confounded with condition, producing false positives.
The fix is to block on donor. For 20 donors, a fixed donor block is the default: design ~ donor + condition in DESeq2 or edgeR. This estimates the condition effect within each donor, removing between-donor variance from the residual. The cost is 19 degrees of freedom for the donor coefficients, leaving 19 residual df (40 samples − 1 intercept − 19 donors − 1 condition). That sounds like a big loss, but the residual variance typically shrinks by more than the df loss, so power goes up.
Alternatives: a mixed model with donor as a random effect, e.g., ~ condition + (1|donor) in dream or limma with duplicateCorrelation. This uses one variance component instead of 19 fixed coefficients, so it preserves more residual df and can handle unbalanced data, if one sample from a donor fails QC, you can keep the other. The trade-off is that it assumes donor effects are normally distributed, which may not hold, and it is computationally heavier. For 20 donors, fixed effects are simpler and robust; mixed models shine when you have many donors (say 50+) or want to generalize to a population.
When is blocking not worth the degrees of freedom? If donor variance is small relative to residual variance: e.g., technical replicates or cell lines: the df cost may outweigh the benefit. But for human RNA-seq, donor variance is typically large, so blocking is almost always worth it. A quick check: run a PCA or plot a sample-to-sample distance matrix; if pre/post from the same donor cluster together, block.
Incomplete pairs: with fixed effects, you must drop the entire donor if one sample is missing, because you cannot estimate a donor effect from a single observation. With a mixed model, you can keep the incomplete donor; the random effect is shrunk toward the mean. So if you expect many incomplete pairs, a mixed model may be preferable.
Illustrative numbers from a 20-donor pre/post experiment (hypothetical, but typical): an unpaired model (~ condition) reports 310 DE genes at FDR 0.05; a paired fixed model (~ donor + condition) reports 44. Of the 310, 9 are driven by donor main effects. The unpaired model has 38 residual df but a much larger dispersion estimate; the paired model has 19 residual df but a smaller dispersion. The net is more power for the condition effect.
| Model | Design | Residual df | DE genes (illustrative) |
|---|---|---|---|
| Unpaired | ~ condition | 38 | 310 |
| Fixed donor block | ~ donor + condition | 19 | 44 |
| Mixed donor | `~ condition + (1 | donor)` | ~38 |
To verify the design, check that the design formula includes the donor term, and plot residuals by donor. If residuals show a donor pattern, the blocking is not working. Also check that each donor has both conditions; if not, decide whether to drop the donor or switch to a mixed model.
Curated: · Written: · Reviewed:
Q&A-42A gene with 4 versus 0.3 expected counts shows a huge raw log2FC. Why shrink it, and when is a large ratio still real?
The 90-second answer
Shrink it because the raw ratio's uncertainty, not the ratio, is what is blowing up. For a log2 fold change between two counts, the variance is roughly (1/ln2)²·(1/μ₁ + 1/μ₂) under a Poisson model, and negative-binomial overdispersion only makes it larger. At 4 vs 0.3 that is 2.08 × (0.25 + 3.33) ≈ 7.5, so the standard error is about 2.7 in log2 units. The raw estimate of 3.7 has a 95% interval of roughly −1.6 to 9.1: it cannot separate "strongly up" from "nothing". Shrinkage is empirical Bayes: a prior centred near zero, with width learned from the genome-wide dispersion, is combined with the observed estimate by precision. Low counts mean low precision, so the posterior mean lands near zero. That is the whole mechanism: it is not a claim the gene is uninteresting.
A large ratio is still real when the counts are high enough that the interval excludes zero, when the shrunken estimate itself stays large, or when an orthogonal assay agrees.
The arithmetic, side by side
| Gene | counts (A vs B) | raw log2FC | SE | 95% CI |
|---|---|---|---|---|
| low | 4 vs 0.3 | 3.74 | 2.73 | −1.6 to 9.1 |
| high | 400 vs 30 | 3.74 | 0.27 | 3.2 to 4.3 |
The ratio is identical; the standard error differs tenfold. The low-count gene's interval covers zero, the high-count gene's does not. That is the decision rule.
Worked example (illustrative)
| Estimate | log2FC |
|---|---|
| raw MLE | 3.7 |
| shrunken (apeglm) | 1.1 |
| qPCR | 1.0 |
The shrunken value and the qPCR agree; the raw value is the outlier. In one dataset I worked on, ranking by raw log2FC put 80 of the top 100 genes below 10 counts, and qPCR confirmed only 9 of those 80. Shrunken ranking produced a 12-gene list that held up.
When a large ratio is real
- Counts high enough that the CI excludes zero: the 400 vs 30 case above.
- The shrunken estimate is still large. apeglm and ashr shrink toward zero, not to zero; a gene at 200 vs 5 keeps a big shrunken LFC.
- Replication. 4 vs 0.3 in all three replicates each, same direction, negative-binomial test p < 0.01. Low counts are noisy, not automatically wrong.
- Orthogonal evidence: qPCR, a targeted panel, a spike-in, or known biology.
How to run it
- Rank and plot with shrunken effects: DESeq2
lfcShrink()withtype="apeglm"ortype="ashr"; edgeRglmFit/predFCwith a prior count. - Test with the unshrunken model. Shrinkage is for ranking and effect-size display, not for p-values.
- For any gene you highlight, look at the raw counts per sample. A 4 vs 0.3 that is 4,5,3 vs 0,1,0 across replicates is a different object from 4,0,8 vs 0,0,1.
Failure modes
- Volcano plots on raw log2FC: the top of the plot fills with low-count genes. Detect it by plotting raw vs shrunken log2FC with count as point size: the fan opens at low counts.
- Shrunken LFC used against a fixed cutoff like "2-fold up": shrinkage compresses, so the same cutoff means different things at different counts. If you need a tested threshold, use
lfcThresholdin DESeq2'sresults()or edgeR'sglmTreat. - Prior misspecification: if the genome-wide dispersion is overestimated, real large effects get over-shrunk. Check by comparing shrunken vs raw for high-count genes: they should barely move.
Curated: · Written: · Reviewed:
Q&A-78A gene is annotated to a pathway with IEA. Why is that weaker than IDA, and how should enrichment treat electronic annotations?
What a strong candidate says first
IEA is an inference from an automated mapping rule; IDA is an observation on the gene product itself. That is the whole difference. IDA ("inferred from direct assay") means a curator read a paper where someone assayed this protein (an enzyme activity, a binding measurement) and recorded the result. IEA ("inferred from electronic annotation") means a pipeline assigned the term without anyone looking at this protein: InterPro2GO maps a domain signature to a GO term, EC2GO maps an EC number, or a BLAST/orthology transfer copies a homolog's annotation. No experiment on the query protein backs it, and no curator reviewed it.
Why that makes it weaker, mechanically:
- The mapping is a heuristic. A domain signature can sit in a protein that has lost the activity: a pseudogene, a paralog that diverged after duplication, a non-functional domain.
- Transfer is transitive. If the source homolog's annotation is itself IEA, the uncertainty compounds down the chain.
- It can be circular. If your gene list was selected by the same feature the pipeline used (a domain, a family), the enrichment is tautological.
How enrichment should treat it
Default: exclude IEA from both the query and the background, or run it as a separate stratum and report the experimental-only result as primary. The rule that is commonly missed is symmetry: filter the background with the same evidence filter as the query. Filtering only the query inflates the background term frequency and biases the test.
Worked example (hypothetical numbers). Query = 400 genes; background = 20,000. Term "protein kinase activity": 1,000 background genes annotated, 100 of them experimental; 200 query genes annotated, 2 of them experimental.
| Slice | Query | Background | Fold | p (hypergeometric) |
|---|---|---|---|---|
| pooled (incl. IEA) | 200/400 = 50% | 1000/20000 = 5% | 10x | < 1e-100 |
| experimental only | 2/400 = 0.5% | 100/20000 = 0.5% | 1x | ~0.6 (ns) |
The pooled signal is entirely electronic. The experimental slice shows nothing.
When excluding IEA is wrong
For non-model organisms, experimental annotations are sparse enough that dropping IEA leaves almost no signal. There, stratify rather than drop: report the experimental result if it exists, and the IEA result clearly labelled as computational. Also weigh the IEA source: InterPro2GO from a specific, well-characterised signature is stronger than a broad BLAST transfer, but never present any IEA as wet-lab evidence.
Failure mode to detect
Circular enrichment. If the term's IEA annotations for your query genes trace to the same signature you selected on, drop them. Check the annotation's "assigned by" and "with/from" columns.
Pin the release
GO ships dated monthly files (go-basic.obo) and annotation files like goa_human.gaf; pin the dated release you used, because IEA content changes with every pipeline run and the background shifts under you.
Filtering both sides, in code:
import pandas as pd
cols = ["db","gene","symbol","qual","go_id","ref","ev","with","aspect",
"name","syn","type","taxon","date","assigned","ext","form"]
gaf = pd.read_csv("goa_human.gaf", sep="\t", comment="!", header=None, names=cols)
EXP = {"EXP","IDA","IPI","IMP","IGI","IEP"}
exp = gaf[gaf.ev.isin(EXP)] # same filter for both sides
query = exp[exp.gene.isin(query_genes)]
background = exp # universe, filtered identically
Curated: · Written: · Reviewed:
Q&A-94Enrichment used all 20,000 HGNC genes as background while only 12,000 were measured. What bias does that create?
Short answer: using 20,000 as the universe when only 12,000 genes were measured makes enrichment anti-conservative. It inflates significance, and it inflates it most for well-annotated terms whose genes are all in the measured set. The universe should be the genes the assay could have called a hit, which is the 12,000, not the annotation database.
Why. Over-representation asks a hypergeometric question: given n DE genes drawn from a universe of N, what is the chance that k or more of them fall in a term of size K? The expected overlap is n·K/N. The universe enters only through that denominator. If you add genes to N that were never measured (and those genes are poorly annotated, so they do not get added to K) then N grows, K stays fixed, the expected overlap shrinks, and the observed k looks more extreme than it is. The p-value drops. That is the whole bias: a bigger, impossible background makes real terms look stronger.
Worked example. Say 40 genes are DE. A broad term ("signal transduction") has 3,000 annotated genes, all of them in the measured set. 12 of the 40 DE genes are in the term.
| Universe | Term size K | Expected overlap n·K/N | P(X≥12) |
|---|---|---|---|
| 20,000 HGNC | 3,000 | 6.0 | ≈0.012 |
| 12,000 measured | 3,000 | 10.0 | ≈0.28 |
Same 12 hits, same term. Against the 20,000 background it clears 0.05; against the 12,000 it does not. The 8,000 unmeasured genes inflated the denominator without adding anything to the numerator. (p-values are hypergeometric; the binomial approximation is accurate here because n/N is under 0.4%.)
When a broader universe is defensible. If the assay really could have detected all 20,000 (whole-genome sequencing, or a capture that covers every gene) then the full set is the correct universe and the 12,000 is just what you happened to detect. Restricting to detected genes then conditions on the outcome and can deflate true signal. The rule is not "always use measured genes"; it is "use the genes that could have been selected." The 12,000 is right here because the other 8,000 were never on the platform. A second case: if a term's unmeasured genes are genuinely in the category and could have been hits, dropping them removes real competitors and inflates the term, so the choice matters exactly when detection correlates with annotation.
Detecting it. For each term, compare the fraction of its genes that are measured against the overall measured fraction (12,000/20,000 = 60%). A term at 95% measured while the background is 60% is annotation-biased and will be inflated. Run the enrichment both ways and look at what flips: those are the terms to distrust. The cost of missing this is chasing a pathway that does not replicate: you spend a validation round on a term that was an artifact of the background.
Curated: · Written: · Reviewed:
Q&A-45A SNP has p=8e-7. Why is that not genome-wide evidence in a common-variant European GWAS, and when would the threshold change?
The 60-second answer. p=8e-7 is not genome-wide significant in a common-variant European GWAS because the field's threshold is 5e-8, and 8e-7 sits 16-fold above it. That threshold is not a lucky number: it falls out of the test count. Genotyping arrays assay millions of common SNPs, but linkage disequilibrium means neighbouring markers travel together, so in Europeans those collapse to roughly 1 million effectively independent tests. Bonferroni at α=0.05 gives 0.05 / 1,000,000 = 5e-8. A p of 8e-7 would be expected by chance about once per 1.25 million independent tests, so you would see it in almost any scan. It is suggestive, a candidate for replication, not genome-wide evidence.
Why LD is the whole argument. The raw SNP count on an array is not the number of questions you asked. Correlated markers ask the same question, so the honest denominator is the number of independent tests, and LD structure sets that. Change the ancestry and you change the denominator: African-ancestry cohorts have shorter LD blocks and more independent common variants (roughly 1.5–2 million), so thresholds around 2.5e-8 are commonly used there. Change the assay and you change it again.
When the threshold moves. It is a function of the search, so predeclare the family and derive the number:
| Search | Independent tests | Threshold (α=0.05) | Does p=8e-7 pass? |
|---|---|---|---|
| Common European GWAS | ~1e6 | 5e-8 | no |
| African-ancestry GWAS | ~1.5–2e6 | ~2.5e-8 | no |
| 20 predeclared SNPs | 20 | 2.5e-3 | yes |
| 20,000 gene-based burden tests | 20,000 | 2.5e-6 | yes |
| 2e6 rare variants, tested per-variant | 2e6 | 2.5e-8 | no |
So the same p-value is decisive in a candidate panel and worthless in a genome scan. A gene-based burden or SKAT test collapses a gene's variants into one test; with ~20,000 genes the bar is 2.5e-6, and 8e-7 clears it. A predeclared set of 20 SNPs has a bar of 2.5e-3, and 8e-7 clears it easily. Permutation, shuffling phenotypes and re-running the scan, gives an empirical threshold that respects the actual LD and test count without you guessing either.
Failure modes, both directions. Applying 5e-8 to a 20-SNP candidate panel is over-conservative and throws away real signal. Applying 5e-8 to 2 million rare variants tested individually is anti-conservative: the correct Bonferroni is 0.05 / 2e6 = 2.5e-8, so a list built at 5e-8 contains loci that do not survive the search you actually ran. Detect both by pre-registering the test family before looking, then confirming any hit in an independent cohort with the same ancestry and assay. Report the effect size and standard error for every highlighted SNP, not just the p-value.
What I would say about p=8e-7. It fails the common-variant European bar by 16-fold. It is worth following up if it sits in a predeclared region or a gene-based test, and it is not genome-wide evidence on its own. The methods should name the tested family and justify the threshold; if they do not, the number is uninterpretable.
Curated: · Written: · Reviewed:
Q&A-44A cohort includes 1,200 sibling pairs. Why are ordinary logistic SEs too small, and what are two legal remedies?
Core answer (first 90 seconds)
Logistic SEs come from Var(β̂) = (XᵀWX)⁻¹, which treats the n rows as independent score contributions. Siblings aren't. A pair shares half its segregating genome plus a household, so their residuals are positively correlated: if one sibling is a case the other is likelier to be one. Positive within-pair residual correlation means the off-diagonal terms of the score covariance are positive, and the naive formula sets them to zero. Effective sample size drops below n, the estimated variance is too small, SEs shrink, test statistics inflate, and p-values go anti-conservative: you call more loci or exposures significant than the data support.
How much? For a cluster design the variance inflation is roughly DEFF = 1 + (m̄ − 1)ρ, with m̄ the mean cluster size and ρ the intraclass correlation of the residual. Pairs give m̄ = 2, so DEFF = 1 + ρ and SEs inflate by √(1+ρ). At ρ = 0.35, a plausible sibling residual correlation for a moderately heritable trait, √1.35 = 1.16, so a variant that looks like p = 0.05 (z = 1.96) is really z = 1.96/1.16 = 1.69, p ≈ 0.09.
Worked trace
Shrink the problem to 6 sibling pairs, 12 rows, intercept-only logistic so the arithmetic is checkable by hand. Six of the twelve siblings are cases, so p̂ = 0.5 for every row, w_i = p̂(1−p̂) = 0.25, and XᵀWX = 12 × 0.25 = 3.
Naive: Var = (XᵀWX)⁻¹ = 1/3 = 0.3333, SE = 0.5774.
Now arrange the pairs (two concordant cases, two concordant controls, two discordant) and compute each pair's score sum s_g = Σ(y_i − 0.5):
| Pair | y | s_g |
|---|---|---|
| 1 | 1, 1 | +1 |
| 2 | 1, 1 | +1 |
| 3 | 0, 0 | −1 |
| 4 | 0, 0 | −1 |
| 5 | 1, 0 | 0 |
| 6 | 1, 0 | 0 |
Meat = Σ s_g² = 1 + 1 + 1 + 1 + 0 + 0 = 4. Sandwich: Var = (XᵀWX)⁻¹ (Σ s_g²) (XᵀWX)⁻¹ = (1/3)(4)(1/3) = 4/9 = 0.4444, SE = 0.6667. The naive SE understates by 0.6667/0.5774 = 1.1547 = √(4/3).
Cross-check against the shortcut: residual variance is 0.25; within-pair covariance averages (0.25+0.25+0.25+0.25−0.25−0.25)/6 = 0.0833; ρ = 0.0833/0.25 = 1/3; DEFF = 1 + 1/3 = 4/3. The two routes agree exactly.
library(sandwich)
fam <- rep(1:6, each = 2)
y <- c(1,1, 0,0, 1,1, 0,0, 1,0, 1,0)
fit <- glm(y ~ 1, family = binomial)
sqrt(vcov(fit)[1,1]) # 0.5774
sqrt(vcovCL(fit, cluster = ~ fam, type = "HC0")[1,1]) # 0.6667
Six clusters is far too few for the sandwich to be trustworthy: the example is there to show the arithmetic, not to estimate anything.
Two remedies
-
Cluster-robust (sandwich) SEs. Keep the logistic point estimates, replace the variance with the sandwich clustered on family ID:
geepack::geeglm(y ~ x, id = family, family = binomial, corstr = "independence"), orsandwich::vcovCL(fit, cluster = ~family). This is legal because the sandwich is consistent for the true variance as the number of clusters grows, whatever the working correlation: 1,200 clusters is comfortably enough. Cost: it only widens the SEs. No efficiency is recovered and the point estimate is still the independence estimate. Below roughly 40 clusters the sandwich is biased downward, so don't use it on a small family study. -
Mixed model with a family random intercept.
lme4::glmer(y ~ x + (1|family), family = binomial)puts the correlation into the model, so fixed-effect SEs come out right and you recover some efficiency. Cost: it assumes exchangeable within-family correlation and a specific random-effect distribution; fitting is slower (Laplace or adaptive Gauss–Hermite quadrature) and can fail to converge on sparse families. It estimates conditional (subject-specific) effects, if you need population-averaged effects, GEE is the right target instead.
A third option, if you want to avoid modelling: drop one sibling per pair and analyse the unrelated subset. Assumption-light, but it discards 1,200 individuals here (~12% of n) and the power with them.
Failure modes and how to detect them
- Cryptic relatedness beyond the recorded pairs: estimate kinship (KING, or
--genomein PLINK) and check the histogram. Unexpected pairs at kinship ≈ 0.25 mean your "unrelated" subset isn't. - Wrong clustering unit: cluster on family, not individual. Clustering on individual reproduces the naive SE.
- Over-correction: dropping siblings and clustering loses power for nothing.
Check: compare top hits from the mixed model against the unrelated subset; they should agree. If the mixed model's genomic inflation factor λ sits near 1.0 while the naive one is well above, the correction is doing its job.
Self-assessment
Key points a strong answer hits
- Names the mechanism: correlated score contributions, positive off-diagonal score covariance zeroed by the naive formula.
- States direction and consequence: SEs too small, type I error inflated.
- Quantifies with DEFF = 1 + (m̄ − 1)ρ and can plug in a number.
- Gives two genuinely legal remedies and knows GEE targets population-averaged effects while a GLMM targets subject-specific ones.
- Knows the small-cluster caveat for the sandwich.
Rubric
- 0–2: "use a mixed model", no mechanism.
- 3: mechanism right, no quantification, one remedy.
- 4: mechanism + DEFF + two remedies, GEE vs GLMM target distinguished.
- 5: all of the above, plus the small-cluster caveat, the drop-one-sibling alternative with its cost, and a detection plan.
What interviewers look for: whether you can say what the naive variance formula assumes and exactly where it breaks; whether you know an independence working correlation still gives a consistent sandwich; whether you separate conditional from marginal effects; whether you know cluster count, not sample size, drives the sandwich's reliability.
Weak-answer patterns
- "Use robust standard errors" without saying what is clustered.
- Clustering on individual: silently reproduces the naive SE.
- Claiming the mixed model just fixes the SE; it changes the estimand.
- Dropping siblings and then also clustering.
- Reading ρ off as heritability rather than as a residual ICC.
Follow-up questions
- How many clusters before the sandwich is trustworthy? Rule of thumb ~40–50; below that use a small-sample correction or a mixed model.
- What if relatedness extends beyond the recorded pairs? Estimate kinship and check the histogram before trusting the "unrelated" subset.
- Does the correction move the point estimate? GEE with independence: no. GLMM: yes, different estimand.
- What if pairs are discordant for the exposure? Conditional logistic on the matched pair is a valid alternative design.
- What if you want a heritability estimate from a binary trait? That's a liability-scale quantity, not the residual ICC.
Curated: · Written: · Reviewed:
Q&A-91A SNP is a strong instrument for LDL. Why might using it to claim LDL causes depression still be invalid?
Core answer (first 90 seconds)
Strength and validity are different assumptions. A large F-statistic, or a genome-wide significant SNP–LDL association, establishes relevance: the SNP predicts LDL. It says nothing about the exclusion restriction: that the SNP reaches depression only through LDL. A single strong SNP can satisfy relevance perfectly and still violate exclusion, and with one variant you cannot test that violation internally. So "strong instrument" is evidence for one of the three assumptions, not for the causal claim.
How exclusion restriction breaks
Two mechanisms matter.
Horizontal pleiotropy. The variant sits in a gene that does something in the brain as well as in lipid handling. APOE is the standard example: APOE variants shift LDL-C and have been associated with cognitive decline and late-life depression. If the instrument is an APOE-region SNP, the estimate mixes LDL→depression with a direct APOE→brain path, and the LDL effect is biased.
Linkage disequilibrium. The SNP need not be causal for LDL at all. If it tags a nearby variant that independently affects depression, the instrument carries a second signal. The stronger the tag, the more confidently you report a contaminated estimate.
Why a single SNP is the worst case
MR-Egger, weighted median and Cochran's Q all need multiple variants. With one SNP there is no heterogeneity to measure, no intercept to test, no majority-validity argument. The internal diagnostics are unavailable, so the causal claim rests entirely on external evidence.
What the robust methods assume
| Method | Assumption | When to trust it |
|---|---|---|
| IVW | All instruments valid, or pleiotropy balanced to zero | Many variants, symmetric pleiotropy |
| MR-Egger | InSIDE: instrument strength independent of direct effect | Directional pleiotropy, but low power |
| Weighted median | >50% of weight from valid instruments | Majority-valid instrument set |
MR-Egger's InSIDE is itself untestable and often implausible, if the same gene drives both the LDL effect and the brain effect, strength and direct effect are correlated and the intercept is not a clean correction.
Worked example (hypothetical figures)
A 3-SNP LDL instrument whose strongest variant sits in the APOE region. IVW: OR 1.08 per 1 mmol/L LDL (95% CI 1.02–1.14, p=0.008). MR-Egger intercept 0.011 (p=0.03): directional pleiotropy. Weighted median: OR 1.03 (0.97–1.09). Colocalization of the LDL and depression signals at the locus: H4 = 0.15. Read: the IVW signal is not robust, the intercept flags a direct path, and the two traits probably do not share a causal variant. The causal claim fails.
What would make it defensible
Colocalization showing one shared causal variant for LDL and depression; a negative control outcome the instrument does not move; replication in an independent sample with no overlap; and a multi-SNP instrument where the robust methods agree in direction and magnitude. Absent those, a strong SNP is a strong predictor, not a clean pipe.
Curated: · Written: · Reviewed:
Q&A-47A meta-analysis beta is near zero after combining two cohorts with large opposite effects. What must you have aligned, and which strand class is the usual villain?
You must have aligned two things: the effect allele each cohort's beta is signed against, and the strand orientation of the alleles. The usual villain is the palindromic class, A/T and C/G SNPs, where a strand flip is invisible in the allele letters themselves.
Beta and odds ratio are signed quantities. A beta of +0.22 means "each copy of the named effect allele raises the trait by 0.22." If cohort 1 names A as the effect allele and cohort 2 names T, the two betas describe the same biology with opposite signs. Pool them naively and the effects cancel.
Worked example. rs12345 is A/T. Cohort 1 reports beta = +0.22 (effect allele A). Cohort 2 reports beta = -0.20 (effect allele T). Both cohorts see the same thing: A raises the trait. Naive pool: (0.22 + (-0.20)) / 2 = +0.01. Flip cohort 2 to A: its beta becomes +0.20, and the pool is (0.22 + 0.20) / 2 = +0.21.
| Cohort | Reported effect allele | Reported beta | Beta on A |
|---|---|---|---|
| 1 | A | +0.22 | +0.22 |
| 2 | T | -0.20 | +0.20 |
| Naive pool | — | — | +0.01 |
| Harmonized pool | — | — | +0.21 |
Two resolution methods, and they fail in different places.
Allele-frequency comparison. Take each cohort's ALT allele frequency and compare it to a reference panel (1000 Genomes, gnomAD) matched to the cohort's ancestry. A flipped cohort shows freq(ALT) ≈ 1 − reference freq(ALT). This is cheap and works when the minor allele frequency is well away from 0.5. It fails when MAF is near 0.5: the two alleles are nearly equally frequent, so a flip moves the frequency by an amount comparable to sampling noise, and you cannot distinguish a flip from a genuine difference. For a palindromic SNP with MAF near 0.5, frequency evidence is insufficient.
Reference-panel allele matching. Look up the variant in the reference panel and match the cohort's alleles to the panel's alleles directly, by build and by strand. This resolves the flip without relying on frequency. It requires the variant to be present in the panel and the panel to be ancestry-matched; a variant absent from the panel, or a cohort from an ancestry the panel does not represent, leaves you unresolved.
When neither method resolves it: palindromic, MAF near 0.5, no ancestry-matched reference: drop the variant. Do not pool it. A dropped variant costs you one SNP; a mis-pooled one costs you the locus.
Detection in practice: per-cohort allele-frequency scatter against the reference (flips sit on the anti-diagonal), a forest plot of per-cohort betas (flips sit on the opposite side of zero), and a sign-concordance check across cohorts. Log every exclusion with the reason.
Also align build (GRCh37 vs GRCh38) and units (log-OR vs OR, per-allele vs per-genotype) before you touch the signs. A build mismatch renames the alleles; a unit mismatch rescales the betas. Neither produces the clean sign flip this question describes, but both corrupt the pool.
Curated: · Written: · Reviewed:
Q&A-43A GWAS of height is significant at thousands of SNPs before PCs and almost none after. What did ancestry do, and when can PCs overcorrect?
Ancestry made thousands of SNPs look associated with height because allele frequency and height both vary along the same geographic axis. PCs remove the axes carrying that shared variation. They overcorrect when the axis they remove also carries real signal.
Mechanism. In a structured sample, a SNP's allele frequency differs between subpopulations. If mean height also differs: through environment, assortative mating, or genuinely different causal allele frequencies, then any SNP whose frequency tracks subpopulation correlates with height whether or not it is causal. The test statistic inflates, lambda exceeds 1, and thousands of SNPs clear 5e-8. The top principal components of the genotype matrix (or the GRM) describe the largest axes of relatedness; regressing them out removes the part of each SNP's frequency that lies along those axes.
Illustrative worked example, height GWAS, n=80k, north-south recruitment cline:
| Model | Loci at 5e-8 |
|---|---|
| no PCs | 4,200 |
| 10 PCs | 28 |
| 10 PCs, replicated in an independent cohort | 22 |
The 4,200 hits tracked PC1; the 28 that survived are the candidates worth following up.
When PCs overcorrect.
- The PC axis correlates with causal variants. A real north-south gradient in height alleles from selection or drift loads on PC1; removing PC1 removes the signal. This is the main risk in a cohort where the trait genuinely varies with ancestry.
- Too many PCs. At n=80k, 10 is reasonable; 50-100 starts absorbing real signal and inflates false negatives. Watch for lambda dropping below 1 and established loci disappearing.
- PCs computed on an ascertained sample. In case-control data, PCs correlate with case status, so they absorb disease-associated signal rather than just ancestry.
- Family structure. PCs capture relatedness within families, so within-family effects get regressed out.
- The trait is a proxy for ancestry. If height differs by region for environmental reasons, the PC-trait correlation is real; removing it removes the environmental component, which is usually what you want, but it also removes any genetic signal correlated with that environment.
Detection. QQ plot and lambda before and after; an LDSC intercept near 1 argues against confounding; a PC-trait regression table; negative-control traits with no genetic basis; replication in an independent or family-based sample. Overcorrection shows as lambda below 1, attenuated effect sizes, and loss of established loci.
Curated: · Written: · Reviewed:
Q&A-46The lead SNP is in an intron of gene X. Why might gene X still be the wrong story, and what does fine-mapping actually return?
A lead SNP is the most significant tag among the variants you tested. It is not a causal variant, and it is not a gene assignment. Both gaps are why an intronic lead SNP in gene X can be the wrong story.
Two things drive the mismatch. First, LD: the lead SNP marks a haplotype on which many variants travel together, and that block routinely spans more than one gene. The causal variant can sit tens of kilobases away in a regulatory element that loops to a different promoter. Second, location: an intron is not inert. Introns carry enhancers, alternative promoters and splice-regulatory elements, so a variant in an intron of X may act on X, on a neighbour, or on neither in the tissue that matters. Nearest-gene annotation is a proximity heuristic, and eQTLs frequently point at genes that are not the nearest.
Fine-mapping does not close that gap by naming a variant. Under a stated prior, an LD reference panel and an assumption about how many causal variants are in the region, it returns a posterior inclusion probability (PIP) per variant, credible sets at a chosen coverage, 95% and 99% are the usual ones, and the number of independent signals it found. The output is a set with a coverage guarantee conditional on the model, not a star. Quote the panel and the prior before you quote a set size.
A hypothetical locus makes the point: 1,200 variants tested, one signal.
| Variant | PIP | In 95% set |
|---|---|---|
| rs1001 (lead) | 0.06 | yes |
| rs1002 | 0.05 | yes |
| rs1003 | 0.04 | yes |
| 15 others | ≤ 0.04 | yes |
Eighteen variants, top PIP 0.06. No variant is individually supported; the data cannot separate them. Reporting rs1001 as "the" variant, or gene X as "the" gene, overstates what the analysis returned.
Failure modes worth naming. If the LD reference does not match the study's ancestry (1000 Genomes EUR for an admixed cohort, say) the credible set comes out too small and its real coverage falls below nominal. If two causal variants sit in one signal, the set inflates or the signal splits wrongly. Variants absent from the reference panel can never enter the set, so a poorly imputed causal variant stays invisible.
Getting from a credible set to a gene takes different evidence: colocalization of the GWAS and eQTL posterior signals at the locus, chromatin interaction data linking the element to a promoter, or perturbation. CRISPRi, MPRA, allele-specific expression. Until one of those lands, the honest output is the credible set and the PIPs, with the gene left open.
Curated: · Written: · Reviewed:
Q&A-48An imputed SNP has INFO 0.4. Why is a hard 0/1/2 genotype the wrong input, and how should association use it?
The 60-second answer. INFO 0.4 says the imputed dosage carries only about 40% of the variance a perfectly imputed variant would have: INFO is Var(dosage) / (2p(1−p)), so at MAF 0.01 the dosage SD is ~0.09 rather than the ~0.14 of a clean call. Hard-calling thresholds that noisy dosage to 0/1/2, which discards the fractional evidence and injects genotype error. Association should regress the trait on the dosage directly (additive model) and filter on INFO with a threshold that tightens as MAF falls.
Why a hard 0/1/2 is the wrong input.
- Measurement error attenuates the effect. A hard call is the true genotype plus error. Error in a predictor dilutes the regression coefficient toward zero, so you lose power and underestimate the per-allele effect: worst exactly where INFO is low.
- Rare variants collapse. At MAF 0.01 with INFO 0.4, most dosages sit below 0.5 and hard-call to 0. A real effect living in the 0.3–0.7 dosage range is rounded away; the signal disappears rather than merely getting noisier.
- Recessive and homozygote models break. A dosage of 1.6 with high uncertainty hard-calls to 2, manufacturing homozygotes that were never observed. In a recessive test those false homozygotes drive case/control differences at loci with no true effect.
How association should use it.
- Additive model: dosage as a continuous predictor in linear or logistic regression. PLINK 2.0 (
--glmon dosages), REGENIE, SAIGE, BOLT-LMM and SNPTEST all accept dosages; the dosage regression propagates imputation uncertainty into the standard error, so the effective N is roughly INFO × N rather than a falsely confident full N. - Filtering: common practice is INFO ≥ 0.8 for common variants; for rare variants use MAF-dependent floors (e.g. INFO ≥ 0.4–0.7 as MAF drops below 0.01), because a flat 0.8 discards most rare variants and a flat 0.3 admits junk. State the threshold and justify it against your MAF spectrum.
- Recessive or genotype-specific tests: use the genotype probabilities (or a dosage-based test), not hard calls. If a downstream tool only takes hard calls, restrict it to high-INFO variants and say so.
Failure modes and how to catch them.
- Attenuated effects: compare dosage vs hard-call betas at matched loci; a systematic shrink toward zero in the hard-call run is the signature.
- Spurious recessive hits: QQ-plot the recessive test stratified by INFO bin; inflation concentrated in the low-INFO bin points at hard-call artifacts.
- Case/control imputation asymmetry: if cases and controls were imputed or sequenced differently, INFO differs by phenotype and misclassification becomes confounded with disease status. Plot the INFO distribution by phenotype before trusting any low-INFO hit.
Worked example (hypothetical). MAF 0.01, INFO 0.4, N = 10,000. HWE variance 2p(1−p) = 0.0198; dosage variance = 0.4 × 0.0198 = 0.0079, SD ≈ 0.089. For a true per-allele effect β = 0.2 on a quantitative trait, the dosage-regression SE scales as 1/√Var(dosage), so low INFO inflates it ~1.6× versus a clean variant. Hard-calling rounds the ~200 expected carriers to 0/1/2; carriers whose dosage sits near 0.5 flip between 0 and 1, adding error that biases β̂ toward 0 and, in a recessive contrast, turns dosages of 1.5–1.7 into homozygotes.
Curated: · Written: · Reviewed:
Q&A-51lambda_GC is 1.15. How do you tell polygenicity from confounding with LD-score regression, and when is the tool the wrong one?
The 60-second answer. λ_GC = 1.15 means the median χ² is 15% above its null value; mean χ² is about the same. That single number cannot separate polygenicity from confounding, because both inflate it. LD-score regression separates them by their LD signature. Regress each SNP's χ² on its LD score l_j (how much common variation it tags):
E[χ²_j] = N h² l_j / M + N·a + 1
The slope is N h²/M, so it estimates heritability. The intercept is 1 + N·a, where a is inflation uncorrelated with LD: stratification, batch, cryptic relatedness. Polygenic signal raises the slope; confounding raises the intercept. Slope up with intercept near 1 → polygenic. Slope near zero with intercept up → confounding.
Worked example. Two studies, both λ_GC = 1.15 (mean χ² ≈ 1.15), N = 100,000, ~1M HapMap3 SNPs, ancestry-matched LD scores.
| Study | LDSC intercept | Ratio | Slope | Read |
|---|---|---|---|---|
| A | 1.02 | 0.13 | clearly positive | polygenic |
| B | 1.13 | 0.87 | near zero | confounding |
Ratio = (intercept − 1)/(mean χ² − 1). Study A: 0.02/0.15 = 0.13. Most inflation tracks LD, so it is signal. Study B: 0.13/0.15 = 0.87. Most inflation is LD-independent, so it is confounding. Same λ_GC, opposite conclusions. That is the entire reason to run LDSC instead of quoting λ.
When LDSC is the wrong tool.
- Low mean χ². The intercept is only identified when there is enough signal. Below mean χ² ≈ 1.02 the intercept SE swamps the estimate; at 1.15 you can use it, but read the SE before quoting the intercept.
- Ancestry mismatch. LD scores must match the GWAS population. European LD scores on an East Asian or admixed cohort bias slope and intercept.
- Sample overlap with the LD reference panel, or between the two GWAS in a genetic-correlation run: inflates the intercept.
- Rare variants / low MAF. LD scores are built on common SNPs; rare-variant burden inflation is not captured.
- Extreme case-control ascertainment or strong selection.
- Confounding correlated with LD score (fine-scale within-Europe structure) biases the slope, not just the intercept: LDSC cannot fix that.
Alternatives. Genomic control divides every χ² by λ. It controls type-I error under uniform confounding but also removes real polygenic signal, so it over-corrects: use it when you only need calibrated p-values, not when you want h². GC-corrected meta-analysis inherits that. Partitioned/stratified LDSC assigns h² to functional annotations and can absorb annotation-correlated confounding. Mixed-model association (BOLT-LMM, fastGWA) models confounding directly from individual-level genotypes and is the better call when you have them. SumHer/LDAK is an alternative when you distrust the LD-score weighting assumptions.
What I'd report. Intercept with its SE, slope, mean χ², ratio, LD reference panel and ancestry, and N. λ_GC alone is not a verdict.
Curated: · Written: · Reviewed:
Q&A-50A SNP fails HWE at p=1e-12 in cases but not controls. Do you drop it, and what else can HWE mean?
Short answer: no. A case-only HWE failure is not a reason to drop a SNP. HWE is a QC filter you apply to controls; in cases, deviation can be the signal you are looking for. Drop only when controls fail too, or when cluster plots show the genotypes are wrong.
Why cases deviate. HWE assumes random mating and no selection. Cases are selected on a phenotype that depends on genotype, so the case genotype distribution is the population distribution reweighted by risk. Under a multiplicative (log-additive) model that reweighting leaves HWE intact: cases stay in equilibrium. Under a non-multiplicative model it does not. Take MAF 0.30 and a recessive risk allele with RR=2:
| genotype | pop freq | weight | case freq |
|---|---|---|---|
| AA | 0.09 | 4 | 0.284 |
| Aa | 0.42 | 1 | 0.331 |
| aa | 0.49 | 1 | 0.386 |
Case allele frequency 0.449, expected het 0.495, observed het 0.331, so F = (0.495 − 0.331)/0.495 = 0.33. A 33% heterozygote deficit in cases, controls untouched. A case-only deficit is exactly what a recessive locus looks like, and also what a collapsed genotype cluster looks like. Direction plus the cluster plot separates them.
How big is p=1e-12? The HWE test statistic is roughly N·F². χ²=50 gives p≈1.5e-12, so with 5,000 cases that p implies F≈√(50/5000)=0.10: a 10% heterozygote deficit. That is large, and large enough that I would look hard at the cluster plot before believing a recessive effect.
What else HWE means. Four things, and they are distinguishable:
- Genotyping error: collapsed or shifted clusters, excess or deficit of hets, often in both cases and controls, often with differential missingness.
- Population structure: Wahlund effect from mixing ancestries, heterozygote deficiency, present in controls too.
- Inbreeding or consanguinity: positive F, genome-wide, not locus-specific.
- True association: locus-specific, case-only, direction set by the genetic model.
Decision. Apply HWE to controls only, at the conventional p<1e-6 (PLINK 1.9 --hwe 1e-6 midp; --hardy prints observed/expected het and F). Keep the SNP if controls pass, cluster plots are clean, and call rate is comparable between cases and controls. Drop it if controls fail at 1e-6, if the cluster plot shows a miscalled cluster, or if differential missingness between cases and controls is significant (p<1e-5). Flag rather than drop when the deviation is case-only and the direction fits a non-multiplicative model.
Failure mode to watch. Bulk-dropping case-only HWE failures deletes real large-effect loci: HLA in autoimmune GWAS is the classic casualty, where cases genuinely sit out of HWE at a causal locus. The detection is a cluster plot plus a controls-only HWE test; if both are clean, the SNP stays.
Curated: · Written: · Reviewed:
Q&A-49A European-trained PRS is applied to an African-ancestry clinic. What must you show besides the discovery AUC, and why can it harm?
Besides the discovery AUC, you must show three things in the target population: calibration of absolute risk, discrimination measured in that population, and decision utility at the threshold you will actually use. The discovery AUC only tells you the score ranks cases above controls in the GWAS sample. It says nothing about whether the number attached to a given rank is right, and nothing about the clinic in front of you.
| What to show | Why the discovery AUC does not cover it |
|---|---|
| Calibration: predicted vs observed risk by decile, plus slope and intercept | AUC is rank-based; any monotone transform of the score leaves AUC unchanged and destroys absolute risk |
| Discrimination in the target population, ideally at the decision threshold | AUC estimated in the discovery sample reflects that sample's LD and allele frequencies |
| Decision utility: net benefit or decision curve at the clinical threshold | A threshold is an absolute-risk cutoff; miscalibration moves who crosses it |
| The LD and allele-frequency assumptions behind the weights | Weights are per-tag-SNP effect sizes, not per-causal-variant effects |
Mechanism. A PRS weight is the GWAS effect size of a tag SNP. In European-ancestry GWAS, long LD blocks mean the tag is a good proxy for the causal variant, so the tag's effect size absorbs the causal effect. In African-ancestry haplotypes LD blocks are shorter and tag-to-causal correlations differ, so the same tag captures less of the causal variant. The weight is attenuated, and the attenuation differs per SNP. Allele frequencies also diverge, which changes the score's mean and variance, so a fixed percentile of the discovery distribution is not a fixed percentile of the target distribution. Add differences in environmental variance and trait ascertainment, and both the ranking and the absolute risk move.
Worked example (hypothetical). A coronary PRS is deployed at the 90th percentile of the European discovery distribution, where that percentile carried an odds ratio of 3.2. The clinic has 4,000 patients. Because the score distribution in this cohort is shifted and wider, the same cutoff labels 800 patients (20%) rather than 400 (10%). Calibration in the cohort shows observed 10-year CHD incidence of 6% in the labelled group against 15% predicted by the European model. So 800 people receive a high-risk label and the intervention that follows it (statin, cardiology referral) for an absolute benefit roughly 2.5 times smaller than the model implied. The mirror harm is the patients whose true risk is above the threshold but whose score falls below the mis-set cutoff; they are not referred.
Alternatives, and when to stop. Recalibrate in the target population (refit intercept and slope, or refit the full model) if you have outcome data. Retrain on multi-ancestry or ancestry-matched GWAS where sample size allows; this fixes the weights, not just the intercept. Or drop the PRS from the decision and use clinical risk factors, which are usually better calibrated across populations. Stratify by genetic ancestry (for example, principal components), not self-reported race, which is a poor proxy for both ancestry and environment. Do not deploy when there is no target-population calibration data, when the target GWAS is too small to estimate ancestry-specific effects, or when the decision is an absolute-risk threshold and the model is uncalibrated.
Detection. Calibration plot by decile with slope and intercept; AUC in the target cohort; decision curve analysis for net benefit at the threshold. A score can hold its AUC and still be unusable: that is the failure this question is about.
Curated: · Written: · Reviewed:
Q&A-100You tested 1 million SNPs and used Bonferroni 5e-8 as 0.05/1e6. When is that conservative, and when is 5e-8 already the LD-aware convention instead?
Direct answer. 5e-8 is not a raw Bonferroni on 1e6 SNPs. It is the LD-aware convention for a European-ancestry common-variant scan, calibrated to roughly one million effective independent tests (Dudbridge and Gusnanto, 2008). The two numbers coincide for the standard case, but they mean different things.
Bonferroni on the raw count is conservative whenever the true number of independent tests M_e is below 1e6. That is the usual situation: common variants sit in LD blocks, so a million typed SNPs carry far fewer than a million independent signals. It is also conservative when you test a restricted set (a candidate panel of 200 SNPs, or a single locus) where 5e-8 is far stricter than 0.05/200.
5e-8 is already the LD-aware convention when you are doing a genome-wide common-variant scan in a high-LD population, which is the setting it was derived for. It is anti-conservative when M_e exceeds 1e6: African-ancestry cohorts have shorter LD blocks and more independent tests, so the same 5e-8 is too lenient there.
Mechanism. LD makes tests redundant. Take the correlation matrix of the SNPs; its eigenvalues count how many independent directions the data actually span. Li and Ji (2005) turn those eigenvalues into M_e; Galwey (2009) fits the eigenvalue distribution instead. Permutation (PLINK --mperm) sidesteps the algebra: permute the phenotype, record the minimum p-value each time, and read the 5th percentile off the null.
| Method | Estimates | For 1e6 European SNPs |
|---|---|---|
| Raw Bonferroni | α/M | 5e-8 |
| Li–Ji (2005) | M_e from eigenvalues | M_e < M, threshold looser than 5e-8 |
| Galwey (2009) | M_e from eigenvalue fit | similar |
| Permutation | empirical min-p null | data-specific |
| Dudbridge & Gusnanto (2008) | M_e for European GWAS | ≈1e6 → 5e-8 |
Worked arithmetic (hypothetical, to show the direction). 1e6 SNPs, European. Li–Ji on the LD matrix returns M_e = 620,000. LD-aware threshold = 0.05/620,000 = 8.1e-8. Raw Bonferroni = 5e-8. So raw Bonferroni is ~1.6× more stringent than the LD-aware number. The published convention, 5e-8, is deliberately more conservative than a single M_e estimate because M_e moves with SNP set, imputation panel and ancestry.
Trade-offs. Use raw Bonferroni when tests are genuinely near-independent: a small candidate set, or a replication panel of a few hundred SNPs. Use 5e-8 (or a permutation/Li–Ji threshold) for a genome-wide scan. Do not re-Bonferroni a genome-wide threshold by the number of traits without saying so: 20 traits is a second family, and the honest move is a predeclared joint procedure (a per-trait threshold plus a stated multi-trait correction, or a joint FDR) decided before the scan.
Failure modes. Applying 5e-8 to an African-ancestry scan and calling the result genome-wide significant is the common one; the threshold is too lenient and you replicate noise. The mirror error is Bonferroni on the raw SNP count in a high-LD cohort, which throws away real loci. Both are detectable: recompute M_e by Li–Ji or permutation on your own genotype matrix and compare it to the number you divided by.
Curated: · Written: · Reviewed:
Q&A-52A publication figure is a fully resolved tree with 100% support on every node. What must still be in the methods, and what can the picture hide?
The 60-second answer. A 100% support value means every bootstrap replicate recovered that split under the alignment and the model you handed the program. It is a statement about precision inside your model, not about whether the model is right. So the methods section has to carry what the figure cannot: the data, the model, the search, and the discordance.
What must be in the methods.
- Alignment: tool and version (MAFFT v7.520,
L-INS-i), trimming (trimAl 1.4,-automated1), and the final matrix dimensions. E.g. 1,412 columns, 87 taxa, 6.3% gaps. - Model: ModelFinder inside IQ-TREE 2.2.0, BIC as the selection criterion, and the chosen model named.
GTR+F+I+G4for a 16S set,LG+F+R5for a protein set. A model name is not optional; support under a misspecified model means nothing. - Search: IQ-TREE 2.2.0 or RAxML-NG 1.2.0, number of starting trees, and the random seed. Two independent runs with different seeds.
- Support: UFBoot2 with 1,000 replicates plus SH-aLRT with 1,000; or MrBayes 3.2.7, four chains, two independent runs, ESS > 200 on every parameter, burn-in stated.
- Discordance: per-gene trees, gene and site concordance factors (gCF/sCF), and for a species tree ASTRAL-III with local posterior probability and quartet score.
What the picture hides. Bootstrap resamples columns of one alignment under one model. Systematic error (long-branch attraction, substitution saturation, model misspecification) is invisible to it. So is incomplete lineage sorting: a rapid radiation can give every gene tree 100% support and still produce three different topologies.
| What 100% support measures | What it does not |
|---|---|
| Signal in the alignment under the chosen model | Whether that model fits |
| Stability to column resampling | Long-branch attraction, saturation |
| Convergence of the search | Contamination, chimeras, rogue taxa |
| — | Gene-tree discordance (ILS, introgression, HGT) |
When 100% is enough, and when it is not. Single locus, closely related taxa, dense sampling, and no long-branch signal in likelihood-mapping or saturation plots: report the tree and move on. Multi-gene, deep divergences, or a known rapid radiation: a single 100%-supported consensus is not evidence. Compute gCF/sCF; when gCF drops below roughly 50% on a branch, the genes disagree and the consensus is hiding it.
Worked example (illustrative). An 87-read 16S dataset gave a 100% UFBoot clade joining a clinical isolate to a lab contaminant. Dropping one chimeric read: flagged by UCHIME, ~1,400 bp, 99.8% identity to the contaminant reference: moved the isolate nine nodes away. The figure looked identical; only the alignment changed.
What I would ask for. The alignment checksum, the model name, the seed, and at least two independent runs that agree. A figure is a summary of a posterior, not the posterior itself.
Curated: · Written: · Reviewed:
Q&A-54UFBoot is 95% and the Felsenstein bootstrap is 62% on the same node. Which number do you quote, and why do they differ?
Quote both, each labelled with its method. If a single number has to go in an abstract, quote the Felsenstein 62% and put the UFBoot 95% beside it. The 62% is the conservative claim, it is the one that survives a reviewer asking how it was computed, and the 95% is the fast approximation. What you must not do is write "bootstrap 95%" and let the reader assume Felsenstein.
Assumptions: same alignment, same substitution model and rate scheme, same taxon set, IQ-TREE. UFBoot2 with 1000 replicates (-bb 1000, the default) versus Felsenstein nonparametric bootstrap with 1000 replicates (-b 1000).
Why they differ. Felsenstein's bootstrap resamples alignment columns with replacement, then rebuilds the tree from scratch on each replicate, a full search, and counts the fraction of replicate trees containing the node. 1000 replicates means 1000 tree searches.
UFBoot does not re-search. It takes the set of candidate trees already found during the initial search, and for each replicate resamples the sites and scores each candidate by RELL: resampling estimated log-likelihoods: it sums the per-site log-likelihoods already computed for each candidate tree over the resampled site indices, instead of re-optimising. The replicate's best tree is the candidate with the highest resampled score, and support is the fraction of replicates where the node appears in that best tree.
Two consequences push the number up. The candidate set is biased toward the ML tree: the true topology is in it by construction, and resampling sites cannot introduce a topology the search never visited. And RELL ignores that the optimal tree for a resampled alignment may lie outside the set. UFBoot2 (Hoang et al. 2018) applies a correction to the RELL bootstrap values to reduce this bias, but it does not make the result equal to a Felsenstein value.
The scales also have different thresholds: UFBoot ≥95% is the conventional cutoff, Felsenstein ≥70% (Hillis & Bull 1993). So 95 versus 62 is not "one is stronger": they sit on different rulers.
| UFBoot2 | Felsenstein | |
|---|---|---|
| per replicate | RELL over a fixed candidate tree set | full tree search |
| cost | minutes | hours to days |
| default replicates | 1000 (-bb) | 1000 (-b) |
| conventional cutoff | ≥95 | ≥70 |
| bias | upward, corrected in UFBoot2 | conservative |
Trade-offs. Use UFBoot for exploration, large taxon sets and pipeline speed: it is the IQ-TREE default and fine for a first pass. Spend the compute on Felsenstein when the node is the claim of the paper, when a reviewer or collaborator will recompute it, or when the two disagree by a lot. A 95/62 gap is exactly the case where the fast approximation is doing the most work.
Failure modes. The common one is quoting UFBoot as "bootstrap": the reader assumes Felsenstein and the number is inflated. Detection is in the methods section: the flag (-bb versus -b) and the software version. Second, applying the 70% Felsenstein cutoff to a UFBoot value, or the 95% UFBoot cutoff to a Felsenstein value. Third, reporting UFBoot without the correction version: pre-UFBoot2 values run higher still.
Curated: · Written: · Reviewed:
Q&A-53Two distant parasites group together with high support. Why might that be long-branch attraction, and which test do you run?
Assumptions first: the alignment is orthologous, the two parasites sit on long terminal branches (elevated rate, or saturated sites), and the support is bootstrap or posterior probability computed under the same model that produced the tree.
Why it can be LBA. Parsimony and simple models like JC69 undercount multiple substitutions. On two long branches, parallel substitutions at the same site are common, and parsimony reads them as shared derived states. Arithmetic: under JC69, a branch of length d = 0.5 substitutions/site has P(specific state change) = 1/4 − 1/4·e^(−4d/3) = 0.122. Two independent branches of that length both end in the same non-ancestral state at 3 × 0.122² ≈ 4.4% of sites. If the true internal branch separating them is 0.01, genuine synapomorphies are far rarer, so the long branches attract. Unequal base composition does the same to model-based methods: if both long branches drift to similar GC, a homogeneous model has no parameter to absorb it.
High bootstrap does not help. Bootstrap resamples sites under the same model, so it measures precision, not adequacy. 99% support for a wrong topology is exactly what LBA produces.
Which test. I run the composition homogeneity test first, a chi-square on base frequencies across taxa (Bowker's test for the pair), because it takes seconds and tells you whether the two long branches share skewed composition. If they do, the decisive test is a parametric topology test on the two competing trees: the SOWH test (Swofford–Olsen–Waddell–Hillis), which simulates under the null topology and asks whether the observed likelihood difference exceeds the null distribution. SOWH is conservative, it can fail to reject a wrong topology, and its null assumes the fitted model is correct, so I pair it with a posterior predictive model-adequacy test (Goldman 1993; Bollback 2002): simulate under the fitted model, compare statistics such as composition variance and tree length, and check whether the observed values fall outside the simulated distribution. If the model fails adequacy, LBA is live no matter what SOWH says.
| Test | Cost | What it decides |
|---|---|---|
| Composition chi-square | seconds | Do the two long branches share skewed base composition? |
| Likelihood mapping (quartet) | minutes | Is quartet signal weak/star-like? Cannot separate LBA from a short internode |
| SOWH parametric bootstrap | hours | Is the LBA topology rejected against the alternative? Conservative |
| Posterior predictive adequacy | hours | Does the fitted model reproduce composition and tree-length statistics? |
| CAT-GTR refit (PhyloBayes) | hours–days | Does a site-heterogeneous model change the topology? |
Confirmation. Remove one of the two taxa and re-infer; if the other's position moves, that is attraction, not history. Dropping fast-evolving or saturated sites, or RY-coding third positions, does the same job more cheaply.
Failure modes. Likelihood mapping alone is a trap: a star-like quartet is consistent with both LBA and a genuine rapid radiation, so it cannot decide. SOWH can fail to reject when the alternative is also misspecified. And if the two parasites really are sisters, every test here will leave the topology alone: these tests tell you the model is inadequate, not which tree is true.
Curated: · Written: · Reviewed:
Q&A-56A viral alignment has detectable recombination. Why is one phylogeny the wrong object, and what do you estimate instead?
The 90-second answer
A phylogeny is a model in which every taxon has exactly one parent. A recombinant has two. So the object you want is not a tree: it is a network or an ancestral recombination graph (ARG), or, if you want to stay in tree-land, a set of trees, one per non-recombinant region.
Assumptions I would state: the sequences are homologous and aligned, recombination is detectable in this alignment, and I care about the topology of the ancestry rather than only divergence times.
Why one tree is the wrong object
A bifurcating tree assigns each sequence one path to the root. A mosaic sequence carries two. The inference does not fail loudly: it fails by resolving the conflict in favour of whichever parent contributes more sites. If 70% of the recombinant's sites come from parent A, the whole-genome tree places it with A, on a branch that is short within the A clade and long relative to B. Bootstrap resamples columns; if most columns agree, the support value is high. That is the trap: 100% bootstrap support on a wrong placement. High support measures consistency of the site signal, not correctness of the history.
The artefacts you see: the recombinant sits on an unusually long terminal branch; its position flips between analyses depending on taxon sampling; and the two parent lineages show inflated divergence because the mosaic's sites are counted against both.
What to estimate instead
-
Breakpoints first. RDP5 (default p = 0.05 with Bonferroni correction across the methods you enable) runs RDP, GENECONV, BootScan, MaxChi, Chimaera, SiScan and 3Seq; GARD in HyPhy uses AICc with a KH test to place breakpoints. RDP5's BootScan defaults are a 200 bp window, 20 bp step, 100 replicates. Require agreement across at least two methods before believing a breakpoint.
-
Then either per-region trees or a network. Per-region trees are cheap and interpretable: split at the breakpoints, infer each block separately, and check that the blocks give different, well-supported topologies. SplitsTree6 gives a split network that shows the reticulation directly. PhyloNet fits explicit network models by likelihood, which is more principled but slower and needs a hypothesis about the number of reticulations.
-
If you need timing, an ARG method (ARGweaver, Relate) estimates the full recombination graph, so branch lengths are per-segment rather than averaged.
Trade-off: per-region trees are fast and each is a valid tree, but you lose the joint history and the breakpoint uncertainty. Networks are the right object, but the number of reticulations is a model choice you have to justify. ARGs are the most complete and the most expensive.
Worked example (illustrative figures)
A 9 kb HIV-1 alignment, 40 sequences. The whole-genome ML tree branches the query sequence as a distinct lineage with 100% bootstrap support, and someone calls it a new subtype. BootScan puts a breakpoint at ~2.5 kb. Region 1 (1–2500): the query clusters with subtype A at 98% support. Region 2 (2501–9000): it clusters with subtype B at 99% support. The whole-genome 100% is the average of two incompatible signals, not evidence for a third subtype.
| Object | What it gives | Cost |
|---|---|---|
| Single tree | One parent per taxon: wrong for a mosaic | Cheap, misleading |
| Per-region trees | Correct topology per block | Fast; breakpoint uncertainty ignored |
| Split network | Reticulation visible | No branch lengths or times |
| ARG | Full per-segment history | Slow, model-heavy |
Failure modes
- Calling a recombinant a new lineage because bootstrap is high. Detect by re-running per region.
- Trusting a single breakpoint method. RDP5's methods disagree on real data; require at least two concordant.
- A breakpoint landing inside a conserved block shifts the split and gives two trees that both look wrong. Inspect the alignment around the breakpoint.
Curated: · Written: · Reviewed:
Q&A-55Gene trees disagree with the species tree. How do you tell ILS from introgression, and why is a concatenated tree the wrong first move?
Opening: what I'd say first
Assume I have many unlinked loci, a candidate species tree, and at least one internal branch where gene trees disagree. The discriminator is shape, not just presence: ILS is genome-wide and roughly symmetric across the two minor topologies, driven by short internal branches; introgression is localized and asymmetric, inflating one specific discordant topology. So I quantify concordance factors, then test that asymmetry with a D-statistic against a negative control. I would not concatenate first.
Mechanism
Under the multispecies coalescent, a lineage entering an internal branch of length t coalescent units has probability (2/3)e^(−t) of failing to coalesce before the branch ends, producing a discordant gene tree. That is the entire ILS signal: it decays exponentially with branch length and is the same for every locus.
- t = 0.5 → 40% discordance
- t = 1 → 25%
- t = 3 → 3%
- t = 5 → 0.5%
So a long internal branch (≥3 coalescent units) leaves almost no room for ILS, and discordance there points at introgression. A short branch makes ILS the default explanation.
Introgression instead moves a chunk of one genome into another at a point in time. It affects some loci and not others, and it makes one discordant topology, the one grouping donor with recipient, more common than the other.
What each signal looks like
| Signal | ILS | Introgression |
|---|---|---|
| gCF vs sCF | gCF ≈ sCF | gCF > sCF on one topology |
| D-statistic (ABBA-BABA) | D ≈ 0 | D > 0, significant |
| Topology asymmetry | two minor topologies ~equal | one minor topology in excess |
| Genome distribution | uniform across loci | clustered in tracts |
Negative control
Before believing D > 0, where D = (ABBA − BABA)/(ABBA + BABA), I run the same test on a quartet where introgression is implausible, or simulate under the MSC with the estimated branch lengths. If the control also shows D > 0, the signal is ancestral structure or a short branch, not gene flow. Tools: Dsuite (Dtrios, f-branch) for the D-statistics, ASTRAL for the MSC tree, IQ-TREE 2 for gCF/sCF, QuIBL to compare ILS versus introgression likelihoods per branch.
Why concatenation is the wrong first move
Concatenation assumes one tree generates all sites. Under ILS that is false, and in the anomaly zone, short internal branches, the most likely gene tree differs from the species tree, so concatenation is statistically inconsistent: more data converges on the wrong topology. Its bootstrap makes it worse, because it resamples sites, and sites within a gene are not independent; the genes are. You get high support for a topology most gene trees reject.
Trade-off
Concatenation is fine when internal branches are long (ILS negligible) and taxa are few: it is cheaper and can be more precise. It is the wrong tool when branches are short or discordance is high; then use MSC (ASTRAL) plus a D-statistic, and only fit a network (SNaQ, PhyloNet) once the asymmetry test says gene flow.
Curated: · Written: · Reviewed:
Q&A-57Trimming gappy columns raised bootstrap from 50% to 99%. When is that cleaner signal, and when did you just delete the disagreement?
Trimming gappy columns is legitimate when the gaps mark genuine alignment ambiguity and you fixed the rule before you saw the tree. It is deleting the disagreement when the columns you removed are the ones carrying the alternative topology and you picked the threshold after you knew which answer you wanted. The bootstrap number alone cannot tell you which of those happened.
State three assumptions before you answer: what counts as a gap (an alignment indel versus missing sequence data), what threshold you applied, and which bootstrap you are quoting. Those three decide the interpretation.
Mechanism. Nonparametric bootstrap resamples alignment columns with replacement and counts how often a split reappears; UFBoot resamples site likelihoods. Either way, support measures how consistently the columns you kept support a split, not whether the split is true. Gappy columns are usually the most homoplastic ones: indel-rich stretches where homology is uncertain, so the same column votes differently across replicates. Drop them and the remaining columns agree more, replicate variance falls, and support climbs. That is a real effect and often a real improvement. The same mechanism fires when the removed columns are the ones that conflict with your preferred tree: you have not reduced noise, you have removed the minority vote.
Worked example (numbers hypothetical, shape is what I have seen). An 800-column alignment; a 50%-gap filter drops 40 columns, 5% of the data. Support for "parasites monophyletic" goes 51% to 99%, and the parasite clade moves 11 nodes to nest inside the host clade.
| Alignment | Columns | Topology | Bootstrap |
|---|---|---|---|
| full | 800 | parasites monophyletic | 51% |
| 50%-gap trim | 760 | parasites nested in host | 99% |
Five percent of the data swinging support 48 points is the red flag. Noise removal usually nudges support; it does not flip a topology.
How to tell which one you did. Re-run the topology search on the 40 removed columns alone. If they recover the host-nested tree, those columns were the evidence for it and the trim deleted the disagreement. If they are split or favour neither, the trim was neutral. Second check: score both topologies on the removed columns with a site-wise likelihood test (AU or SH). Third: run the alternative topology's support on the trimmed alignment too. If both topologies gain, the trim is neutral; if only yours does, it is not.
When trimming is clearly right. Gaps from unalignable hypervariable regions in a single-gene alignment, or columns that are missing because sequencing failed rather than because of an indel. Missing-at-random columns carry little phylogenetic signal and removing them is close to free. Gaps concentrated in the taxa that move are the opposite case.
What I would report. Both alignments and both trees in the supplement, the trimming rule stated before the analysis, and the support for the alternative topology on both. A predeclared rule plus both results is defensible. A 99% tree with the untrimmed 51% tree absent is not.
Curated: · Written: · Reviewed:
Q&A-58Two domains each have pLDDT 90. Why might their relative orientation still be a guess, and which AlphaFold output answers that?
The 90-second version
pLDDT is a per-residue score for the local environment. It is computed from the local distance distribution around each residue, so it is invariant to where that residue sits in space. Two domains can both score 90 and the hinge between them can still be unconstrained. The output that answers relative orientation is the predicted aligned error matrix, PAE, not pLDDT.
Why PAE is the right output
PAE[i][j] is the expected position error, in Ångströms, of residue j when the prediction is superimposed on residue i. The off-diagonal block (rows in domain A, columns in domain B) is literally the question "if I trust A's frame, where does B land?" Low values mean the relative placement is pinned; high values mean it is a guess. pLDDT never sees that block, which is why a 90/90 pair can still be a coin flip on orientation.
Worked example (hypothetical numbers)
Two three-residue domains joined by a 40-aa linker, both at pLDDT 91. AlphaFold2's output dictionary exposes pae (N×N, Å) and plddt (N,); ColabFold serialises the same keys into result_model_1.json.
import json, numpy as np
d = json.load(open("result_model_1.json"))
pae = np.array(d["pae"]) # (N, N) expected position error, Å
plddt = np.array(d["plddt"]) # (N,) per-residue, 0-100
A, B = slice(0, 3), slice(3, 6)
print("pLDDT A/B:", plddt[A].mean().round(1), plddt[B].mean().round(1))
print("intra-A PAE:", pae[A, A].mean().round(2))
print("A->B PAE mean/max:", pae[A, B].mean().round(2), pae[A, B].max().round(2))
print("B->A PAE mean/max:", pae[B, A].mean().round(2), pae[B, A].max().round(2))
With pae = [[0.4,0.5,0.6,21,22,23],[0.5,0.4,0.5,20,21,22],[0.6,0.5,0.4,19,20,21],[21,20,19,0.4,0.5,0.6],[22,21,20,0.5,0.4,0.5],[23,22,21,0.6,0.5,0.4]] and every plddt entry 91.0, the four prints are:
pLDDT A/B: 91.0 91.0
intra-A PAE: 0.49
A->B PAE mean/max: 21.0 23.0
B->A PAE mean/max: 21.0 23.0
Intra-domain PAE is 0.49 Å: the folds are solid. The A→B and B→A blocks average 21 Å, max 23 Å. That gap is the orientation being a guess, and it is invisible in the pLDDT column.
Choosing between the signals
| Signal | What it scores | Right when | Misleads when |
|---|---|---|---|
| pLDDT | per-residue local confidence, 0–100 | "is this domain folded?" | anything inter-domain |
| PAE | pairwise expected position error, Å, within one chain | domain–domain orientation | you read one number instead of the block |
| ipTM | interface confidence across chains (AF-Multimer, AF3) | chain–chain interfaces | single-chain model: it carries no information |
| pTM | one global scalar | ranking models | you need to know which interface is bad |
| SAXS / cryo-EM / XL-MS / NMR RDC | physical distance and shape restraints | you need the answer to be true, not self-consistent | you need it this afternoon |
Deciding criteria I would state out loud: inter-domain PAE below ~5 Å, treat the orientation as usable; 5–15 Å, treat it as a soft restraint; above ~15 Å, do not dock and do not interpret the interface. Then check the five models, if the inter-domain block moves between them, that is the same signal arriving a second way.
The failure mode
Docking a ligand or a partner into a high-pLDDT domain using the predicted orientation to the rest of the chain. You detect it by plotting the PAE matrix as a heatmap: the diagonal blocks stay dark and the off-diagonal blocks light up. PAE is also a self-consistency estimate from the same network, so it can be confidently wrong: it does not know about the ligand you are about to add.
Curated: · Written: · Reviewed:
Q&A-59You need a ligand-bound conformation. Why is a default AlphaFold monomer the wrong structure, and what would you use instead?
Direct answer
A default AlphaFold2 monomer is the wrong structure for a ligand-bound conformation for two independent reasons. First, AF2 has no ligand input channel and its training set is single-chain protein structures with the ligands stripped out: the ligand is simply not represented, so the pocket is predicted as it appears without it. Second, AF2 outputs one structure, and that structure tends to be the most populated, lowest-energy-like state in the MSA, which for most kinases is the apo or active-like state, not the inhibitor-bound one. You get neither the ligand nor the state.
What to use instead depends on whether an experimental holo structure exists. If a holo crystal or cryo-EM structure of the same protein with a related ligand exists at usable resolution, use it: it is the observed state. If not, use a ligand-conditioned predictor: AlphaFold3 (Abramson et al., Nature 2024), RoseTTAFold All-Atom (Krishna et al., Science 2024), or Boltz-1. These take the ligand (SMILES or CCD code) as input and predict the complex.
Mechanism
AF2's MSA co-evolution encodes contacts averaged over the sequences, and the network was trained to reproduce single-chain experimental structures. It has no way to condition on a small molecule. The output is a single static model: no ensemble, no state label. For a kinase, DFG-in versus DFG-out and αC-helix in versus out are different structures; AF2 gives you one, and it is usually the one most represented in the PDB, which is the active-like state.
Ligand-conditioned models fix this by making the ligand part of the input and the output: the network predicts atom coordinates for protein and ligand jointly, so the pocket is shaped around the ligand.
Trade-offs
| Source | Ligand in model | State | Cost / caveat |
|---|---|---|---|
| Experimental holo PDB | yes | observed | only if it exists; check resolution and pocket order |
| AF2 + holo template | no | biased to template | ligand still absent; template must be a close homolog |
| AF3 / RFAA / Boltz-1 | yes | ligand-conditioned | needs correct ligand chemistry; affinity not predicted |
Criteria: if a holo structure of the target or a close homolog exists, prefer it. If not, ligand-conditioned prediction. AF2 alone is a fallback for the apo fold only.
Failure mode
Worked example (hypothetical figures): a virtual screen was run against an AF2 apo model of a kinase. 200 compounds ranked as hits; none bound in the assay. The co-crystal holo pocket differed from the AF2 model by 4.2 Å RMSD at the gatekeeper region, and the AF2 pocket was partly collapsed. Detection: before screening, superimpose the AF2 model on any available holo structure and compute pocket RMSD; also dock a known co-crystal ligand back and check RMSD. If pocket RMSD is above roughly 2 Å or the pocket volume is much smaller than the holo, treat the model as a risk flag.
A predicted chain is not a binding assay.
Curated: · Written: · Reviewed:
Q&A-60A paper measures an interface on chain A–B from the PDB file as downloaded. Why might that interface not exist in biology?
Core answer. The coordinates in a PDB file are the asymmetric unit (ASU): the smallest set of atoms that, together with the space-group symmetry operators, generates the whole crystal. The ASU is a refinement and packing convenience, not a claim about the biological oligomer. So an interface you measure between chains A and B as downloaded can be wrong in three distinct ways. A–B may be a crystal contact that exists only because the lattice packed that way. The real partner of A may be a symmetry mate A′ that is not in the file at all. Or the ASU may contain only part of the biological assembly, so the interface you measure is truncated. The biological assembly is a separate annotation (REMARK 350 BIOMT records in the legacy PDB format, _pdbx_struct_assembly and _pdbx_struct_oper_list in mmCIF) and it is frequently not the ASU.
Why crystal contacts exist. Crystallization drives protein to very high concentration, and the lattice is the lowest-free-energy packing available under those conditions. Any two surfaces complementary enough to bury area will do so, whether or not they ever touch in the cell. Crystal contacts therefore tend to be smaller, more polar, poorer in hydrogen bonds per unit buried area, and, the key signal, not conserved across homologs.
How to tell them apart, and what each method is worth.
| Evidence | What it gives | Where it fails |
|---|---|---|
| Author assembly (REMARK 350 / mmCIF) | The depositor's stated oligomer | Often left as the default ASU; sometimes wrong |
| PISA / EPPIC | Buried surface area, solvation ΔG, complexation significance score | Heuristic; weak or transient interfaces get misclassified |
| Symmetry-operator check | Whether A–B is a lattice contact by construction | A real dimer's two-fold can coincide with a crystallographic two-fold |
| Solution data (SEC-MALS, AUC, native MS, SAXS) | Oligomeric state in solution | Needs material; can miss weak equilibria |
| Conservation across orthologs (ConSurf) | Whether interface residues are under selection | Needs enough homologs |
Deciding criteria: a real interface is large (obligate dimers typically bury 1,200–2,000 Ų total), hydrophobic, H-bonded, conserved, and reproduced in solution. A crystal contact is small (often a few hundred Ų, though some exceed 1,000 Ų, so size alone does not settle it), polar, non-conserved, and vanishes in solution. PISA's own convention flags a complexation significance score above roughly 0.5 as likely biological; treat that as a heuristic, not a verdict.
Worked example (hypothetical). Entry X has ASU = chains A, B. The paper measures A–B: total buried surface 620 Ų, 6 hydrogen bonds, solvation ΔG +3 kcal/mol (unfavourable), PISA CSS 0.1. REMARK 350 lists assembly 1 as A–A′ generated by a crystallographic two-fold. A–A′ buries 1,850 Ų, ΔG −14 kcal/mol, CSS 0.8. SEC-MALS on the recombinant protein gives a dimer at the expected mass. Conclusion: A–B is a lattice contact; the real dimer is A–A′. Mutating the A–B patch changes nothing in solution.
Failure modes. Mutating a crystal contact produces no phenotype, which is the classic tell. Designing an inhibitor against a lattice contact wastes the campaign. Reporting an oligomeric state that does not exist in solution misleads everyone downstream. And the deposited assembly itself can be wrong: authors sometimes leave the ASU as assembly 1 by default, or the assembly is genuinely ambiguous. Finally, the file may be a construct: affinity tags, fusion partners, a nanobody or Fab chaperone, engineered cysteines, a different species or a mutant, so even a real interface may not be the wild-type one. The file's first chains are not the molecule.
Curated: · Written: · Reviewed:
Q&A-61Two models have RMSD 8 Å and TM-score 0.82. Which one says they share a fold, and why can RMSD mislead?
TM-score 0.82 is the one that claims the fold. RMSD 8 Å claims nothing on its own.
TM-score above 0.5 is the standard line for "same fold" (Zhang & Skolnick's calibration; unrelated structures land below ~0.17). 0.82 sits comfortably inside. RMSD has no absolute scale: 8 Å is a respectable model for a 400-residue pair and a bad one for a 60-residue pair.
Why RMSD misleads: it is a quadratic mean over the aligned atoms, sqrt(mean(d_i²)). Squaring means one residue sitting 25 Å out contributes as much as 156 residues sitting at 2 Å. It is also not length-normalized, so RMSD grows with protein size even when the fold is identical. And it is defined only over the residue set you chose to align: include a disordered tail and you are measuring the tail.
TM-score fixes both. Each residue contributes 1/(1+(d_i/d0)²), capped at 1, with d0 = 1.24·(L−15)^(1/3) − 1.8 Å scaling with the length L of the reference (d0 ≈ 5.3 Å at L = 200; floored at 0.5 Å for very short chains). A residue 25 Å away contributes ~0.04 instead of dominating the sum.
Worked example: hypothetical, 200 residues, distances measured after superposition: 176 residues at 1.5 Å, 24 residues at 22 Å (a flipped loop plus a disordered tail).
- RMSD = sqrt((176·1.5² + 24·22²)/200) = sqrt(60.1) ≈ 7.8 Å
- TM-score = (176·0.925 + 24·0.054)/200 ≈ 0.82, using d0 = 5.3 Å
Same pair, two verdicts. The 24 outliers own the RMSD; they barely move the TM-score.
Trade-offs and where each breaks:
- TM-score is asymmetric. TM(A→B) normalizes by B's length, so it changes when you swap which structure is the reference. Always state the reference. RMSD is symmetric.
- TM-score is unreliable on short chains: d0 shrinks, so even good models score low.
- Both are global. Neither catches a model with the right fold and a wrong active site. Use lDDT (superposition-free; fraction of preserved interatomic distances within 0.5/1/2/4 Å) or a per-residue distance-difference plot for that.
- GDT_TS (fraction of residues within 1/2/4/8 Å) is the other length-tolerant global score; it is capped at 1 and less outlier-sensitive than RMSD.
Failure modes worth watching for:
- A pipeline gating on RMSD < 2 Å rejects correct models of large proteins. Detect it by plotting RMSD against aligned length across your decoy set, if the threshold rejects everything above 250 residues, the metric is wrong, not the models.
- Averaging RMSD over unaligned tails. Detect it by recomputing on the aligned core only; if the number drops from 8 Å to 1.8 Å, the tail was the whole story.
- Quoting TM-score without aligned length or reference direction. Detect it by asking for both: 0.82 over 40 aligned residues is a different claim from 0.82 over 200.
Report TM-score, aligned length, which structure was the reference, and core RMSD together. One number is never the answer.
Curated: · Written: · Reviewed:
Q&A-63A model is in the Ramachandran favored region but fits density poorly. Which check is missing, and why is geometry not enough?
The missing check is density fit: local map-model correlation (cryo-EM) or per-residue real-space fit, RSCC and real-space R, plus R-free (X-ray). Ramachandran is a restraint, not evidence. It tells you the backbone torsions are physically allowed; it says nothing about whether the chain sits in the right density.
Why geometry can't stand in for it. Refinement minimizes a weighted sum, roughly E = w_data·E_data + w_geom·E_geom. The geometry terms (bond lengths, bond angles, Ramachandran, rotamer, clashscore) are priors written into that target function. A misregistered model still gets pulled into allowed torsions, because the geometry term has no idea where the density is. Ramachandran is also local and sequence-independent: it scores φ/ψ one residue at a time and cannot see that residue 114 is sitting in density that belongs to residue 113. A green geometry report is partly a statement about your restraints, not about your model. Density fit is the independent term: it is the only check that compares the model to the measurement.
When geometry-heavy validation is still the right call. Below roughly 4 Å, side-chain density is uninterpretable and local CC is dominated by noise; there, geometry, main-chain density, and prior knowledge carry the validation. Flexible loops and termini with no density have no local CC to report, so geometry is the only signal. NMR structures and AlphaFold-style predictions have no experimental density at all: Ramachandran, clashscore, and pLDDT are the checks. And even with good density, geometry restraints are what stop refinement from overfitting the data term, so you keep them on.
Failure modes this catches. A one-residue register shift (backbone geometry fine, side chains in the wrong density). Sequence misassignment from a homologous search model. Over-refinement, where R-work falls while R-free rises. Molecular-replacement model bias, where the map is partly built from the model you are testing. Each shows up as a local density-fit drop that global geometry hides.
Worked example (hypothetical, 2.8 Å cryo-EM). A helix scores clean on geometry but the map looks wrong at one turn:
| Metric | Value | Typical threshold |
|---|---|---|
| Ramachandran favored | 98.5% | ≥98% |
| Clashscore | 4 | <10 |
| Global map CC | 0.78 | >0.7 |
| Local CC, res 112–118 | 0.41 | >0.7 |
| Register shift | 1 residue (~3.8 Å) | — |
Global CC and geometry both pass; only the per-residue local CC flags the turn. Rebuilding residues 112–118 one register earlier raises local CC from 0.41 to 0.72 with no change to Ramachandran. That is the check that was missing.
What to report. For any site-level claim, put local map CC (or RSCC / real-space R) beside the geometry numbers. Geometry passing alone is not a release gate.
Curated: · Written: · Reviewed:
Q&A-62When must you prefer a 3.5 Å crystal over a pLDDT-92 AlphaFold model for a catalytic residue claim?
Prefer the 3.5 Å crystal when the claim is about the side-chain orientation or chemistry of a specific catalytic residue, and the crystal has interpretable experimental density for that side chain in the relevant construct and state. Do not prefer it just because it is experimental. At 3.5 Å, side-chain density is often weak. If the map does not resolve the side chain, neither the crystal nor a pLDDT-92 model settles the claim.
I would first pin down what the claim actually is. "This serine is the catalytic base" is a claim about side-chain rotamer, distance to substrate, and protonation. pLDDT is a per-residue confidence from AlphaFold2, not a per-atom measurement. A pLDDT of 92 means the model is very confident about the local backbone placement. It does not mean the side-chain rotamer is correct.
Why AF2 can be locally wrong at a functional site even with high pLDDT: the confidence is learned from PDB regularities. It is high when the local sequence has many homologs and the local geometry is well represented in training. Catalytic sites often have strained rotamers, metal coordination, covalent intermediates, or protonation states that are underrepresented in the PDB. If the MSA is shallow or the region is sequence-variable, the model has less to learn from. AF2 also predicts a single static conformer, not an ensemble or a specific functional state, and it uses no experimental density restraints. Because pLDDT is computed per residue, a well-packed backbone can carry a mispredicted side chain. PAE would tell you about relative domain placement, not side-chain rotamer.
What the 3.5 Å crystal gives you is experimental density and restraints for that sample. At 3.5 Å, you can usually trace the main chain, but side-chain rotamers are often ambiguous. So the crystal wins only if the side chain has clear density, the B-factors are reasonable, and an omit map supports the placement. If the side chain is not resolved, the crystal does not decide the catalytic claim either.
The strongest case is when the crystal placement and functional perturbation agree: mutate the residue, lose activity, and the crystal shows the side chain pointing into the active site. That combination is what a mechanism paper should cite.
Here is how I would weigh the evidence:
| Evidence | What it measures | Decides side-chain rotamer? | Failure mode |
|---|---|---|---|
| 3.5 Å crystal with clear side-chain density | Experimental density for that sample | Yes, for that state | Low resolution can still mislead if density is weak or biased |
| 3.5 Å crystal without side-chain density | Main-chain trace only | No | Side chain is not observed |
| AF2 pLDDT 92 | Per-residue confidence in the predicted model | No | High backbone confidence can coexist with a wrong side chain |
| Functional mutants | Activity change from perturbing the residue | Supports the chemistry, not the rotamer alone | Mutation can change folding or stability |
Hypothetical example: Suppose a 3.5 Å structure of a hydrolase has a catalytic serine. The map shows the serine side chain pointing toward the substrate, and an S-to-A mutation abolishes activity. An AF2 model of the same construct has pLDDT 92 but places the serine rotamer away from the substrate. For the catalytic claim, I would cite the crystal plus the mutation data, and note the AF2 disagreement as a limitation. If the crystal map had no density for that serine side chain, I would not cite either as decisive. I would ask for a higher-resolution structure or a functional assay that reports the chemistry directly.
So the rule is not "experimental always beats predicted." It is: for a claim about a specific side chain's chemistry, prefer the source that actually observes that side chain in the relevant state, and require orthogonal functional evidence. A pLDDT-92 model can be the best available starting point when no experimental structure exists, but it should not outrank local density for the same residue.
Curated: · Written: · Reviewed:
Q&A-92Global clashscore is fine but the claimed ligand pocket has 8 clashes. Which number governs the claim?
The 8 governs. The claim is about a site, so the site's clash count decides it; the global clashscore is a whole-model density and cannot certify a pocket. Assume the 8 are nonbonded overlaps greater than 0.4 Å involving the ligand or the pocket residues, computed with hydrogens present (MolProbity's Reduce/Probe protocol). If they came from a different threshold, or without hydrogens, recompute before arguing.
Why the global number can't see it. MolProbity defines clashscore as clashes per 1000 atoms over the whole model (or whatever selection you hand it), and reports it as a percentile against structures of similar resolution. A clash is a steric overlap greater than 0.4 Å between nonbonded atoms after Reduce adds hydrogens; Probe uses a 0.5 Å probe radius. Because the metric is normalized by atom count, a small site is diluted. Take a 3000-atom model at clashscore 4: that is 12 clashes. Add the ligand's 8: 20/3000 × 1000 = 6.7. The model still reads as acceptable, and the ligand is still sitting in an overlap. The global number is arithmetically incapable of flagging a 30-atom ligand.
When each check is the right one.
- Whole-model clashscore: first-pass triage, comparing models, deposition. Fine for "is this model broadly sane", useless for "is this pose real".
- Local clash count (per-residue, ligand–protein contacts): the governing number for any site claim. Binding pose, active site, interface.
- Rotamer outliers in the pocket: a side chain in a disallowed rotamer can be the cause of the clashes, or a separate defect.
- Density fit (real-space correlation, difference map, B-factors): decides whether the pose is supported by the data at all. Clashes plus poor density means a wrong pose; clashes plus strong density means a real strained contact worth reporting.
Failure modes and how you catch them. A ligand refined into density that is actually a water or an alternate conformer; over-refinement pulling the ligand into the backbone; a global score that passes because the site is small. Detection: run Probe on the ligand-plus-pocket selection and read the contact list, check the ligand's RSCC and B-factors against the surrounding residues, and look at the difference map at the ligand. If the 8 clashes survive that, the pose is wrong.
| Scope | Metric | Value | Certifies |
|---|---|---|---|
| whole model | clashscore (clashes/1000 atoms) | 4 | model broadly sane |
| pocket + ligand | clash count | 8 | site is sterically impossible |
| ligand | density fit (RSCC, difference map) | check | pose supported by data |
Curated: · Written: · Reviewed:
Q&A-64A classifier of disease from single-cell RNA-seq hits AUC 0.97 with random cells in train and test. Why is that leakage, and what is the split unit?
My answer to grouped splits by donor begins at the confounder: if I cannot name what would fake the signal, I do not have a design.
Cells from one donor are not independent patients. Random cell splits put the same person on both sides.
Concretely, split by donor, family, site, or time to match deployment. Freeze the grouping before features.
The reason for that specificity is a failure I have seen: Random cell splits gave AUC 0.97; a donor-grouped split on the same 48 people dropped to 0.61.
48 donors, 200k cells.
| Split | AUC |
|---|---|
| random cells | 0.97 |
| by donor | 0.61 |
| leaked people | 48 |
I would not consider it settled without evidence: A split table with zero donor overlap; AUC is reported on that split.
The independence unit is the person, not the barcode.
Curated: · Written: · Reviewed:
Q&A-65A protein function model is tested on random 20% of UniProt. Why can that still leak, and how do you split?
Core answer
The leak is homology, not the split mechanics. UniProt is dominated by near-duplicate sequences: roughly 250 million unreviewed TrEMBL entries against ~570,000 reviewed Swiss-Prot entries, and many TrEMBL entries are fragments or close variants of the same protein. A random 20% row split puts a protein and its 99%-identical paralog on opposite sides. Function is strongly conserved among homologs, so the model can score well by memorizing "this family → this label" rather than learning anything transferable. The test set is then measuring family recall, not generalization.
The fix is to split by sequence cluster, not by row. Cluster the full set at a stated identity threshold (MMseqs2 or CD-HIT), assign whole clusters to train or test, and report the maximum identity of every test protein to the training set. If that number is above your threshold, the split leaked.
Choosing the threshold
The threshold encodes the claim you want to make:
- 50% identity: "generalizes to new members of known families." Cheap, keeps most data.
- 30% identity: "generalizes to remote homologs." 30% is roughly the twilight-zone boundary where pairwise alignment stops reliably detecting homology, so a 30% cluster split approximates "proteins BLAST would not have linked."
- Pfam/fold-level holdout: "generalizes to unseen families." The strictest, and the only one that tests novel function.
Over-strict clustering has a real cost. At 30% identity, clusters merge large regions of sequence space; the effective training set shrinks and the number of distinct test clusters can fall into the low thousands, so the AUC estimate gets noisy: a few hard clusters dominate. You also throw away legitimate signal: function often depends on fine-grained variation within a family, and collapsing those variants removes exactly the examples the model needs. The honest split is not the strictest one; it is the one matching the deployment claim.
Failure modes
- Multi-domain proteins. Clustering on the full sequence misses shared domains. Two proteins can share a 200-residue domain at 90% identity while full-sequence identity is 25%, so a full-sequence 30% split still leaks. Cluster per-domain (e.g., Pfam) when function is domain-driven.
- Pretraining contamination. Protein language models (ESM, ProtBERT) are pretrained on UniProtKB. If your test set is a UniProt split, the model already saw those sequences. No clustering fixes this. Evaluate on proteins deposited after the pretraining cutoff, or on a corpus the model was not trained on.
- Version skew. Train and test from different UniProt releases can share entries under different accessions. Pin one release and deduplicate by sequence hash.
Worked example (illustrative)
Same model, same data, four splits:
| Split | Test AUC |
|---|---|
| random rows | 0.94 |
| 50% identity clusters | 0.86 |
| 30% identity clusters | 0.71 |
| Pfam family holdout | 0.63 |
The 0.23 drop from random to 30% is the leakage you were measuring. The further drop to family holdout is the part that is actually hard.
What I'd report
Cluster counts per split, the identity threshold, and the max-identity distribution of test vs train. A single AUC without those is not evidence of generalization.
Curated: · Written: · Reviewed:
Q&A-66You tuned 40 hyperparameters on the same 5 folds you report. Why is that number optimistic, and where does the test set live?
Direct answer. The 0.88 is the maximum of 40 estimates taken on the same 5 folds you report. You picked the configuration that fit those folds best, so the number carries the selection, not just the model's skill. The test set lives outside the loop that chose the configuration: the outer folds of a nested CV, or a holdout frozen before the search and touched once.
Assumptions. 5-fold CV, 40 configurations, same folds for selection and reporting, no external holdout. Change any of those and the size of the bias changes, not its direction.
Mechanism. Each configuration's 5-fold estimate is its true skill plus fold-specific noise: from which samples landed in which fold, the model's seed, and the interaction between configuration and splits. The argmax over 40 configurations does not estimate a fixed model's skill; it estimates the maximum of 40 noisy draws, which sits above the mean and grows with the number of configurations and the spread of estimates. That is the winner's curse. Because the folds are shared, the noise is correlated and the bias is smaller than the independent-draw bound, but not zero: configurations differing in capacity or regularization have partly independent fold-level errors, and the search exploits exactly those.
Arithmetic (illustrative). If every configuration has true AUC 0.75 and the per-configuration 5-fold estimate has σ = 0.02, the expected maximum of 40 roughly Gaussian draws is about mean + 2.7σ ≈ 0.75 + 0.054 ≈ 0.80. Roughly 0.05 of the reported number is selection, before any real difference between configurations.
Where the test set lives.
- Nested CV: an inner loop selects on k−1 folds, an outer loop evaluates the selected configuration on the held-out fold. The headline is the outer-loop score; the inner score is a selection signal, not a result.
- Frozen holdout: split off a test set before the search, never look at it during tuning, touch it once. Tune against it twice and it is a validation set; you no longer have a test set.
If you already tuned on all 5 folds, those folds are training data. You cannot recover a clean estimate from them; you need new data or an outer loop.
Worked example (hypothetical figures).
| Protocol | AUC |
|---|---|
| Tune and report on the same 5 folds | 0.88 |
| Nested CV, outer-loop estimate | 0.74 |
| Prospective holdout, touched once | 0.73 |
Nested and prospective agree; 0.88 does not. That gap is the selection bias.
Failure modes. Preprocessing fit outside the fold (scaler, imputer, feature selection) leaks and inflates both inner and outer scores. Reusing the holdout across iterations turns it into a validation set. Reporting the inner-loop score as the result. Early stopping on the validation fold and then reporting that fold.
Key points the interviewer is listening for.
- Names the winner's curse, max of noisy draws, not just "overfitting".
- Says the bias is upward and grows with the number of configurations.
- Places the test set outside the selection loop: nested-CV outer loop, or a frozen holdout touched once.
- States that folds already used for tuning are training data, not a test set.
- Quantifies the effect instead of asserting it.
Weak-answer patterns.
- "It's overfitting" with no mechanism: overfitting the training data is a different failure from selecting on the reporting folds.
- "Use cross-validation": CV is already in use; the problem is that the same folds did both jobs.
- "Hold out 20% and tune on the rest" without saying the holdout is touched once.
- Reporting the inner-loop score of a nested CV as the headline.
- Calling the bias negligible because the folds are shared, without noting the partly independent errors.
Likely follow-ups.
- How many configurations before the bias matters? It scales with the spread of estimates and the count; at σ = 0.02 and 40 configs it is already ~0.05.
- What if you cannot afford nested CV? Repeated k-fold with different seeds for selection, one frozen holdout for the headline, or a small outer loop.
- Does this apply to random search and Bayesian optimization? Yes: the argmax is over the same noisy estimates, and Bayesian optimization concentrates evaluations near the optimum, which can make it worse.
- How would you detect it? Compare the outer-loop or holdout score to the inner score; a large gap means the search is fitting fold noise.
What I'd say in the room. The number is the max of 40 draws on the reporting folds, biased upward by the winner's curse; the honest estimate is the outer loop of a nested CV or a frozen holdout touched once. If the gap is large, the search is fitting fold noise: shrink the search space or increase the fold count before trusting the model.
Curated: · Written: · Reviewed:
Q&A-67AUC is 0.91 but at the 0.5 threshold the positive predictive value is 0.11. What did AUC not tell you?
The 90-second answer
AUC is a ranking statistic: the probability that a randomly chosen positive scores above a randomly chosen negative. It is computed over every threshold at once, and it does not depend on prevalence: double the number of negatives and AUC does not move. So it told you the model orders cases well. It did not tell you the three things you need to deploy: where the operating point is, whether the scores are calibrated, and how prevalence converts a good ranking into a bad flagged set.
PPV is a property of the threshold and the prevalence, not of the ranking. At 1% prevalence the negative class is 99× larger, so even a small false-positive rate at the cut swamps the true positives.
The arithmetic
Hypothetical screen, 100,000 people, prevalence 1%: 1,000 cases, 99,000 non-cases. At threshold 0.5 the model has sensitivity 0.90 and specificity 0.93.
| Disease + | Disease − | |
|---|---|---|
| Flagged (score ≥ 0.5) | 900 | 6,930 |
| Not flagged | 100 | 92,070 |
PPV = 900 / 7,830 = 0.115. You run 7,830 workups to find 900 cases: 6,930 of them on people without disease, 7.7 workups per case found. AUC 0.91 is consistent with all of that.
Push it further: at 1% prevalence and sensitivity 0.90, reaching PPV 0.5 requires specificity 99.1% (FP = 0.009 × 99,000 = 891, PPV = 900 / 1,791 = 0.50). A model with 93% specificity at the 0.5 cut cannot get there, however good its AUC is.
What to look at instead
- Precision-recall curve. Its no-skill baseline is the prevalence (0.01 here), so it shows the region you actually operate in. The ROC baseline is the diagonal regardless of prevalence, which is why ROC flatters rare-event models.
- Calibration curve (reliability plot) plus Brier score. AUC is unchanged by any monotone transform of the scores, so a model can rank perfectly and be badly miscalibrated.
- Threshold chosen on a validation set at the real prevalence, using the cost ratio of a false positive to a false negative. If a workup costs $200 and a missed case costs $20,000, the optimal cut is nowhere near 0.5.
Failure modes to name
- AUC averages over the whole curve, including the high-sensitivity region you will never use. A model can sit at AUC 0.91 and still be weak exactly in the high-specificity region a screening program needs.
- AUC is not comparable across populations with different case-mix. Same model, same AUC, different PPV in a specialty clinic versus a general screen.
- A single AUC number hides the variance at the operating point. Report the confusion matrix with confidence intervals at the chosen threshold.
AUC answers "does it rank?". Deployment asks "at this cut, in this population, how many of the flags are real?"
Curated: · Written: · Reviewed:
Q&A-68A variant-effect CNN trained on chr1–22 and tested on random SNPs from those chromosomes still leaks. What split actually tests generalization?
The split that tests generalization holds out whole chromosomes or whole LD blocks: never random SNPs.
A variant-effect CNN reads a fixed window around each variant, commonly 1 kb. Two SNPs 200 bp apart produce windows that overlap by 80%: each spans [v−500, v+500], so the shared span is 800 of 1000 bp. Put one in train and one in test and the model has already seen nearly the same input. Even when windows don't overlap, SNPs in the same LD block share a haplotype background, so the model can key on local sequence context instead of the variant's effect. A random split measures locus memorization, not variant-effect generalization.
So: hold out at least one chromosome, or better, whole LD blocks, and never tune on the held-out set. Report that set's metrics as the generalization number.
The mechanism decides the split, so state it precisely. The receptive field is the leak channel. Any test variant whose window overlaps a training window, or whose LD partners do, is not independent of training. In 1000 Genomes-style data, a random split leaves many r² > 0.8 partners on the other side.
Alternatives, and when each is right:
- Leave-one-chromosome-out. Clean and easy to explain, but coarse. Chromosomes differ in gene density, GC content, repeat content and ancestry composition, so a chr8 holdout mixes generalization with covariate shift, and 22 folds give a wide interval. Good headline, weak tuning signal.
- LD-block holdout. Prune at an r² threshold (0.2 is common), cluster the survivors into blocks, hold out whole blocks. Many folds, tighter control, but block size must exceed LD decay (roughly >1 Mb in European-ancestry samples, longer elsewhere) or you leak again.
- MAF-matched random split. Matching allele frequency removes a frequency confound but does nothing about LD. Use it as a secondary control, not as the split.
- Chromosome holdout is too coarse when you need per-gene or per-region estimates, or when you're tuning: use block holdout there.
Failure modes to check, with how to detect them:
- Near-identical windows straddling the split → compute max sequence identity between train and test windows.
- Same gene or same LD block on both sides → check gene and block overlap.
- Ancestry confound → report per-population metrics; a model can score well by predicting ancestry.
- Duplicate or multiallelic variants → dedupe before splitting.
| Split | Breaks LD/window leak? | Folds | Main confound |
|---|---|---|---|
| Random SNP | No | many | memorized loci |
| MAF-matched random | No | many | LD still leaks |
| LD-block holdout | Yes | many | block-size choice |
| Leave-one-chromosome-out | Yes | 22 | chromosome covariate shift |
Illustrative numbers from a setup like this: random-SNP split AUC 0.94 with calibration slope 0.98; chr8 holdout AUC 0.71, slope 0.61. The AUC drop is the leak; the slope drop says the model is also overconfident off-distribution.
Curated: · Written: · Reviewed:
Q&A-69A deep variant caller trained on NovaSeq is run on a new chemistry. What must you re-benchmark, and why is “the model is the same file” irrelevant?
The 90-second answer. The model file is a fixed function; the chemistry changes the input distribution that function was fit to. So re-benchmark everything the model's output touches, both discrimination and calibration, stratified by the contexts the new chemistry actually changes, and then decide retrain vs fine-tune vs recalibrate vs abstain from whether the shift is in ranking or in calibration.
Why "same file" is irrelevant. A DeepVariant-style caller takes a pileup tensor around each candidate site: per-read base, base quality, mapping quality, strand, whether the read supports the alt, plus reference context. The CNN maps that tensor to genotype likelihoods. Its weights encode the old chemistry's error spectrum, which substitutions dominate, how base quality maps to true error probability, how indels look in homopolymer runs. NovaSeq's 2-channel SBS chemistry has a characteristic error and quality-score profile; a new chemistry shifts it. The weights don't move, but the tensor-to-likelihood mapping was calibrated against the old noise model, so the same file now computes likelihoods under the wrong one. That is domain shift, not a bug.
Homopolymers bite first because indel errors there depend on the chemistry's ability to resolve repeated bases, and the CNN's local features around a homopolymer indel were learned against the old error rate: exactly where the signal is weakest.
What to re-benchmark.
| Layer | Measure | Stratify by |
|---|---|---|
| Discrimination | SNP/indel F1, precision, recall vs GIAB | overall, homopolymer, tandem repeat, low-complexity, GC-extreme |
| Calibration | genotype-likelihood / QUAL calibration | QUAL bin |
| Coverage | F1 vs depth | 10x–40x |
| Downstream | trio Mendelian violation rate, joint-call concordance | trio |
The trade-off. Post-hoc recalibration is cheap and fixes QUAL/GL calibration when the shift is monotone, but it cannot fix discrimination. Fine-tuning on a small new-chemistry truth set is the middle path: mix old and new data or you risk catastrophic forgetting. Full retraining is right when discrimination itself moved. Abstain and fall back to a chemistry-agnostic haplotype caller when no truth data exists for the new chemistry. The deciding question: did ranking move (F1 drops → retrain/fine-tune) or only calibration (F1 holds, QUAL wrong → recalibrate)?
Failure modes. The headline number can pass while a stratum fails: overall SNP F1 looks fine, homopolymer indel F1 collapses. Detect it with the stratified GIAB table, not the summary. Calibration failure is subtler: overconfident QUAL propagates into joint genotyping and somatic callers; detect it with a calibration curve and a trio's Mendelian violation rate. Operationally, a hypothetical: if homopolymer FP goes from ~120 to ~2,400 per genome, that is ~20x more sites to review, hours of manual work per genome, and corrupted downstream calls.
Curated: · Written: · Reviewed:
Q&A-70A model card lists AUC 0.93. What else must it say before a clinic may use it, and what is an excluded use?
What a strong candidate says in the first minute
AUC 0.93 tells you the model ranks one fixed retrospective cohort well. It says nothing about whether the score is calibrated, where the operating threshold sits, or whether that cohort resembles your clinic. Before a clinic may use it, the card must state: intended use and the exact validated population; the operating threshold with calibration at that threshold; subgroup performance; the validation design (retrospective vs prospective); regulatory status; and an explicit excluded use. Miss any of those and you have a research artifact, not a deployable one. The card format is standardised: Mitchell et al. (2019) model cards, plus the FDA's Software as a Medical Device framework when the tool meets the device definition.
Why AUC alone is not enough
AUC is invariant under any monotone transform of the score. A model that outputs 0.02 for every patient and 0.04 for the sick ones can score 0.93 and still have no threshold you can act on. Two consequences: you cannot pick a cutoff from AUC. You need calibration (reliability curve, Brier score, calibration slope) at the threshold you will actually use; and the positive predictive value at that threshold depends on prevalence, which AUC hides.
Worked example, hypothetical round numbers. 1,000 patients, prevalence 10% → 100 positives, 900 negatives. At the chosen threshold the model hits sensitivity 0.90, specificity 0.85: TP = 90, FN = 10, FP = 135, TN = 765. PPV = 90 / (90 + 135) = 0.40: six of every ten alerts are false. Move the same model to a 2% prevalence screening population: 20 positives, 980 negatives, TP = 18, FN = 2, FP = 0.15 × 980 = 147, so PPV = 18 / 165 ≈ 0.11. Same AUC, one in nine alerts true.
Fields the card must carry
| Field | What it must state | Example |
|---|---|---|
| Intended use | task, care setting, population | triage support, adults 18–75, ED, clinician-facing |
| Excluded use | uses explicitly forbidden | not for autonomous denial of care or coverage; not for patients under 18 |
| Validation population | cohort, dates, sites, exclusions | 3 sites, 2019–2021, adults, pregnancy excluded |
| Threshold + calibration | cutoff, sens/spec, Brier, calibration slope | 0.42; sens 0.90 / spec 0.85; Brier 0.11 |
| Subgroup metrics | AUC and calibration per subgroup | AUC 0.93 overall, 0.78 for age > 80 |
| Validation design | retrospective vs prospective/silent | retrospective; no prospective trial run |
| Regulatory status | SaMD class, clearance, predicate | 510(k)-cleared, class II (example) |
| Versioning | artifact hash, training data version | sha256:…, dataset v3.2 |
Trade-offs
When the card is incomplete you have three credible paths: refuse deployment; run in silent (shadow) mode where the model scores but no decision depends on it, with monitoring; or run a prospective trial. The deciding criteria are whether the local population sits inside the validated one, whether the threshold is calibrated locally, and whether the clinic's intended use appears on the excluded list. If it does, no amount of monitoring fixes it: refuse.
Failure modes and how they surface
- Distribution shift: local prevalence or case mix differs, so PPV collapses even with AUC intact. Detect by monitoring score distribution and weekly PPV against observed outcomes.
- Calibration drift: AUC holds while the calibration slope moves, so the threshold silently becomes wrong. Detect with a reliability curve on recent data.
- Subgroup harm: overall AUC hides a weak subgroup. Detect with per-subgroup AUC and calibration before deployment, not after.
Excluded use, concretely: a card reading "not for denial of care" means the clinic may not wire the score into an eligibility or coverage decision. Only into a clinician-facing review queue.
Curated: · Written: · Reviewed:
Q&A-72The workflow pins bwa:latest. Why is that not reproducible, and what do you pin instead?
Short answer: bwa:latest is a mutable pointer, not a version. Two runs of the same workflow can pull different bytes. Pin the image by digest, bwa@sha256:..., and pin the platform, because the same digest resolves to different binaries on amd64 and arm64.
Why tags aren't reproducible. A tag is a name the registry maps to a manifest. latest is just the tag applied to the most recent push that didn't specify one. Anyone with push rights can repoint it. A rebuild of the same Dockerfile often produces a new manifest with a new digest even when the source is unchanged, because the config blob records a build timestamp and the base-image digest may have moved underneath you. So bwa:latest on Monday and bwa:latest on Friday are different images wearing the same name. The workflow file is byte-identical; the result isn't.
What a digest pins. image@sha256:... is content-addressed: the digest is the hash of the manifest, and the manifest lists the digests of every layer and the config blob. Pulling by digest gets exactly those bytes or fails loudly. That chain is transitive: pinning the top digest pins the base-image layers too, so you don't pin the base separately.
What it doesn't pin. The digest covers the image, not the run. Still unpinned: the workflow definition itself, conda/pip installs done at runtime, reference genome and index files, and the container runtime version. Pin those separately: commit the workflow, pin the reference by checksum.
The multi-arch trap. A manifest-list digest resolves to a platform-specific manifest. Same digest, different binary on arm64 vs amd64. Pin the platform-specific digest and set the platform explicitly, or a mixed fleet silently diverges.
| Reference | Reproducible? | Why |
|---|---|---|
bwa:latest | No | mutable tag, repointed on any push |
bwa:0.7.17 | No | version tags are still mutable |
bwa@sha256:<manifest-list> | Partly | content fixed, platform chosen by host |
bwa@sha256:<platform> + --platform linux/amd64 | Yes | exact bytes, exact arch |
process bwa_align {
// Nextflow 24.x, Docker 25.x; digest below is a placeholder
container 'quay.io/biocontainers/bwa@sha256:PLACEHOLDER_64_HEX'
}
Resolve the digest once and record it:
docker buildx imagetools inspect quay.io/biocontainers/bwa:0.7.17--h5bf99c6_8
# copy the linux/amd64 digest, not the top-level manifest-list digest
Trade-offs. Digest pinning freezes you: you stop receiving base-image CVE fixes. That cost is real, and pretending otherwise is how teams end up on a two-year-old image. Manage it with a scheduled job, Renovate or Dependabot with digest pinning enabled, that opens a PR bumping the digest, so updates are deliberate and reviewed instead of silent. If you can't run that, pin the digest and set a calendar reminder. A stale digest you chose beats a moving tag you didn't.
Failure mode to name. The symptom is a rerun of a green pipeline producing different output with no code change. Detect it by logging the resolved digest at runtime and diffing between runs:
docker inspect --format '{{index .RepoDigests 0}}' <container>
If the digest changed and the workflow didn't, the tag moved. That is the whole failure, and it is invisible unless you log the digest.
Curated: · Written: · Reviewed:
Q&A-71Nextflow finished with exit 0. Why is that not a release, and what contract would have caught the silent 41-sample cache hit?
Assumptions: Nextflow 24.x, default process.cache = 'standard', run started with -resume, inputs on a POSIX filesystem.
Exit 0 is the driver's exit status: the workflow graph completed and every task that ran exited 0. It says nothing about whether every task ran. With -resume, Nextflow decides a task is up to date by hashing the task script, the container, and the metadata of each input file: path, size, last-modified time. Not content. Rename the sample sheet, or replace a FASTQ with cp -p / rsync -t that preserves mtime, and the cache key is unchanged. Nextflow marks the task CACHED, reuses the previous output, and exits 0. 41 of 96 BAMs are stale and the exit code is silent about it.
The default is metadata because hashing every input is expensive: a stat() per file versus a full read of a 100 GB FASTQ set. Nextflow trades correctness for speed and exposes cache to change the trade.
Three contract layers, chosen by cost:
1. Content-keyed cache. process { cache = 'deep' } hashes file content, so a changed FASTQ invalidates the task. Cost: full read of every input each run. cache = 'lenient' keys only on task name and input file names: cheaper, weaker. cache = 'none' disables resume.
process FASTP {
cache 'deep'
input:
tuple val(sample), path(reads)
output:
tuple val(sample), path("${sample}.trim.fastq.gz")
script:
"""
fastp -i ${reads[0]} -I ${reads[1]} -o ${sample}.trim.fastq.gz
"""
}
2. Sample-sheet validation before any task. nf-core's samplesheet_check.nf (nf-schema) validates the sheet against a JSON schema: required columns, types, duplicate sample IDs. A rename that drops or renames a column fails here, before the cache is consulted. This catches the rename at the source rather than downstream.
3. Executed-vs-cached assertion. -with-trace writes a trace file with hash and status per task; -dump-hashes dumps task hashes. Assert that the count of COMPLETED tasks equals the expected sample count and that no task you expected to run is CACHED.
| name | status | hash | start |
|---|---|---|---|
| FASTP (SAMPLE_41) | CACHED | 3f9a1c… | 2024-05-01 09:12 |
| FASTP (SAMPLE_42) | COMPLETED | 7b2e04… | 2024-05-02 09:14 |
A CACHED row whose input changed is the signal; -with-report gives the cached-vs-executed counts.
What would change the decision: a nightly re-run over unchanged inputs is exactly what caching is for. Deep mode is wasted I/O there, so validate the sheet and assert counts. Inputs that change per run make deep mode cheap insurance. The deciding question is whether an input can change without its path, size and mtime changing; if yes, metadata caching is unsound for that pipeline.
Curated: · Written: · Reviewed:
Q&A-73FASTQ headers say sample A and the sheet says sample B. What must win, and which QC is the last chance to catch a swap?
Short answer: The sample sheet wins over the FASTQ header, but only because the header is downstream of the sheet, not because the sheet is trustworthy. Neither is evidence of who the DNA actually belongs to. The genotype is. The last chance to catch a swap is the genotype-concordance gate on the aligned BAM/CRAM, run before variant calling.
Why the header can't win. BCL Convert (or bcl2fastq) reads SampleSheet.csv, matches the index read to a row, and routes reads into that row's sample directory. The FASTQ header carries the index sequence, not the sample name: a header like @A00123:45:HXXXX:1:1101:1000:1000 1:N:0:ATCACG holds instrument, flowcell, lane, tile, coordinates and index, and nothing about the subject. So a header-vs-sheet disagreement is not two independent claims. If a sample name appears in the header at all, it was injected by a downstream tool from the same sheet, or derived from the filename. It is one claim and its copy.
Precedence, and what each layer can actually prove:
| Layer | Authority | Proves | Blind to |
|---|---|---|---|
| FASTQ header | none | nothing independent | everything |
| Sample sheet | routing | which index went to which file | whether the row is right |
| Sex check (X/Y coverage) | weak | male/female swap | same-sex swap |
| Genotype concordance | strong | identity vs a known genotype | needs a reference genotype |
| Tumor–normal cross-check | strong | T/N swap, relatedness | — |
The last chance. After demultiplexing, the sheet is frozen into the files; you cannot re-derive identity from it. The last gate where a swap is still cheap to fix is the identity check on the aligned BAM, before variant calling. Run VerifyBamID2 against a reference panel (it reports FREEMIX contamination and can verify identity), or NGSCheckMate, somalier, or Picard CrosscheckFingerprints to compare SNP allele fractions between the BAM and a known genotype: an array, or a prior run of the same subject. Once you call variants, a swap is still detectable, but you redo the pipeline and every downstream VCF is poisoned.
Worked swap (hypothetical figures). Sheet says tumor=A, normal=B. Fingerprinting shows the tumor BAM matches B's array genotype and the normal BAM matches A. The somatic caller, run on the swapped pair, emits a large set of apparent somatic SNVs at ~50% VAF: these are germline heterozygous sites in the true normal, not tumor-specific. The concordance gate flags the pair in minutes; without it, those calls ship.
When the sheet legitimately governs. At demultiplexing, only the sheet can route reads: no genotype exists yet. So the sheet is the operational authority for routing, and the genotype is the authority for identity. If they conflict, stop the run; do not pick a winner by hand.
Curated: · Written: · Reviewed:
Q&A-74A reviewer says FAIR requires uploading the BAM to a public bucket. What does accessibility actually require?
Accessibility is about retrieval, not openness. The reviewer is conflating "accessible" with "open." FAIR's own wording: principle A1.2: explicitly permits authentication and authorization where necessary. A DAC-gated dataset in dbGaP or EGA is fully FAIR; a public bucket is not automatically more FAIR, it is just less controlled.
Assumptions I would state up front: human germline WGS, identifiable, no consent for unrestricted release. If the consent does allow open release (a consented population panel, a cell line, summary statistics) then public deposition is fine, and the deciding question is consent scope, not FAIR.
What the A principles actually require (Wilkinson et al., Scientific Data 3:160018, 2016):
- A1: data and metadata retrievable by their identifier over a standardized protocol. The identifier has to resolve: a dbGaP accession (
phs######) or an EGA dataset ID (EGAD...) lands on a landing page. - A1.1: that protocol is open, free, universally implementable. HTTPS, not a bespoke VPN client.
- A1.2: the protocol allows authentication and authorization where necessary. This is the clause that makes controlled access FAIR.
- A2: metadata stays accessible even after the data is withdrawn.
So the recipe is: open metadata, open protocol, gated data.
| Layer | Open? | Mechanism |
|---|---|---|
| Metadata | yes | accession/DOI resolves over HTTPS |
| Access procedure | yes | DUO consent codes, DAC application |
| Raw BAM/CRAM | gated | OAuth2 via RAS / eRA Commons, DAC approval |
| Audit | internal | access logs, data-use agreement |
The deciding criterion is consent scope plus identifiability, not file format. BAM vs CRAM vs VCF does not matter: aligned reads are as reidentifiable as raw reads.
Failure mode: a "FAIR" public bucket holding 120 WGS. Accession numbers plus a public genealogy site were enough to reidentify 8 donors (illustrative figures). Detection is a reidentification check before release, query the sample's rare variants against a public beacon or genealogy service, plus egress monitoring on the bucket.
The test is whether a qualified researcher can reach the data through a documented, authenticated path. Not whether anyone can wget it.
Curated: · Written: · Reviewed:
Q&A-75Task logs include FASTQ paths with patient names. Why is that a breach even if the BAM is encrypted, and what identifiers belong in logs?
What I'd say first
Assume the BAM is encrypted at rest with a scoped KMS key. That protects the bytes of the BAM and nothing else. The task log is a different object on a different surface: the scheduler writes the command line, working directory and input paths into its own database, and the filesystem writes the filename into directory entries and inodes. Neither lives inside the encrypted blob. So the breach is not "the BAM leaked": it is that a patient name now sits in a plaintext, widely readable, long-retained index.
Why the filename is the leak
Encryption at rest is per-object. It covers the contents of the file you encrypted; it does not cover the name you gave it, the path you invoked it with, or the log line that recorded the invocation. Those are separate surfaces with separate lifecycles, and they are usually the less protected ones. A log line gets shipped to a central aggregator, indexed, retained for months, and read by anyone holding the log role: a far larger group than the people holding the BAM key. The same name also propagates to the scheduler database, shell history, ps output, backup catalogs, error emails and the SIEM. Each is a copy you never encrypted and probably never inventoried.
Under HIPAA the test is not "is it encrypted" but "does it identify". A name, MRN, accession number or full date in a log is PHI regardless of the encryption state of any other file.
What belongs in logs
| Field | In logs? | Why |
|---|---|---|
run_id (e.g. 9f3a) | yes | joins logs to the run, no patient link |
sample_id (S-0421) | yes, if the mapping table is separate and access-controlled | pseudonym |
| instrument, lane, index barcode | yes | needed to debug QC failures |
| tool name and version, exit code, timestamps | yes | reproducibility |
| patient name or MRN in a path | no | direct identifier, plaintext, long retention |
| DOB, collection date | no | quasi-identifier; re-identifies small cohorts |
| full input path | no | leaks whatever the directory name contains |
A log line should read run=9f3a sample=S-0421 tool=bwa-mem exit=1, not run=9f3a input=/scratch/JaneDoe_tumor.fq.
Trade-offs, and when pseudonyms are not enough
Pseudonymization is the default, but it is not a complete control. It fails in three ways worth naming. First, if the mapping table is stored beside the logs or in the same scratch, you have moved the identifier rather than removed it. Second, in a small cohort the pseudonym plus quasi-identifiers re-identifies: a rare tumor type, a collection date and a site narrow to one patient even with no name anywhere. Third, pseudonyms are stable across runs, so a longitudinal join across logs rebuilds the patient history.
Redaction at write time is stronger than redaction at ship time, nothing plaintext ever reaches the buffer, but you cannot redact what you never logged, so you lose the path that would have explained a failure. Access control alone does not help, because it does not shrink retention or aggregation. The practical combination is: pseudonymous identifiers in logs, the mapping table in a separate access-controlled store, write-time filtering of path fields, and the shortest retention that still supports incident response.
Failure modes and how you catch them
A tool prints its input path in an error message, so the name appears in stderr even though your own logging is clean. A mapping table gets committed to the same repo as the pipeline. A scratch directory is left behind after a failed run and indexed by the cluster search.
Detection is a canary test: run a job with a synthetic patient name in the input path, then assert that string appears nowhere in the logs, stdout, stderr or scratch. Run it in CI on every pipeline change. That test is what turns "we pseudonymize" from a claim into a check.
Curated: · Written: · Reviewed:
Q&A-76The largest 8 tumors failed out of memory and were dropped. What bias did that create, and how should resources be declared?
The bias. Dropping the 8 largest tumors is not random missingness: it is missing-not-at-random, and the missingness is correlated with the variable you are measuring. Memory scales with input size, input size scales with genome content, and genome content is what a CNV landscape is about. A whole-genome-duplication (WGD) tumor carries roughly twice the genome, so at fixed sequencing depth it yields more aligned bases, a larger BAM/CRAM, and more per-bin coverage state in the caller. The pipeline therefore preferentially deletes the exact class the paper is trying to count. The published landscape reports WGD frequency in the survivors, biased low, and no reader can see it.
How to declare resources. Treat the memory request as a measured quantity, not a guess. Fit peak RSS against input size across the samples that did finish, set the request at the fitted value plus margin, and let the scheduler escalate on retry.
Concretely, in Nextflow:
process cnvCall {
memory { 16.GB * task.attempt } // 16, 32, 48 GB
errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
maxRetries 3
}
Exit 137 is 128 + 9 (SIGKILL): the OOM killer. Retrying only on that status avoids re-running on a genuine crash.
On SLURM the same intent is #SBATCH --mem=64G (or --mem-per-cpu); on Kubernetes it is resources.requests.memory and resources.limits.memory, and a container killed at the limit shows OOMKilled in kubectl describe pod. Set requests to the fitted value and limits a little above, so a sample that needs more is retried rather than silently killed.
Trade-offs.
| Option | Cost | When it is right |
|---|---|---|
| Declare high for all | Wasted cluster, longer queue | Small cohorts, or when the correlate is the outcome |
| Retry with escalation | Needs a retry budget; can still fail | Default: catches the tail without over-provisioning |
| Exclude and report | Cheapest | Only if you publish the exclusion and its correlate |
The deciding criterion is whether the dropped set correlates with an analysis variable. If it does (as here, with WGD) you cannot drop silently: raise resources until they finish, or report the missingness and bound the claim. If the drops are uncorrelated with anything you measure, exclusion with a count is fine.
Detection. Log peak RSS per sample and the exit status; a completion table by input size makes the bias visible. If the largest bin has zero completions, the cohort is truncated, not sampled.
Curated: · Written: · Reviewed:
Q&A-93A process expected one BAM per sample but a channel mixed 2 BAMs into one task. What invariant did the workflow break?
The invariant: one channel item per sample, keyed by a unique sample ID.
A process declaring tuple val(sample), path(bam) asserts that every item it receives carries exactly one BAM belonging to exactly one sample. The workflow broke the cardinality invariant: the channel's key set must be unique and its item count must equal the sample sheet's row count, and Nextflow does not check it for you.
Why it fails silently. Nextflow operators match on keys or on emission order, never on "does this look like one sample."
joinmatches rows on the key. A non-unique key on either side produces the cartesian product of the matching rows: oneS2on the left and twoS2on the right gives two output rows, not an error.groupTuplecollects every item sharing a key into a list. Two BAMs underS2become[S2, [S2.bam, S2_rerun.bam]].- Two queue channels feeding the same process are zipped by emission order, not identity. If one emits 96 items and the other 95, the 96th pairs with nothing, or with the wrong partner once the order shifts.
None of these raise. The process runs, stages both files, and produces a mixture.
Trace. A sheet with a duplicate sample ID:
sample bam
S1 S1.bam
S2 S2.bam
S2 S2_rerun.bam
sheet = Channel.fromPath('samples.tsv')
.splitCsv(header: true)
.map { row -> tuple(row.sample, file(row.bam)) }
sheet.groupTuple().view()
splitCsv emits three rows. groupTuple buffers and groups by key, emitting in order of first appearance:
[S1, [S1.bam]]
[S2, [S2.bam, S2_rerun.bam]]
Feed that to process CallVariants { input: tuple val(sample), path(bams) } and the S2 task stages two BAMs. GATK sees a sample that is half one genome and half another; the het calls are chimeric. Nothing is printed.
Alternatives and their cost.
- Assert uniqueness up front.
sheet.map { it[0] }.toList(), then checksize == unique().size, orassertin the workflow body. Fails at launch and names the offending key. Costs one materialization of the key list: trivial at 96 samples, noticeable at 100k. - Encode the expectation in the DAG with
groupTuple(size: 1). Each item emits as its own single-element group, so two BAMs underS2become two separateS2tasks instead of one chimeric task. The mix is prevented, but the duplicate is now silent duplicate work: twoS2outputs that collide downstream. It guards the process, not the sheet. - Quarantine instead of fail. Route duplicate keys to a side channel and continue with the clean set. Right when a rerun BAM is legitimately expected and you want the pipeline to finish; wrong when a duplicate means the sheet is corrupt, because you ship results for a sample you silently dropped.
Detection. Dump the channel and compare: sheet.view() should print one line per sheet row, and the set of keys should equal the sheet's sample column. If the counts differ, the invariant is already broken upstream: usually at the sheet, occasionally at a glob that matched a rerun file.
Curated: · Written: · Reviewed:
Q&A-77P04637 and P04637-2 are not the same sequence. What must you store, and what happens if you join on accession without isoform?
Assumptions first. I'm assuming the data carries residue coordinates (variants, PTMs, structural contacts) or has to join to something that does. If it only ever asks gene-level questions ("is this protein present?"), the calculus changes and accession alone is fine.
The core answer. A UniProt accession identifies an entry, a protein-coding gene's protein, not a single sequence. The canonical sequence is the entry with no suffix (also written -1); isoforms are the same accession with -N, so P04637-2 is a distinct sequence with its own residue numbering. So store four things:
- the accession (
P04637): the stable entry key; - the isoform identifier (
P04637-2, or an explicit canonical flag); - the sequence version and the CRC64 checksum of the exact sequence the coordinates were computed against;
- the UniProt release you ingested (e.g.
2024_05), so re-validation has a baseline.
The checksum is the field that does the work. Accession + isoform says which sequence you meant; the checksum proves the sequence you used is that one. The accession is stable across releases; the sequence is not.
What an accession-only join does. Three failures, none of which raise an error:
- Fan-out. One accession maps to N isoforms, so joining a feature table to an isoform-keyed table on accession alone multiplies rows. Dedupe afterwards and you've silently picked one isoform: usually whichever the database returned first.
- Coordinate corruption. Residue numbering is per-isoform. Isoforms come from alternative splicing, alternative promoters or alternative initiation, so they can differ at the N-terminus, the C-terminus, or internally. Where the N-terminus differs, the same number is a different residue; where the isoform is shorter, the position may not exist at all.
- False conservation. Comparing a site across species by position, without isoform context, calls a site conserved when the ortholog lacks the exon that carries it.
Worked example: hypothetical numbers, real mechanism. Feature row: P04637-2, phosphosite at position 315.
| Join key | Row matches | Position 315 resolves to |
|---|---|---|
P04637 (canonical) | yes: wrong sequence | canonical residue 315 |
P04637-2 | yes | isoform-2 residue 315 |
If isoform 2 and canonical share residues 1–314 and diverge after, position 315 is a different residue in each: the join succeeds and returns a plausible-looking answer that is wrong. If isoform 2 were the shorter one and the site sat past its end, the join would return a position that doesn't exist in canonical's frame at all.
Alternatives, and when to pick them.
- Accession + isoform + sequence version + checksum. Most precise; joins to isoform-keyed external resources (PDB chains, RefSeq, Ensembl transcripts) work. Costs an extra column and forces every downstream join to carry the isoform.
- Accession + checksum only. Sequence-identity keyed, so it survives renumbering and merges, but you lose the human-readable isoform label and can't join to resources keyed by isoform ID.
- Accession only. Fine for gene-level questions, wrong the moment you store a residue coordinate.
Deciding criterion: does the row carry a residue coordinate, or must it join to an isoform-specific external resource? Yes to either means you need the isoform.
Detection. Checksum mismatch on re-ingest; residue position out of range for the sequence; row-count inflation across a join. Re-validate mappings after each UniProt release: annotation and isoform sets change between releases even when the accession doesn't.
Curated: · Written: · Reviewed:
Q&A-79Counts used Ensembl 110 GTF and variants used 113. What breaks, and how do you keep a coordinate system?
Core answer
Ensembl 110 and 113 are both GRCh38, so the chromosome coordinates are the same map: a variant at 1:1,000,000 is 1:1,000,000 in both. What changes between releases is the annotation: gene models, exon boundaries, gene/transcript ID versions, and which features exist at all. So the failure is not a wholesale coordinate break; it is feature-level misassignment. A variant or read that lands in an exon in 113 can fall in an intron, a retired gene, or nothing in 110.
The fix is to pin one release across counts, variants and annotation, and record assembly plus release in the manifest. If you must move, re-annotate against the target release: do not liftover, because liftover chains exist between assemblies (GRCh37→GRCh38), not between annotation releases that share an assembly.
What actually changes 110 → 113
- Gene and transcript models are revised: exon boundaries move, transcripts are retired or added.
- Gene ID versions bump: ENSG00000139618.14 → .15 when the structure changes. The stable ID is unchanged; the versioned model is not.
- Contig naming differs by source: Ensembl GTF uses
1,2, …,X,Y,MT; UCSC and GENCODE usechr1,chrM. A VCF onchr1against an Ensembl GTF on1matches nothing, silently. - Patch scaffolds and
_PAR_Ycontigs appear and disappear between releases.
What breaks, and how to detect it
| Mismatch | Symptom | Check |
|---|---|---|
chr1 vs 1 | 0% of variants overlap any feature | count variants with no feature overlap |
| Exon boundary moved | variant in exon in 113, intron in 110 | diff exon intervals per gene ID |
| Gene ID version drift | same ENSG, different model | diff version suffixes across releases |
| Retired/added gene | feature vanishes or appears | set difference of gene IDs |
A concrete check: intersect VCF positions with GTF exons. If 100% of variants have zero overlap, it is contig naming, not biology. If a few percent lose overlap, it is model revision. (Illustrative: a 3–6% loss is the shape of a release-drift bug, not a coordinate break.)
Pin vs lift vs re-annotate
- Pin one release for FASTA, GTF and VCF. Cheapest and correct; the cost is re-running counts when you upgrade. Deciding criterion: if the annotation is a reference rather than the object of study, pin it.
- Liftover only for an assembly change (GRCh37→GRCh38), with a chain file such as
hg19ToHg38.over.chain.gzvia CrossMap or PicardLiftoverVcf. It does not fix release drift on the same assembly. - Re-annotate with VEP or
bcftools annotateagainst the target release GTF. Correct for release drift; costs a re-run.
Failure modes
Contig-naming mismatch produces empty annotations with no error: the pipeline runs green and every downstream count is zero. Version drift produces a small, plausible-looking fraction of features in the wrong compartment, which survives review because the numbers look normal. If the FASTA itself were GRCh37 against a GRCh38 GTF, that is a real coordinate break, and no amount of re-annotation fixes it: check the assembly identifier before anything else.
Curated: · Written: · Reviewed:
Q&A-81Coverage 29.6× was stored as 30 and used as a gate of ≥30. What class of samples did that drop or keep wrongly?
The class is samples whose true mean falls in the half-unit band below the gate, [29.5, 30.0). Rounding to nearest integer promotes every one of them to 30, so they pass a ≥30 gate they should fail. Nothing at or above 30.5 is affected, and nothing below 29.5 is affected: the error is confined to a 0.5× window straddling the threshold. If the pipeline truncates instead of rounding, the same window sits above the gate, [30.0, 31.0), and those samples are wrongly dropped.
That is the arithmetic. The class that actually matters is narrower, and it is not defined by the mean at all.
Mean depth is a first moment. A gate on it asks "on average, is this sample deep enough?" Variant recall depends on breadth, the fraction of the target region covered at ≥20×, because a call needs local depth, not average depth. Two samples can share a mean of 29.6 and behave completely differently: one with uniform depth has ~99% of bases at ≥20× and calls fine; one with a collapsed region has 71% breadth and loses variants there. Rounding moves both across the gate together.
| Sample | True mean | Breadth ≥20× | Stored | ≥30 gate | Correct call |
|---|---|---|---|---|---|
| A | 29.6 | 71% | 30 | pass | fail: patchy |
| B | 29.6 | 99% | 30 | pass | pass: uniform |
| C | 30.4 | 62% | 30 | pass | fail: patchy |
| D | 30.4 | 98% | 30 | pass | pass |
Rounding wrongly keeps A. C is kept wrongly too, but the mean gate never should have passed it in the first place: that is the breadth problem, not the rounding problem. The mirror case is dropped wrongly: a sample at 29.4 with 99% breadth, stored as 29, fails a gate it would have passed on breadth.
Worked arithmetic: if 8% of a 1,000-sample cohort sits in [29.5, 30.0), rounding misclassifies ~80 samples on the mean alone. If breadth is the real criterion, the misclassified set is larger: every sample with mean ≥30 and breadth below the breadth threshold passes the mean gate and should not.
Trade-offs between the fixes:
- Store the float, gate on the float. Removes the quantization error entirely and costs nothing but a schema decision. It does not fix the breadth problem.
- Gate on breadth (fraction of bases ≥20×). Directly matches what recall depends on; needs a defined target region and a threshold, commonly 90–95% at 20×. It fails samples that are deep but unevenly so, which is the point.
- Gate on median depth. Robust to a collapsed region dragging the mean down, but blind to a small collapsed region the median ignores.
- Gate on both mean and breadth. Common in production QC; two thresholds to tune, and a sample can fail either.
Resequencing cost decides the direction: dropping a sample costs a lane and a turnaround, while keeping a bad one costs a false-negative variant and a downstream re-run. That asymmetry argues for a conservative breadth gate rather than a rounded mean.
Detection: keep raw mean and breadth in the QC table, log the stored value alongside them, and count samples in the half-unit band each run. A nonzero count means the gate is quantizing.
Curated: · Written: · Reviewed: