Retrieval-Augmented Generation (RAG)

Source documents are split into chunks, each chunk is embedded into a vector and stored in a vector index. At query time, the user's query is embedded the same way, the index returns the top-k most similar chunks, optionally a reranker re-scores them for precision, and the selected chunks are inserted into the LLM's prompt as context so it generates an answer grounded in that retrieved text rather than purely from its parameters.

Small chunks produce precise, focused embeddings that retrieve well for narrow queries, but can lose surrounding context needed to fully understand a fact, and may require retrieving more chunks to cover a topic. Large chunks preserve more context per chunk, but blend multiple topics into one embedding, making retrieval less discriminative, and eat more of the LLM's context budget per chunk retrieved.

Dense retrieval uses embeddings to match on semantic meaning, capturing similar-meaning text even without exact word overlap. Sparse retrieval (e.g. BM25) matches on exact keyword/lexical overlap, which is often better for rare terms, IDs, acronyms, or exact phrases embeddings might blur past. A hybrid approach combines both signals to get the strengths of each.

A bi-encoder embeds the query and each document independently, then compares the resulting vectors — this lets document embeddings be precomputed and cached, making retrieval fast even over huge corpora. A cross-encoder instead processes the query and a candidate document jointly through the model to produce a relevance score directly, capturing richer query-document interactions and generally scoring more accurately, but it can't be precomputed and must run per pair — too slow to apply to an entire corpus, so it's used only to rerank a small shortlist from the bi-encoder.

LLMs tend to attend more reliably to content near the beginning or end of a long context than content buried in the middle, likely reflecting patterns in how they were trained. Mitigations include keeping the retrieved context shorter (fewer, higher-quality chunks rather than many marginal ones), placing the most relevant chunk near the start or end of the prompt, and using a reranker to ensure the best chunk is positioned advantageously.

Inspect the actual chunks that were retrieved for the failing query. If the chunk containing the answer-critical information isn't among them, it's a retrieval problem — investigate chunking, embedding model choice, or top-k. If the correct chunk was retrieved but the LLM's answer doesn't use it, contradicts it, or ignores it in favor of its own prior knowledge, it's a generation problem — investigate prompt design or how context is presented to the model.

Faithfulness is whether every claim in the generated answer is actually supported by the retrieved context, rather than fabricated or drawn from the model's own prior knowledge. It's commonly measured with an LLM-as-judge that compares each claim in the answer against the retrieved passages, or with NLI-style entailment checks that verify the context entails each claim.

Brute-force search compares the query vector against every stored vector, which is O(n) per query — infeasible at millions or billions of vectors. HNSW builds a multi-layer navigable graph over the vectors, enabling search to traverse toward the nearest neighbors in roughly logarithmic time instead of scanning everything, trading a small chance of missing the true exact nearest neighbor for a large speed improvement.

HyDE generates a hypothetical answer to the query using an LLM first, then embeds that hypothetical answer (instead of the raw query) to retrieve real documents similar to it. It helps when queries are short, vague, or phrased very differently from how the source documents word the answer, since the hypothetical answer's language tends to be closer to the vocabulary and structure of the actual source content.

Even a very capable LLM cannot produce a correct grounded answer if it's given the wrong or irrelevant chunks — it simply has nothing correct to work with. In practice, most production RAG failures trace back to poor chunking, a mismatched embedding model, a stale index, or a badly tuned top-k, rather than to a limitation of the generation step itself.

Single-shot top-k retrieval on the raw query often misses chunks needed to answer intermediate sub-questions. Options include iterative or agentic retrieval — retrieve, reason about what's still missing, retrieve again — or a Graph RAG-style approach that explicitly connects related entities so multi-hop relationships can be traversed rather than relying purely on flat-chunk similarity.

Metadata filtering restricts similarity search to chunks matching structured attributes like date, source, or access level, before or alongside the semantic search. For example, "what changed in the Q3 policy" needs filtering to documents actually tagged or dated Q3, since a pure semantic search might surface an older policy version that reads very similarly to the current one.

Precision is the fraction of retrieved chunks that are actually relevant; recall is the fraction of all truly relevant chunks that were actually retrieved. Low precision wastes context budget on noise and can distract the model; low recall means the chunk that actually contains the answer was never retrieved at all, so the model has no chance of answering correctly — a system needs to balance both, often via top-k and reranking.

Even given exactly the right context, the LLM can still ignore it, misinterpret it, or blend it with its own parametric knowledge and extrapolate beyond what the context actually supports. Grounding the model in retrieved text reduces the tendency to hallucinate but doesn't remove the model's underlying capacity to generate unsupported claims.

Favor RAG when the need is knowledge that changes over time, needs citations, or is too large to bake into weights — updating it is cheap, just re-indexing. Favor fine-tuning when the need is a behavior, format, or style change that's hard to reliably get through prompting alone. RAG is generally the cheaper, more current option for injecting facts specifically.

Retrieval metrics (precision, recall, mean reciprocal rank) are computed against a labeled set of query-to-relevant-chunk pairs, checking whether the right chunks were actually surfaced. Generation metrics (faithfulness, answer relevance) assess whether the model's answer is grounded and actually addresses the question, often via LLM-as-judge, validated periodically against human judgment on a sample.

Different embedding models learn different, generally non-comparable vector spaces, so a query embedded by one model isn't meaningfully comparable via cosine similarity to documents embedded by a different model. Using mismatched embedding models for corpus and query breaks similarity search entirely — the same model must be used for both.

If a key sentence is split across two chunks with no overlap, neither chunk contains the complete fact, so neither chunk's embedding may sufficiently match the query, and even if one half is retrieved, the generation step lacks the full context needed to answer correctly — overlap ensures boundary-spanning information still appears intact in at least one chunk.

A reranker adds latency (an extra model pass over the candidate chunks) and compute cost, since it typically can't be precomputed like embedding-based retrieval. It's worth it when the resulting precision/answer-quality improvement matters more than the added latency — often the case for quality-sensitive applications where initial bi-encoder retrieval alone is noisy — and less worth it for latency-sensitive use cases where retrieval is already reasonably precise.

Re-run ingestion (chunk, embed, index) for new or changed documents, and remove or mark stale entries for deleted or superseded documents — ideally incrementally, updating only what changed rather than reindexing the entire corpus every time, so updates stay cheap and fast to apply.

The LLM may attribute a correct claim to the wrong retrieved source chunk, or generate a plausible-looking citation that isn't actually present in the retrieved context, since citation generation isn't inherently verified by the model. It's worth validating citations separately — e.g. checking that the cited text actually appears in the referenced chunk — rather than trusting them at face value.

Balance recall (needing enough retrieved chunks to include the passage that actually supports the answer) against noise, cost, and latency (each extra chunk adds tokens and can dilute the model's attention). This is typically tuned empirically against a labeled evaluation set, often combined with reranking — retrieving a larger initial candidate set and letting the reranker narrow it down to a smaller, higher-precision final set.

Fixed-size chunking splits text at arbitrary character/token boundaries regardless of the document's actual structure, which can cut a fact, table, or logical unit in half — degrading both that chunk's embedding quality and the model's ability to reason over a complete idea. Structure-aware chunking respects natural boundaries like headings and paragraphs, keeping each chunk more self-contained and semantically coherent.

RAG is a poor fit when the need is a stylistic, formatting, or behavioral change rather than a knowledge lookup — fine-tuning suits that better — or when the task requires complex multi-step reasoning or computation rather than fact retrieval, where an agent with tools might fit better. It's also a poor fit when the latency budget can't tolerate an extra retrieval round-trip at all.

Instead of retrieving unstructured text chunks by embedding similarity, Graph RAG retrieves and reasons over a structured knowledge graph of entities and their relationships. It helps for queries requiring explicit relational or multi-hop reasoning — e.g. "who are all the people connected to X through Y" — which is hard to answer reliably from independently-retrieved flat text chunks that don't capture those relationships explicitly.