LLM Evaluation

Classifiers have a single correct label to check predictions against, giving an unambiguous accuracy metric. LLM outputs are open-ended free text where multiple different responses can be equally valid, so "correctness" itself requires a judgment call — via reference comparison, a rubric, an LLM-judge, or human review — rather than a simple equality check.

BLEU and ROUGE measure n-gram or word overlap between generated text and one or more reference texts. Because they reward surface-level textual overlap rather than semantic correctness or quality, a generated response can be semantically excellent but score poorly if phrased differently from the reference, or score well while being subtly wrong if it happens to reuse reference wording.

A typically stronger or differently trained LLM scores or compares responses against a rubric. Known biases include favoring longer or more confident-sounding responses, positional bias in pairwise comparisons (favoring whichever response is presented first or second), and self-preference bias if the judge shares training lineage or style with the model being evaluated.

The judge may systematically prefer outputs that match its own generation style and quirks, since it's essentially judging by "does this look like what I'd produce." This inflates scores for the generating model's own patterns rather than reflecting true quality — a different, ideally stronger, model is generally preferred as judge to reduce this self-preference bias.

Faithfulness checks whether the answer's claims are actually supported by the retrieved context, not fabricated or extrapolated beyond it. Answer relevance checks whether the answer actually addresses the user's question. An answer can be perfectly faithful to context yet fail to actually answer what was asked, or vice versa, so both need to be measured separately.

Even a well-designed metric produces a misleading, falsely confident signal if it's computed over unrepresentative examples, such as easy hand-picked cases that don't reflect real usage. The test set's representativeness determines whether a good score actually predicts real-world performance, which no amount of metric sophistication can fix on its own.

Sample real or realistic representative user queries spanning common and edge cases, pair each with an expected or ideal response or a set of acceptable criteria written or reviewed by domain experts, and periodically refresh it to reflect evolving product or support scenarios so it doesn't go stale relative to actual usage.

Two candidate responses are shown side by side and a judge, human or LLM, picks the better one, rather than independently scoring each on an absolute scale. Relative judgments ("which is better") tend to be more consistent and reliable than absolute scores ("rate this 1-10"), since absolute scales are prone to drifting calibration across different raters or sessions.

Red-teaming deliberately crafts adversarial inputs designed to elicit undesired behavior — harmful content, jailbreaks, policy violations, leaking system prompts — to proactively find failure modes before real users or bad actors do. For example, it might find that a rephrased or obfuscated harmful request bypasses a safety filter that catches the direct phrasing.

A prompt tweak or model upgrade intended to fix one issue can silently break previously-working behavior elsewhere. Re-running a fixed, comprehensive evaluation set after every change catches such regressions before they reach users, rather than relying on spot-checking only the specific case that motivated the change.

General benchmarks test broad academic or knowledge coverage across many domains in a fixed format, which may not resemble the actual domain, task structure, or input distribution of a specific application. A model can excel at broad benchmarks while still performing poorly on a narrow, differently-formatted real task it wasn't specifically evaluated against.

Calibration is whether a model's expressed confidence matches its actual accuracy — among answers it states with 90% confidence, roughly 90% should actually be correct. In decision-support contexts, poor calibration is dangerous because users may over-trust confidently-stated wrong answers or under-trust correct hedged ones, undermining the value of confidence signals entirely.

Collect a sample of responses, have humans independently rate them against the same rubric the judge will use, then compare the judge's scores or rankings to the human ratings, such as agreement rate or correlation. Only adopt the judge for large-scale evaluation once it shows sufficiently strong agreement with human judgment on this validation sample, and periodically re-check as the model or task evolves.

They isolate which part of a compound system is failing — a low answer-quality score alone doesn't say whether the problem is retrieval (wrong or missing chunks) or generation (the model ignoring good context). Grounded, component-specific metrics let you diagnose and fix the actual failing stage rather than guessing.

A demo is typically a handful of hand-picked or lucky examples, which doesn't reveal how the change performs across the full distribution of real inputs. A change can look impressive in a demo while introducing regressions or failing on common but less flashy cases that were never checked, giving false confidence in the change.

For each generated claim, check whether it's supported by a trusted reference — retrieved context, a knowledge base, or ground truth — using either an NLI/entailment-style automated check or an LLM-judge prompted specifically to verify claim-by-claim support, flagging any unsupported claim as a potential hallucination rather than scoring the answer only holistically.

Offline evaluation is fast, repeatable, and catches regressions before deployment, but it can't fully capture real user behavior, satisfaction, or downstream business metrics. A/B testing measures actual impact on real users and traffic, catching effects offline metrics might miss or get wrong, while still needing offline evaluation as a cheaper first gate before committing to a live experiment.

Break "quality" into concrete, separately-scoreable dimensions — factual accuracy, tone appropriateness, whether it fully answers the question, policy compliance — rather than one vague "rate this response" prompt. Specific criteria are both easier for the judge to apply consistently and easier to validate against human agreement per-dimension.

Human evaluation is slow and expensive relative to automated or LLM-judge scoring, making it impractical to run on every prompt tweak or frequent iteration. It's instead used strategically — to validate that cheaper automated signals track real quality, and for critical launch decisions where the cost is justified by the stakes.

Retrieved chunks are all genuinely relevant (high precision) but the LLM still produces a poor answer — for example, misreading the context, failing to synthesize across multiple relevant chunks, or ignoring context in favor of its own prior knowledge. This implies the fix belongs in generation (prompting, context formatting, or model choice), not retrieval, since retrieval is already doing its job.

Production input distributions, model versions, and user behavior can drift over time, so a metric or an LLM-judge validated against human judgment on an earlier sample may no longer track quality as reliably on newer traffic or tasks. Periodic re-validation catches this drift before decisions are made on a stale, no-longer-trustworthy signal.

Response length or fluency scores are easy to compute automatically but don't measure whether the response is actually correct, helpful, or relevant. Optimizing for such easy-to-compute proxies without checking they correlate with real user satisfaction risks improving the metric while the actual user experience stays flat or worsens.

The judge may systematically favor whichever response appears first or second in the prompt, regardless of actual quality. Mitigate by running each comparison twice with the response order swapped and checking for consistent judgments, or averaging across both orderings to cancel out the positional effect.

As the product, user base, or model capabilities evolve, a static evaluation set can become unrepresentative of current real-world usage, missing new query types, edge cases, or failure modes that have since emerged. A stale test set can keep reporting good scores while missing genuinely new problems users are actually hitting.

Score factual correctness or faithfulness separately from a distinct dimension like clarity, completeness, or tone — either via separate rubric criteria within one LLM-judge pass, or entirely separate evaluation passes. Conflating them into one holistic score obscures which dimension actually needs improvement when quality is judged unsatisfactory.