Q1.Why is evaluating LLM output generally harder than evaluating a traditional classifier?
Q2.What do BLEU and ROUGE primarily measure?
Q3.What is a key limitation of BLEU/ROUGE for evaluating open-ended generation?
Q4.What is "LLM-as-judge"?
Q5.What is a known bias risk of LLM-as-judge evaluation?
Q6.Why is using the same model to both generate and judge its own answers considered risky?
Q7.What does "faithfulness" measure in RAG-specific evaluation?
Q8.What does "retrieval precision" measure?
Q9.Why is human evaluation still considered valuable despite being slow and expensive?
Q10.What is a "golden dataset" in the context of LLM evaluation?
Q11.Why does a held-out, representative test set matter more than which specific metric is chosen?
Q12.What is "pairwise comparison" evaluation?
Q13.What is a red-teaming exercise in the context of LLM evaluation?
Q14.What does "regression testing" mean when applied to an LLM-based system?
Q15.Why can a benchmark like MMLU be a poor proxy for how well a model performs on a specific production use case?
Q16.What is "calibration" in the context of model evaluation?
Q17.What is a practical reason to validate an LLM-as-judge against human ratings before trusting it at scale?
Q18.Why do reference-based metrics (BLEU/ROUGE) struggle specifically with tasks that have many valid correct answers (e.g. open-ended summarization)?
Q19.What is an example of a task-specific grounded metric used in evaluating RAG systems specifically (beyond generic text-quality metrics)?
Q20.Why might an evaluation pipeline track both automated metrics and periodic human spot-checks in production?
Q21.What's a risk of using an evaluation set that's too small or unrepresentative of real user queries?
Q22.What does it mean for an LLM-as-judge to use a "rubric"?
Q23.Why is it important to distinguish whether an evaluation failure originates in retrieval vs. generation for a RAG system?
Q24.What is a common evaluation approach for detecting hallucination specifically?
Q25.Why might A/B testing in production be used alongside offline evaluation for an LLM feature?
