Most AI projects don’t fail because the model is bad. They fail because nobody could tell whether the model was bad. A demo that works on five hand-picked prompts feels finished — right up until real users send the sixth prompt. Evaluation is the discipline that separates a working AI product from a convincing demo, and it’s the single most underrated skill in AI engineering.
This guide covers how to measure LLM quality honestly: what to measure, how to measure it, and how to avoid the traps that make a system look better than it is. It pairs with our AI interview questions on evaluation & testing if you’re preparing for a role.
Why evaluating LLMs is genuinely hard
Traditional software is deterministic: the same input gives the same output, so a test asserts an exact expected value. LLMs break both halves of that assumption. They’re probabilistic — the same prompt can yield different wordings, all correct — and open-ended, so there’s often no single right answer to assert against. Worse, quality is partly subjective (is this summary good?), and a model can silently regress on things you never tested the moment you change a prompt or upgrade a model version.
The consequence: exact-match unit tests are almost useless on their own. Instead you evaluate on representative datasets using a blend of automated metrics, model-graded scoring, and human review — and you re-run it continuously, because behaviour drifts.
The three families of metrics
1. Reference-based (lexical) metrics
When you have a known correct answer, you can measure overlap with the model’s output. BLEU measures n-gram precision (how much of the output appears in the reference) and comes from machine translation. ROUGE measures recall (how much of the reference is captured) and is standard for summarisation. Both are cheap and reproducible — but purely lexical. They punish valid paraphrases that use different words, and they can’t judge meaning. Use them for constrained tasks like translation or extraction, not open-ended generation.
2. Semantic metrics
BERTScore and embedding-similarity metrics compare the meaning of two texts rather than their exact words, so they credit a good paraphrase that BLEU would penalise. They cost more compute and still need a reference answer, but they’re a better fit when wording is free to vary.
3. Model-graded evaluation (LLM-as-a-judge)
For open-ended quality that lexical metrics can’t touch, you use a strong model to score outputs against a rubric — rating a response 1–5 for helpfulness, or picking which of two answers is better. This scales far beyond human review. The landmark study on this approach, MT-Bench and Chatbot Arena, found LLM judges can agree with humans over 80% of the time — but also documented their biases: judges favour longer answers, the first option presented, and their own model family’s style, and can be inconsistent run to run.
G-Eval (arXiv:2303.16634) improves reliability by first having the model generate explicit evaluation steps from the criteria, then scoring against them — and by weighting the score across the probability of rating tokens rather than a single sampled number. It correlates better with human judgement than naive prompting, but still inherits judge bias, so you calibrate it against human labels rather than trusting it blindly.
The practical rule: use a clear rubric, prefer pairwise comparison over absolute scores, randomise option order, and validate the judge against a human-labelled sample before you rely on it.
Evaluating RAG systems: separate the stages
Retrieval-Augmented Generation is where most production LLM apps live, and evaluating it well means resisting the urge to score only the final answer. Evaluate the two stages separately:
- Retrieval: on a labelled set of questions with known relevant documents, measure recall and precision — did the right chunks get fetched, and how many irrelevant ones came along? Use hit rate and ranking metrics like MRR or NDCG.
- Generation given context: measure faithfulness (is every claim grounded in the retrieved chunks, with no hallucination?) and answer relevance (does it actually address the question?). Frameworks like RAGAS formalise exactly these dimensions.
This separation is the whole point: a wrong final answer is either a retrieval miss (fix chunking, embeddings, or re-ranking) or a grounding failure (fix the prompt). Only stage-wise metrics tell you which one to go fix. If you’re choosing the underlying model for the generation step, our AI tools comparison breaks down the trade-offs.
Measuring hallucination
Hallucination is fluent output that’s unsupported or false. To measure it, define it against a source. In RAG, that’s the faithfulness check above — an LLM-judge (or a natural-language-inference model) flags claims not entailed by the retrieved context. Other signals: self-consistency (sample several answers; wide disagreement means the model is guessing) and asking for citations you can automatically verify. Report a hallucination rate on a labelled set and track it over time. You can lower it a lot with grounding, low temperature, and an explicit “say you don’t know” instruction — but never to zero, so keep human verification for high-stakes facts.
Public benchmarks: useful, but don’t be fooled
Benchmarks like MMLU (arXiv:2009.03300, broad multiple-choice knowledge), HumanEval (code that’s run against unit tests), and GSM8K (grade-school math reasoning) are useful for roughly comparing base models — and holistic efforts like HELM push for measuring many dimensions at once, not a single score.
But interpret them cautiously. Benchmark contamination — test questions leaking into training data — inflates scores. A high average hides weakness on your specific domain. A model topping MMLU may still be worse for your particular RAG chatbot. Use public benchmarks to shortlist candidates, then run your own task-specific evaluation on data that reflects real usage. That’s what actually predicts performance for you.
Golden datasets and evaluation-driven development
The backbone of all of this is a golden dataset: a curated, versioned set of inputs paired with known-good expected outputs (or acceptance criteria) that represents your real use cases and important edge cases. It’s the fixed yardstick every prompt and model change is scored against, so regressions are immediately visible.
Wire it into CI and you get evaluation-driven development — the AI equivalent of test-driven development. You define how you’ll measure success before you build, then iterate on prompts, retrieval, and models with the eval as your feedback loop. Every change is scored; you keep what improves the number and revert what doesn’t. It turns prompt engineering from vibes-based tweaking into measurable progress. Open frameworks like OpenAI Evals give you a structure to codify these tests.
Keep the golden set out of any training or few-shot data to avoid contaminating the measurement, and grow it whenever a production bug reveals a gap.
Offline vs online: you need both
Offline evaluation runs before deployment on your fixed dataset — fast, reproducible, and it gates releases. But it only measures what your dataset covers. Online evaluation happens in production on live traffic: A/B tests and real user signals (thumbs up/down, edits, task completion, retention), plus sampling real outputs for review. It reflects reality but is slower and noisier.
Ship with offline confidence; learn what actually works — and catch the drift your offline set never anticipated — with online signals.
The honest bottom line
There is no single perfect metric, and anyone selling you one is overselling. Real evaluation combines automated checks, model-graded judging, and human review, run continuously against a dataset you trust. It’s less glamorous than a launch demo — and it’s the difference between an AI feature that survives contact with users and one that quietly embarrasses you. If you’re building, start your golden dataset today, before you write another prompt.