What is LLM-as-a-judge evaluation, and what are its limitations?

LLM-as-a-judge uses a strong model to score or compare outputs against a rubric — rating a response 1–5 for helpfulness, or picking which of two answers is better. It scales far better than human review and handles open-ended quality that lexical metrics can't. But it has real limitations: judges show biases (favouring longer answers, the first option presented, or their own model family's style), can be inconsistent run to run, and may confidently mis-grade on domains they're weak in. Mitigate with a clear rubric, few-shot examples, randomising option order, using pairwise comparison rather than absolute scores, and calibrating the judge against a human-labelled sample. Treat it as a strong, cheap signal to validate against humans — not ground truth on its own.