How do you evaluate LLM outputs, and what metrics do you use?

Match the method to the task. For tasks with a reference answer (translation, extraction), use overlap or similarity metrics (exact match, F1, BLEU/ROUGE, embedding similarity). For open-ended generation, reference metrics break down, so use LLM-as-a-judge scoring against a rubric, or human rating. For RAG, measure retrieval quality (are the right chunks fetched?) and faithfulness (is the answer grounded in them?) separately. Task-specific checks matter too: does the JSON parse, is the code runnable, are required facts present. Always evaluate on a fixed, representative dataset so results are comparable over time, report the metric distribution not just an average, and pair automated scores with periodic human spot-checks to catch what metrics miss.