How do you evaluate an LLM application before and after deployment?

Build a golden set of representative inputs with expected outputs or grading criteria, and run it in CI on every prompt or model change — block regressions. Use automated graders (exact match, rubric-based, or LLM-as-judge with care) plus human review for nuanced cases. In production, sample real traffic, log inputs/outputs, collect user feedback, and feed failures back into the eval set. Evals turn model upgrades and prompt tweaks from leaps of faith into measured engineering.