How do you evaluate an AI agent's performance?

Agent evaluation goes beyond single-response accuracy because outcomes depend on a trajectory of steps. Measure end-to-end task success (did it achieve the goal, verified against ground truth or a checkable outcome) as the headline metric. Also assess the trajectory: did it pick the right tools, in a sensible order, without wasted steps? Track efficiency (number of steps, tokens, latency, dollar cost), and reliability (success rate across many runs, since agents are stochastic). Use curated benchmark tasks and held-out test suites; combine automated checks (assertions, unit-test-style graders) with LLM-as-judge for subjective quality, and human review for high-stakes cases. Watch for reward hacking and overfitting to the eval set. Because agents are non-deterministic, run each case multiple times and report success rate and variance, not a single pass/fail.