Evaluating agents

In plain English

Testing a chat reply is hard; testing an agent is harder. Success might mean the right database row updated, three tools called in order, and no unauthorized email sent—not a pretty paragraph.

Agent evaluation means scoring whole runs: did it finish the task, how many steps, where did it diverge, and did it break rules?

How it works

Build a task suite with clear success checks: mocked APIs that record calls, golden files, human rubrics for fuzzy outputs. Run the agent on each task, log traces (plan, tools, observations), and aggregate pass rates.

Regression tests catch prompt or model upgrades that silently break tool choice. Red-team tasks probe unsafe tool use and data exfiltration.

• Outcome checks: final state matches expected.
• Process checks: required tools used, forbidden tools absent.
• Efficiency: step count and token cost within budget.
• Robustness: variation in phrasing and noisy tool responses.

Going deeper

LLM-as-judge can grade open-ended steps but should not be the only signal—pair with deterministic asserts where possible.

Production monitoring mirrors offline evals: sample live traces, alert on failure patterns, keep human review for edge cases.

Common misconceptions

A high benchmark score on a base LLM predicts agent success.

Tool orchestration, memory, and permissions dominate many agent failures.

One end-to-end demo is enough validation.

Agents fail on phrasing shifts, empty tool results, and edge permissions.

Automated judges replace all human review.

Judges drift and hallucinate scores; use them as one signal among many.

Key facts

• Agent evals measure trajectories and environment effects, not just text.
• Task suites should include tool mocks with assertable side effects.
• Trace logs enable debugging of wrong tool choice and planning loops.
• Regression runs catch changes in models, prompts, and tool schemas.
• Safety evals test refusal, scopes, and data handling—not only task completion.