AI agent evaluation: how to know it actually works
To test an AI agent, give it real tasks where you can check the result, and count how often it succeeds. That number is the task success rate. Also watch how consistent, fast, and costly the agent is. Public tests like SWE-bench, GAIA, τ-bench, and OSWorld compare models; your own set of 20–50 real cases tests your agent.
A demo shows an agent can succeed once. Evaluation asks the question that really matters: how often does it succeed, what does it cost, and how does it fail?
The key number is the task success rate. Pick a set of real tasks where you can check the result. Run the agent. Count the wins. Public benchmarks — shared tests anyone can run — work the same way. SWE-bench Verified tests coding agents on real GitHub bug reports, checked by each project's own tests [1]. GAIA tests general assistants on questions that take tools and several steps to answer [2]. τ-bench tests customer-service agents talking to a simulated customer, with rules they must not break [3]. And OSWorld tests agents that control a computer, on real desktop tasks [4].
Benchmarks help you choose a model. They say little about your task. For that, build your own test set — it is step five of the build sequence, and the one teams skip. Even 20–50 real cases with clear pass rules beats gut feeling. Score the set the same way every time you change a prompt, tool, or model [5]. Some results can't be checked automatically. The usual fix is to have a second AI model grade the answer against a checklist — people call this "LLM-as-judge". Compare its grades with human grades on a sample before you trust it.
One success is a demo. A success rate is an evaluation.
What does a success rate hide?
Two agents with the same average score can behave very differently. τ-bench has a measure called pass^k that shows this. It asks: if you run the same task k times, does the agent succeed every time? Even strong agents drop sharply as k grows [3]. Also track cost and speed per task — an agent pays for every step it takes, not just the final answer. And read the failed runs. Where the agent went wrong matters more than how often. Picking the wrong tool, ignoring a tool's output, and never knowing when to stop are different failure modes with different fixes. You can only tell them apart by reading the run.
Sources
- Jimenez et al. — SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv 2310.06770, Oct 2023
- Mialon et al. — GAIA: A Benchmark for General AI Assistants, arXiv 2311.12983, Nov 2023
- Yao et al. (Sierra) — τ-bench: A Benchmark for Tool-Agent-User Interaction, arXiv 2406.12045, Jun 2024
- Xie et al. — OSWorld: Benchmarking Multimodal Agents in Real Computer Environments, arXiv 2404.07972, Apr 2024
- Claude docs — Define success criteria and build evaluations (platform.claude.com)
Frequently asked questions
How do you measure whether an AI agent works?
Give the agent a set of real tasks where you can check the result. Count how often it succeeds — that is the task success rate. Then repeat the runs to see if it stays consistent, track cost and speed per task, and read the failed runs to see where the agent went wrong.
What benchmarks exist for AI agents?
SWE-bench Verified tests coding agents on real GitHub bug reports. GAIA tests general assistants on multi-step questions that need tools. τ-bench tests customer-service agents talking to simulated customers under set rules. OSWorld tests agents that control a computer on real desktop tasks. Benchmarks help you pick a model; your own test set checks your agent.
What is pass^k?
A reliability score from τ-bench. It measures the chance an agent succeeds on every one of k repeat runs of the same task, not just once. Even strong agents drop sharply as k grows. That is why one average success rate can hide real inconsistency.