Evaluation
Agents' Last Exam
Category
Evaluation
Abstraction level
Benchmark
Operation level
Evaluation (runtime)Agent runtime
Use cases
Evaluating autonomous AI agentsComparing planning and tool-use abilitiesTracking progress of agentic systems
How it works
Agents are given complex goals requiring sequences of actions — planning, calling tools/APIs, web search and reasoning. Scoring assesses whether the agent autonomously achieves the goal, often accounting for cost and number of steps.
Problem solved
How to reliably measure the frontier of AI agent capabilities on hard, realistic multi-step tasks?
Strengths & limitations
Strengths
✓Measures real multi-step agentic abilities
✓Difficulty marks the capability frontier
✓Reflects practical tool-use scenarios
Limitations
✗Results sensitive to agent and tooling configuration
✗Costly and time-consuming evaluation
✗Risk of overfitting to the specific benchmark