The agent receives an instruction and access to a set of REST API tools representing simulated SaaS applications. It must autonomously discover the right endpoints, execute the sequence of API calls that accomplish the task, and follow business rules while ignoring decoy records. When finished, a grader compares the environment's end state against the expected state: a partial_credit metric (0.0–1.0) measures the share of satisfied assertions, while task_completed_correctly is a strict pass/fail for the whole task. Grading is deterministic, with no LLM judge involved.
Existing automation benchmarks rarely combined cross-application coordination, autonomous discovery of the right API endpoints and adherence to business rules and policies at once — and rarely tested agents in environments seeded with irrelevant or misleading data. AutomationBench measures an agent's ability to carry a realistic, multi-step business workflow end-to-end and leave the systems in the correct state.
Isolated, simulated SaaS applications (CRM, inbox, calendar, sheets, etc.) seeded with data including decoy records; the agent acts only through REST API calls. The public set spans 47 tools.
600 scored tasks across 6 domains (sales, marketing, operations, support, finance, HR; 100 each) plus an unscored "simple" set of 200 one- to two-step tasks. The official leaderboard uses a separate, harder, held-out private set.
Deterministic assertion-based grader: partial_credit (0.0–1.0) and task_completed_correctly (pass/fail). Compares the final state against the expected state, with no LLM judge.
AutomationBench (Zapier) and AutomationBench-AA (Artificial Analysis's independent leaderboard on a private subset) use different metrics. Zapier's leaderboard reports the share of tasks completed fully; AutomationBench-AA reports the average share of objectives completed without guardrail violations — which is why its numbers tend to be much higher.
The public set (600 tasks) is easier than the private set used by the official leaderboard; scores are not directly comparable.
Zapier publishes the arXiv paper (21 April 2026) and the public 600-task set on GitHub; the best frontier models score below 10% on the hard set.
Artificial Analysis, in partnership with Zapier, launches an independent leaderboard on a private task subset, with a metric based on the average share of objectives completed without triggering guardrail violations (a distinct entity from AutomationBench itself).
Public (600 tasks, GitHub) vs private/held-out (Zapier's official leaderboard).
The leaderboard reports results at different reasoning-effort levels of the evaluated model (e.g. max / xhigh).
Six domains: sales, marketing, operations, support, finance, HR (100 tasks each in the public set).
