Evaluation

AutomationBench

2026ActivePublished: 23 September 2026Updated: 23 September 2026Published

Key innovation

The first benchmark to combine cross-application coordination, autonomous REST API endpoint discovery and business-rule adherence in simulated SaaS environments, graded purely on end-state (no LLM judge).

How it works

The agent receives an instruction and access to a set of REST API tools representing simulated SaaS applications. It must autonomously discover the right endpoints, execute the sequence of API calls that accomplish the task, and follow business rules while ignoring decoy records. When finished, a grader compares the environment's end state against the expected state: a partial_credit metric (0.0–1.0) measures the share of satisfied assertions, while task_completed_correctly is a strict pass/fail for the whole task. Grading is deterministic, with no LLM judge involved.

Problem solved

Existing automation benchmarks rarely combined cross-application coordination, autonomous discovery of the right API endpoints and adherence to business rules and policies at once — and rarely tested agents in environments seeded with irrelevant or misleading data. AutomationBench measures an agent's ability to carry a realistic, multi-step business workflow end-to-end and leave the systems in the correct state.

Components

Simulated SaaS environmentsTask execution environment

Isolated, simulated SaaS applications (CRM, inbox, calendar, sheets, etc.) seeded with data including decoy records; the agent acts only through REST API calls. The public set spans 47 tools.

Task setTask and domain definition

600 scored tasks across 6 domains (sales, marketing, operations, support, finance, HR; 100 each) plus an unscored "simple" set of 200 one- to two-step tasks. The official leaderboard uses a separate, harder, held-out private set.

Programmatic end-state graderScoring

Deterministic assertion-based grader: partial_credit (0.0–1.0) and task_completed_correctly (pass/fail). Compares the final state against the expected state, with no LLM judge.

Implementation

Reference implementations

zapier/AutomationBench (GitHub)

Python · Zapier

Official

Implementation pitfalls

Confusing with AutomationBench-AAMedium

AutomationBench (Zapier) and AutomationBench-AA (Artificial Analysis's independent leaderboard on a private subset) use different metrics. Zapier's leaderboard reports the share of tasks completed fully; AutomationBench-AA reports the average share of objectives completed without guardrail violations — which is why its numbers tend to be much higher.

Fix:Always state which leaderboard/metric is quoted and on which task set (public vs private).

Comparing scores across task setsLow

The public set (600 tasks) is easier than the private set used by the official leaderboard; scores are not directly comparable.

Fix:Only compare results from the same task set and benchmark version.

Evolution

Original paper · 2026 · arXiv preprint (cs.AI) · Daniel Shepard

AutomationBench

Daniel Shepard, Robin Salimans

2026

AutomationBench released (Zapier)

Inflection point

Zapier publishes the arXiv paper (21 April 2026) and the public 600-task set on GitHub; the best frontier models score below 10% on the hard set.

AutomationBench (paper)

2026

AutomationBench-AA independent leaderboard launched (Artificial Analysis)

Artificial Analysis, in partnership with Zapier, launches an independent leaderboard on a private task subset, with a metric based on the average share of objectives completed without triggering guardrail violations (a distinct entity from AutomationBench itself).