CursorBench
How it works
Tasks are collected from real Cursor sessions (including the internal codebase and controlled sources), and the Cursor Blame mechanism traces committed code back to the agent request that generated it. Evaluation runs in a hybrid online-offline mode: an offline benchmark is complemented by controlled analysis of live traffic. Because tasks are often underspecified, agentic graders are used for scoring. For each model the suite reports a score, cost per task (computed from token usage and per-million-token pricing: input, cache read/write, output) and the number of steps. Results are subject to variance, so small differences may not be statistically meaningful.
Problem solved
Standard coding benchmarks evaluate models on isolated, well-specified tasks that poorly reflect day-to-day IDE work: multi-file, ambiguous requests where not only correctness but also code quality and efficiency matter. CursorBench aims to measure an agent's real-world usefulness under those conditions.
Components
Tasks drawn from real Cursor IDE sessions, including the internal codebase and controlled sources; refreshed every few months.
Agentic graders used to reliably score solutions despite the ambiguity of the requests.
A mechanism that traces committed code back to the agent request that produced it.
Implementation
Results are subject to variance; small differences in scores may not be statistically meaningful.
CursorBench is an internal Anysphere suite based partly on a private codebase; it is not fully reproducible by external parties.
Evolution
Cursor's blog post "How we compare model quality in Cursor" (March 11, 2026, Naman Jain) describes CursorBench as an internal evaluation suite based on real sessions.
Update to the evaluation suite (May 2026).
Expanded task types: edit, refactor, investigation, intent understanding, managing jobs, and design adherence (September 10, 2026).