Evaluation
SWE-bench Verified
2024ActiveUpdated: 17 September 2026Published
Key
innovation
A human-validated 500-task subset of SWE-bench, in which engineers confirmed the problems are solvable — a more reliable measure of agents' ability to fix real code.
Category
Evaluation
Abstraction level
System
Operation level
Evaluation (runtime)
Use cases
Evaluating coding agents and assistantsComparing models on real code fixesVerifying progress in agent self-improvement loops
How it works
The agent is given the issue text and a repository snapshot, then proposes a code change (patch). The change is applied and the repository's tests are run (including the bug-reproducing tests and regression tests). A task counts as resolved when all required tests pass; the headline metric is the percentage of resolved issues (pass@1) across the 500 tasks.
Problem solved
The full SWE-bench contains unsolvable or poorly specified tasks that depress and add noise to scores; Verified provides a reliable, human-confirmed subset for comparing coding agents.
Evolution
Original paper · 2023 · ICLR 2024 · Carlos E. Jimenez
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang
2023
SWE-bench (Princeton/Stanford)
The original benchmark for resolving GitHub issues (arXiv 2310.06770, ICLR 2024).
2024
SWE-bench Verified (500 tasks)
Inflection pointA human-validated 500-task subset; a collaboration with OpenAI Preparedness (August 13, 2024).