The exploitation process is decomposed into 16 measurable capabilities arranged as a ladder of increasing difficulty: from code coverage and triggering a crash, through sandbox primitives, arbitrary memory read/write and control-flow hijack, up to arbitrary code execution. Each task is built on a real V8 engine bug. Reaching a given step is confirmed by a deterministic oracle, and result reliability is strengthened by randomized challenge variants and differential execution analysis that limit false positives. A model or agent is characterized by the highest ladder step it reliably reaches.
Existing LLM security benchmarks typically equate a crash with successful exploitation, collapsing the hardest parts of an attack: the transition from triggering a bug to building reusable primitives and actually taking control of the target. ExploitBench addresses this with a graded, multi-step measure of real exploitation capability.
A 16-step scale from code coverage and crash, through sandbox primitives, arbitrary read/write and control-flow hijack, up to arbitrary code execution.
41 real V8 JavaScript engine vulnerabilities used as exploitation tasks.
Randomized challenge variants and differential execution analysis that confirm a step was reached and limit false positives.
Benchmark published on arXiv (2605.14153); evaluation of 8 public frontier models and 1 private model on 41 V8 engine bugs.
Number of measurable exploitation capabilities composing the ladder.
Number of bugs used as exploitation tasks.
Models and agents evaluated on the benchmark.
