Evaluation

ARVO

2024ActivePublished: 2 September 2026Updated: 2 September 2026Published

Key innovation

Turns ephemeral OSS-Fuzz bug reports into a stable, reproducible benchmark of 6,100+ real-world vulnerabilities packaged as Docker environments, enabling consistent evaluation of vulnerability detection and repair tools, including AI systems.

How it works

ARVO uses OSS-Fuzz metadata to reconstruct a complete, self-contained environment for each vulnerability. Its pipeline reproduces the build configuration and compiles the vulnerable revision, runs the supplied proof-of-concept to confirm the bug triggers, then identifies the patching commit and verifies the patch removes the crash. Each case is packaged as a Docker image containing both vulnerable and patched builds. In evaluation, ARVO reproduces about 81% of vulnerabilities and locates the patching commit with high accuracy (around 89%).

Problem solved

Fuzzing-derived vulnerability reports (e.g. from OSS-Fuzz) are hard to reproduce reliably: build environments, dependencies, and code revisions decay over time, leaving no large, stable benchmark for comparable evaluation of vulnerability-detection and repair tools or AI models.