1) Real vulnerabilities are collected from popular open-source projects (code, description, patch history). 2) An AI agent (with tools: repository access, code execution, fuzzing, etc.) is given a task - e.g. reproduce a vulnerability, generate a PoC or write a patch. 3) The result is automatically verified against real behavior (e.g. whether the exploit triggers the vulnerability, whether the patch removes it without regressions). 4) Metrics (discovery/PoC/patch success) are used to compare agents and models and to track progress over time.
Prior cybersecurity benchmarks were small, synthetic or easy to game. CyberGym measures AI agents' real capabilities on genuine OSS vulnerabilities, giving a credible picture of their offensive and defensive potential and risk.
1,507 real vulnerabilities from 188 OSS projects (E2E: 920/139) with code and patch history.
Tools for the agent: repository, code execution, fuzzing, analysis.
Official
Checks the result against reality (the exploit triggers the flaw, the patch removes it).
Official
Discovery/PoC/patch success rates to compare agents and models.
Official
Evaluation produces working exploits - it requires strict isolation to avoid harm.
Agents able to find zero-days can be misused offensively.
Public OSS vulnerabilities may enter model training data, inflating scores.
RDI (UC Berkeley) releases CyberGym - 1,507 real vulnerabilities from 188 OSS projects (arXiv:2506.02548).
ICLR 2026 presentation and an E2E variant (920 vulnerabilities/139 projects); discovery of 35 zero-days and 17 incomplete patches.
Time complexity: Nie dotyczy (benchmark ewaluacyjny, nie algorytm). Space complexity: Nie dotyczy.
Evaluation requires running code/exploits in isolated (sandboxed) environments for 1,500+ tasks; the bottleneck is safe execution and verification, not the model itself.
Vulnerability discovery/reproduction, PoC generation or patch development.
Full (1,507/188) vs CyberGym-E2E (920/139).
Tools available to the agent (repo, code execution, fuzzing).
This describes the agent evaluation run; the benchmark itself is not a model compute paradigm.
Tasks can be run in parallel across many sandboxes.
An evaluation benchmark/protocol - independent of accelerator type; task execution runs on CPU (sandbox), while the evaluated agent may use GPUs.
