Evaluation

CyberGym

ActivePublished: 6 August 2026Updated: 6 August 2026Published

Key innovation

A large-scale benchmark evaluating AI agents on real-world cybersecurity tasks (discovering and reproducing vulnerabilities, generating PoC exploits and developing patches) built on real vulnerabilities from open-source projects - not synthetic tasks.

How it works

1) Real vulnerabilities are collected from popular open-source projects (code, description, patch history). 2) An AI agent (with tools: repository access, code execution, fuzzing, etc.) is given a task - e.g. reproduce a vulnerability, generate a PoC or write a patch. 3) The result is automatically verified against real behavior (e.g. whether the exploit triggers the vulnerability, whether the patch removes it without regressions). 4) Metrics (discovery/PoC/patch success) are used to compare agents and models and to track progress over time.

Problem solved

Prior cybersecurity benchmarks were small, synthetic or easy to game. CyberGym measures AI agents' real capabilities on genuine OSS vulnerabilities, giving a credible picture of their offensive and defensive potential and risk.

Components

Real vulnerability datasetTest tasks

1,507 real vulnerabilities from 188 OSS projects (E2E: 920/139) with code and patch history.

Agent environment (harness)Task execution

Tools for the agent: repository, code execution, fuzzing, analysis.

Official

Automated verificationScoring

Checks the result against reality (the exploit triggers the flaw, the patch removes it).

Official

Metrics and leaderboardComparison

Discovery/PoC/patch success rates to compare agents and models.

Official

Implementation

Reference implementations

CyberGym (RDI Berkeley)

UC Berkeley RDI

Official

Implementation pitfalls

Safely running exploitsHigh

Evaluation produces working exploits - it requires strict isolation to avoid harm.

Fix:Sandboxing, ephemeral environments, network control.

Dual-use riskHigh

Agents able to find zero-days can be misused offensively.

Fix:Responsible disclosure, access controls, governance frameworks.

Leakage into training dataMedium

Public OSS vulnerabilities may enter model training data, inflating scores.

Fix:Date controls, held-out variants, verification by real behavior.

Evolution

2025

CyberGym released (RDI, Berkeley)

Inflection point

RDI (UC Berkeley) releases CyberGym - 1,507 real vulnerabilities from 188 OSS projects (arXiv:2506.02548).

2026

CyberGym-E2E and ICLR presentation

ICLR 2026 presentation and an E2E variant (920 vulnerabilities/139 projects); discovery of 35 zero-days and 17 incomplete patches.

Hyperparameters (configurable axes)

Task typeCritical

Vulnerability discovery/reproduction, PoC generation or patch development.

Benchmark variantHigh

Full (1,507/188) vs CyberGym-E2E (920/139).

Agent toolingHigh

Tools available to the agent (repo, code execution, fuzzing).

Computational complexity

Time complexity: Nie dotyczy (benchmark ewaluacyjny, nie algorytm). Space complexity: Nie dotyczy.

Compute bottleneck

Running and verifying tasks in isolation

Evaluation requires running code/exploits in isolated (sandboxed) environments for 1,500+ tasks; the bottleneck is safe execution and verification, not the model itself.

Depends on

Izolacja/sandboxLiczba zadan i prob