Beihang University, ELLIS, Singapore Management University and IQuest Research released CyberFactory, an open pipeline turning public vulnerability reports into runnable, automatically checkable agent tasks. That data trained OpenAegis, which solves 58.1% of CyberGym tasks under a one-hour budget. The paper landed on arXiv on 24 August 2026.
Key takeaways
- CyberGym Pass@1 under a one-hour budget: OpenAegis 58.1% versus 29.6% for Qwen 3.5
- The margin is 6.4 points over Kimi K2.7 and 14.8 over GLM 5.2
- OpenAegis runs 397B parameters with 17B active — fewer than either rival
- Tasks come from three sources: ARVO, OSS-Fuzz?OSS-Fuzz: A free continuous-fuzzing service for open-source projects, run by Google with OpenSSF. Fuzzing floods a program with generated inputs until one triggers a crash that exposes a bug. and public CVE reports
- The GitHub repository named in the paper was empty on 1 September 2026
From a CVE entry to a task you can verify
Plain CVE reports are hardest. They carry a description, a vulnerability class and a fix commit, but no runnable environment. CyberFactory rebuilds the pre-patch and post-patch versions and adds a differential oracle: a candidate input runs twice, and the task counts only if it crashes the vulnerable build and leaves the patched one intact.
- CVE entry — a description, a vulnerability class and a fix commit, with no environment anything can run in
- Rebuild — the pre-patch and the post-patch version of the code
- Build — both images, separately
- Differential oracle — the candidate input runs twice, once against each image
- Pass — only if the input crashes the vulnerable build and leaves the patched one intact
The CyberFactory pipeline turns a static CVE entry into a task a machine can grade without a human. The weight sits on the last step: if the vulnerable and patched builds behave the same, the task fails validation.
The skill goes to the teacher, not the student
The second piece is a reusable vulnerability-analysis skill — a procedure, not a ready answer. The authors apply it only when the teacher generates trajectories: GLM 5.2 without the skill hits 43.3% Pass@1?Pass@1: The share of tasks solved with a single attempt per task. A strict measure — the model gets no second try. in one 60-minute run, and 46.5% with it across five 15-minute runs. OpenAegis never sees that text at inference — the procedure stays in the weights after fine-tuning.
| Model | Pass@1 (1 h budget) |
|---|---|
| OpenAegis | 58.1% |
| Kimi K2.7 | 51.7% |
| GLM 5.2 | 43.3% |
| Qwen 3.5 (base) | 29.6% |
Context compaction?Context compaction: Summarising an agent's work history once the context window fills up, so the remaining steps still fit inside the limit. decides long tasks
Reproducing a vulnerability often takes dozens of tool calls, so context runs out before the task. At 90% usage the agent compresses evidence, failed attempts and build state into a description it can work on. Against full history that yields 58.1% instead of 52.1% overall and 48.7% instead of 40.2% above 40 interactions.
Why it matters
Open models lost in cybersecurity not for missing knowledge but for missing data showing the work. Open weights without an open training process were never enough. CyberFactory shows procedural discipline can be synthesised and moved into a smaller model's weights rather than bolted on by prompt. The conclusion cuts both ways: the skill that shortens a code audit also lowers the barrier to automated flaw hunting in other people's software.
What next?
- The CyberFactory collection on Hugging Face holds only the paper entry, no weights
- Full numbers exist only for vulnerability reproduction, with patching and CyberQA awaiting separate evaluation
- An arXiv revision on 25 August updated scores after training on new agentic data
