Robots Atlas>ROBOTS ATLAS
Evaluation

Differential Oracle

2026ResearchPublished: 2 September 2026Updated: 2 September 2026Published
Key innovation
Reframes exploit/PoC validity as a machine-checkable, two-sided differential predicate across pre- and post-patch builds, turning exploit generation into a propose–verify–refine loop.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)Agent runtime
Use cases
Automated proof-of-concept (PoC) exploit generationVerifying security patch effectivenessTraining and evaluating cybersecurity agentsSoftware differential testing

How it works

For a vulnerability instance, two builds of the same software are prepared: vulnerable (b⁻) and patched (b⁺). A candidate input x is executed on both builds under sanitizer instrumentation. The oracle accepts x as a valid PoC only if b⁻ triggers the target sanitizer failure and b⁺ does not. Because the predicate is programmatic and decidable, an agent can test each candidate without supervision; the verdict together with the sanitizer trace provides grounded feedback, turning PoC construction into an executable propose–verify–refine loop.

Problem solved

How to objectively and without human supervision decide whether a model-generated exploit/PoC genuinely triggers a given vulnerability rather than crashing for an incidental reason.

Components

Pre-patch build (b⁻)Vulnerable version

The vulnerable version of the software on which a valid PoC must trigger a crash.

Post-patch build (b⁺)Fixed version

The patched version on which a valid PoC must NOT trigger a crash.

Crash predicate / sanitizer instrumentationRuntime signal

Crash(b,x) is true iff executing b on x triggers the target sanitizer failure; provides the signal for memory errors.

Two-sided verdict š’±(x)Oracle decision

š’±(x)=šŸ™[Crash(b⁻,x)]āˆ§šŸ™[¬Crash(b⁺,x)] — a machine-checkable success signal.

Evolution

Original paper Ā· 2026 Ā· Jian Yang
CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild
Jian Yang, Haau-Sing Li, Shawn Guo, Zixi Zhao, Yibo Tan, Jiajun Wu, Aishan Liu, Xianglong Liu, Tianyu Zheng, Bryan Dai, Chengran Yang
1998
Differential testing
Inflection point

The principle of comparing program variants' behavior on identical inputs as an oracle.

2026
Differential oracle in CyberFactory
Inflection point

A two-sided, machine-checkable differential oracle (pre/post-patch builds) for autonomous PoC generation by AI agents.

Hyperparameters (configurable axes)

Target sanitizer / failure signalHigh

The sanitizer instrumentation type defining the Crash predicate (e.g. memory-error detection).