Researchers from Princeton, the University of Texas at Austin, the University of Toronto, NVIDIA and AGIBOT have introduced WorldOracle, a benchmark that scores action-conditioned world models by comparing their predictions with the real execution of the same motion sequence. The first results across seven models show a shared weakness: arm motion is tracked well, the position of the manipulated object much less so.
Key takeaways
- The benchmark compares a predicted rollout with a recording of the real execution from the same start frame and the same action sequence
- IRASim: 78.8 points for arm motion against 29.1 for the manipulated object. Cosmos 3: 89.4 against 45.4
- Over the full sequence DreamDojo falls from 50.56 to 27.08, and Cosmos 3 from 53.99 to 26.60
- GE-Sim 2.0 posts the highest score at both horizons and the smallest decline: 73.86 to 63.33
- The platform opened to developers on 16 October and accepts both open-weight and closed models
Picture this situation
A robot is to pick a piece of fruit off a counter. It receives an action sequence, executes it and misses — the gripper closes beside the fruit, which stays put.
The world model was given exactly the same initial frame and exactly the same action sequence. It generated a video in which the robot lifts the fruit and places it in a container. The video looks plausible. The success it depicts never happened.
But here is the problem: such a model, used for planning, will suggest a motion to the robot that does not work in reality.
That exact case is shown as an example in the WorldOracle material, and that exact error is what the benchmark is meant to catch.
What exactly does WorldOracle measure?
The team collected real manipulation episodes in advance, recording both video and the actions performed. A developer receives initial frames and action sequences from the evaluation set, generates a predicted video and submits it to the platform, which scores the submission against the recording of the actual execution.
The asymmetry against ordinary video-generation benchmarks is the key point. What is scored is not whether the footage looks good, but whether the predicted consequence of the actions matches what actually happened.
The score combines four metrics. Physical State Correctness checks whether object positions, grasp state and liquid transfer at the end of each interaction match the real outcome. Interaction Spatial Alignment scores the consistency of positions and movement paths of objects, arms and hands. Key Interaction Area Fidelity focuses on the regions around the object and gripper so a large background does not inflate the result. Multi-View Action Coherence checks whether motion is consistent across synchronised cameras.
The first three metrics are averaged with equal weights into the single-view WorldOracle-SV score. For multi-view evaluation the fourth joins them, also equally weighted. Sub-scores are retained, so a developer sees state error, trajectory error and local detail separately.
This is a simplified view of the scoring — the full specification of metrics and tracks sits in the paper published by the team.
Where do the models fail?
The first pattern concerns the split between arm and object. All five models evaluated on AGIBOT G2 teleoperation data scored higher on the arm than on the object. IRASim reached 78.8 for arm motion and only 29.1 for the manipulated object; Cosmos 3 scored 89.4 and 45.4 respectively. The remaining models showed gaps of varying size but the same direction.
The practical meaning is simple: a model can reproduce how the arm reaches the fruit and closes the gripper, and still get wrong whether the fruit was lifted and where it ended up.
The second pattern concerns time. WorldOracle scores the first 30 seconds of a trajectory separately from the complete long sequence. On the same AGIBOT G2 data the WorldOracle-SV scores fell for all five models — DreamDojo from 50.56 to 27.08, Cosmos 3 from 53.99 to 26.60, nearly halving in both cases.
GE-Sim 2.0 did best at both horizons and posted the smallest decline, from 73.86 to 63.33. The ordering also shifted: IRASim ranked below Ctrl-World, DreamDojo and Cosmos 3 over the first 30 seconds, and above all three over the full sequence.
Why not just judge a model by how the video looks?
Existing generative benchmarks mostly measure perceptual image quality and agreement with a prompt. A model that produces sharp, coherent video of a robot performing a task scores highly on them.
In robotics that criterion stops making sense. A world model exists to predict the consequences of actions before they are executed, so its value depends on agreement with the physics of a specific episode, not on aesthetics.
There is a second, harder problem. Generative models trained mostly on successful demonstrations tend to depict the version of the task that worked. WorldOracle deliberately includes failed executions — missed grasps, drops, spills — and awards a high score for correctly predicting a failure. Without that there is no way to separate a model that understands consequences from one that guesses success.
What do these names mean?
Four terms needed to read the results:
- ACWM (action-conditioned world model) — a world model that receives not only a starting image but a specific action sequence, and must predict what follows from it. See also World Models and Action-Conditioned Video Generation.
- Rollout — the course of events generated by the model from the start frame. The equivalent of a predicted simulation, only produced by a network rather than a physics engine.
- WorldOracle-SV — the single-view score, an equally weighted mean of three metrics. SV stands for single view.
- Horizon — the length of the scored run. The benchmark separates the first 30 seconds from the full, multi-stage task sequence.
Splitting arm from object and short horizon from long is precisely what exposed the gaps described above.
Is it working yet?
Yes, as an open evaluation platform. WorldOracle opened to developers on 16 October 2026 and accepts both open-weight and closed models. The team published a project page and a paper on arXiv.
The benchmark covers two robot embodiments — AGIBOT G2 and YAM — plus first-person human-hand manipulation. Four tasks map to four object classes: fruit picking (rigid body), towel hanging (deformable object), water pouring (fluid) and drawer pulling (articulated object). Task coverage varies by embodiment and is defined in the evaluation set.
The data come from two sources: human-operated collection (robot teleoperation and recordings of human hands) and policy rollouts, in which an autonomous model controls a real robot. The platform offers three tracks — WorldOracle-Core over the first 30 seconds of human-operated data, WorldOracle-Policy over complete episodes of autonomous rollouts on AGIBOT G2, and WorldOracle-Long over complete long human-operated sequences.
Why it matters
World models are currently offered as a cheaper substitute for testing on a physical robot. If their error grows exactly where contact with the object begins, and at horizons longer than half a minute, then they substitute for the hardest part of testing. A benchmark with failed executions in its set forces a conversation about that limit instead of papering over it with image quality.
What's next
- The platform has accepted submissions since 16 October 2026 — further leaderboard positions depend on how many teams submit rollouts across the three tracks
- The WorldOracle-Policy track, built on autonomous rollouts on AGIBOT G2, tests models against actions containing pauses and corrections absent from teleoperated data
- The paper and project page are public, so the split between arm and object scores can be reproduced independently
Sources
- The Robot Report — WorldOracle Tests Whether World Models Can Reliably Predict Real-World Executions
- WorldOracle — project page
- arXiv — the paper describing the WorldOracle benchmark
