Aether AI, with UC San Diego and UC Davis, released RobotAgentBench on 26 October 2026 — a benchmark testing whether a household robot can work out what needs doing and then do it. The result is unambiguous: the best standalone control model averaged 18.0 points out of 100, and in open-ended scenarios it barely moved at all.
Key takeaways
- 50 apartment layouts, 90 atomic tasks, 18 short-horizon, 8 long-horizon and 4 open-ended scenarios
- Best standalone model (InternVLA-A1.5) — 18.0 points average, best agent architecture — 48.5 points
- In open-ended scenarios standalone models scored 0.3 to 2.0 points, the best agent 15.4
- Task discovery without an instruction: standalone models below 2 points, coding agents up to 37.2
- 101.8 hours of training data collected on the AgiBot G1 humanoid
Picture the situation
A robot stands in the living room and, instead of a task list, gets one sentence:
Guests are arriving in an hour.
Nobody told it that a hat was left on the sofa or that the table is cluttered. The robot has to walk the apartment, set an order — food leaves the fridge before it goes into the microwave — and only then grasp anything. This is an editorial simplification, though hosting guests is one of the benchmark's four scenarios.
But a problem appears: today's robot control models are trained to execute instructions, not to recognise situations.
That is precisely the gap RobotAgentBench sets out to measure.
What exactly does this benchmark measure?
Robot tests so far have examined isolated skills — mostly whether a robot performs a named action. RobotAgentBench splits autonomy into five separately scored dimensions: task discovery, context reasoning, manipulation, failure recovery and robustness.
Each has its own metric. Task discovery counts the fraction of correct objects touched while exploring without an instruction. Context reasoning uses four settings — drink, snack, waste and sofa-object selection — where the robot gets user preferences or a classification rule, not an object name. Robustness uses mid-task disturbances: in one of six variants the coffee machine is moved while the robot pours beans into it.
A separate layer holds four open-ended scenarios: preparing dinner, hosting guests, preparing to leave and tidying the room. No task list — the robot works out what needs doing, and in what order.
Where did the 50 apartments come from?
The starting point was one physical room in a real home, rebuilt in NVIDIA Isaac Sim. A coding agent retrieved 3D models and matched them to photographs and measurements, with human reviewers validating fidelity throughout. That reconstruction produced 50 layouts — floor plans reshaped into T and L configurations, furniture rearranged, textures and styles varied.
Data came from teleoperation of the AgiBot G1 humanoid, a platform from China's AGIBOT, replayed in simulation with physics tuned to reality, after which a coding agent multiplied the trajectories?trajectories: recorded sequences of robot motion carrying one task through from start to finish across other object positions. In total 101.8 hours: 50 trajectories per atomic task?atomic task: a single, indivisible robot action, for example grasping one object and short-horizon task, 10 per long-horizon task and open-ended scenario.
What does a robot agent's loop look like?
The paper introduces a shared planning–execution–verification skeleton, written P/E/V. C denotes code-based control, L a learned VLA or WAM policy, H a hybrid, a dash an absent component. Five combinations were tested under identical conditions.
This is a simplified view of one of the five architectures studied, not a full system specification.
What do these names mean?
Four abbreviations recur in the paper, and the results are unreadable without them.
- VLA — turns camera images and a text instruction straight into robot motion. A controller trained only on recordings of correct executions.
- WAM — a world model coupled to action. Rather than mapping an image to motion, it predicts how the scene changes after a move, then plans.
- Coding agent — a programming assistant as the robot's brain. It composes programs from control functions such as
move_eef_to(). - RSI — recursive self-improvement. After each attempt the agent refines its control functions, keeping what worked.
These four are not competitors — the benchmark shows they work best combined.
What did the results show?
The gap between manipulation and autonomy is clearest in one figure: the best standalone model, InternVLA-A1.5, averaged 18.0 points, while the hybrid C/H/C architecture — the same model invoked by a coding agent — reached 48.5. Eight VLA models were evaluated, among them π0.5 from Physical Intelligence, Xiaomi Robotics-1 and X-VLA, plus three WAM models.
Task discovery diverges most sharply: every standalone model stayed below 2 points, while the best coding agent reached 37.2. Without an explicit instruction, VLA models produce erratic movements instead of looking around.
Among fully code-based architectures the strongest was Antigravity CLI powered by Gemini 4 Argon at 39.7 points, ahead of Claude Code on Opus 5.5 with 33.3 and Codex on GPT-6 Astra with 31.0. Instruction style points to why: Antigravity CLI writes shorter commands and omits execution detail, landing closer to the language the executor saw in training.
Ten rounds of self-improvement lifted Codex from 31.0 to 42.3 points, as the agents learned to break joint-limit-violating motions into feasible intermediate poses.
Open-ended scenarios remain unsolved for everyone — best score 15.4 out of 100.
Why isn't a stronger VLA model enough?
An obvious question follows: if VLA models keep growing, won't the next generation solve this on its own?
The data suggests not, because the bottleneck sits elsewhere. VLA models train on demonstrations of correct executions, so they learn to reproduce motion rather than decide what is worth doing. On context reasoning their accuracy was close to random.
A second problem scale does not remove is getting stuck. The stagnation metric — the share of time spent on repetitive, unproductive attempts — hit 88.3% for InternVLA-A1.5 and 84.6% for OpenWAM-Alpha. The best agent setup cut it to 12.1%, because a verifier notices something is failing.
Coding agents alone are not enough either. The purely code-based setup loses to the hybrid — code brings the arm near the object, but a learned model closes the grasp.
Is it working yet?
This is a research benchmark, not a product. The paper landed on arXiv on 26 October 2026 under CC BY 4.0?CC BY 4.0: a Creative Commons licence allowing any reuse of the material provided the authors are credited, with code on GitHub and the fine-tuned InternVLA-A1.5-RAB model on Hugging Face.
No tested system is ready for a real apartment. A simulation-to-reality comparison on a matched subset showed a drop across all three.
| System | Simulation | Reality |
|---|---|---|
| InternVLA-A1.5 | 17.8 pts | 13.5 pts |
| OpenWAM-Alpha | 8.5 pts | 8.2 pts |
| Codex | 34.2 pts | 31.9 pts |
Why does this matter?
The industry measures robotics progress mainly by grasp quality. This benchmark separates two things previously conflated: motion proficiency and the ability to set a goal independently. The split shows the second does not grow with the first, and that the fastest improvement comes not from a larger model but from pairing it with a planning and verification layer.
What next?
- The benchmark code and fine-tuned model are public on GitHub and Hugging Face, so further agent architectures can be compared across the same 50 layouts
- The 15.4 out of 100 in open-ended scenarios is a reference point for follow-up work — the state of the art, not a saturated result
- The simulation-to-reality gap stays open: all three systems checked on the physical AgiBot G1 did worse than in simulation
Sources
- arXiv — RobotAgentBench: Towards Open-world Robot Autonomy
- Aether Labs AI — RobotAgentBench project page
- GitHub — AetherLabsAI/RobotAgentBench
- Hugging Face — InternVLA-A1.5-RAB
