Evaluation

ERQA

2025ActivePublished: 16 October 2026Updated: 16 October 2026Published

Key innovation

The first open benchmark that isolates the embodied spatial-physical reasoning of multimodal models from robot control — pure visual question answering instead of hardware trials.

How it works

ERQA consists of 400 multiple-choice questions combining interleaved images and text. Each question has a single correct answer labelled with a letter (A, B, C, D). A model is given the image(s) and the question and selects an answer; the score is the percentage of correct answers. Questions cover spatial reasoning and world knowledge in real-world scenarios, particularly in a robotics context. The data is distributed as TFRecord examples (features: question, image/encoded, answer, question_type, visual_indices) together with a lightweight evaluation harness for multimodal APIs.

Problem solved

General multimodal benchmarks poorly capture the physical, spatial-world reasoning that robotics requires. ERQA isolates and measures this embodied reasoning independently of controlling a physical robot.