Simulation ยท Simulation & Digital Twins
1.0 (RSS 2026)
NVIDIA RoboLab (full name: RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies) is an open-source NVIDIA benchmarking platform for evaluating generalist robot policies in high-fidelity simulation. Created by the NVIDIA Seattle Robotics Lab in collaboration with the University of Toronto and The University of Sydney. Presented at RSS 2026 (Robotics: Science and Systems, Sydney). Code: github.com/NVLabs/RoboLab (Apache 2.0, Python). Authors: Xuning Yang, Rishit Dagli, Alex Zook, Hugo Hadfield, Ankit Goyal, Stan Birchfield, Fabio Ramos, Jonathan Tremblay.
Key innovation: RoboLab is robot- and policy-agnostic - the platform evaluates policies trained on real-world data directly in simulation, WITHOUT co-training on simulation data, in order to measure task generalization. Task libraries are independent of the robot and policy: users combine their own robot + their own policy with a ready-made task library to create ready-to-run environments. The same task set works with different robots and policies. Designed for fast scene and task generation with evolving task libraries that prevent benchmark oversaturation.
The RoboLab framework works in three steps: (1) Scene Generation - physically arranging objects in simulation (by hand in minutes or at scale via LLMs); (2) Task Generation - adding natural-language instructions to scenes (manually or via LLMs), with three specificity variants: default, vague, specific; (3) Environment Generation - specifying the robot, policy, action/observation configs, and scene variations that turn tasks into runnable environments. Scenes are saved as USD (.usda) files - Pixar's Universal Scene Description. A built-in predicate solver ensures collision-free placement, and a physics settle simulates how objects come to rest.
Agentic workflows: RoboLab supports code agents for agentic scene and task generation directly from the terminal. The commands /robolab-scenegen and /robolab-taskgen let you describe a need in natural language, and the agent handles asset placement (312 objects: containers, kitchenware, fruits, tools, blocks), constraint satisfaction, and instruction authoring. The RoboLab Benchmark evaluates policies across three competency axes: Visual (recognition of color, semantics, size), Relational (temporal, numerical, spatial inter-object relationships), and Procedural (action-oriented reasoning: affordances, reorientation). Benchmark: 120 tasks, on average 2.02 subtasks and 9.0 objects per task, average difficulty score 2.90.
Evaluation methodology: RoboLab uses Neural Posterior Estimation (NPE) to quantify which environment parameters most influence policy success - for an observation x (task outcome) and environment parameters theta it estimates the posterior p(theta|x). The analysis showed high sensitivity to the wrist camera. The platform supports controlled perturbations: lighting, camera pose, backgrounds, textures, shadows - for systematically testing policy robustness. Policies tested include pi0.5 (Physical Intelligence). Comparison with the DROID dataset shows RoboLab emphasises multi-step tasks (only 68.7% of benchmark objects appear in DROID's training vocabulary). RoboLab typically runs on a workstation with an NVIDIA GPU (CUDA) and builds on NVIDIA Isaac Sim.
Simulation software is used for modelling, testing, and validating robot behaviours, sensor characteristics, environments, and algorithms without requiring physical hardware. It enables safe, repeatable, and cost-effective development cycles. Common robotics simulators include Gazebo, Isaac Sim (NVIDIA), MuJoCo, PyBullet, and Webots, each offering different trade-offs between physics accuracy, rendering fidelity, and integration with middleware frameworks such as ROS 2.
A Developer Tool is software designed to support the development workflow, including configuration, debugging, testing, monitoring, validation, and integration of robotic and embedded systems. Examples include IDE plugins, visual debuggers, log analysers, hardware-in-the-loop (HIL) test harnesses, and code-generation utilities specific to robotics platforms.
The benchmarking and evaluation role describes software responsible for standardized measurement of robot and AI model capabilities. The benchmark component includes: a set of defined tasks (manipulation, locomotion, perception, instruction), automated success and execution quality metrics, test scenarios covering generalization dimensions (lighting, background, camera noise, control delay), reproducible processes for running multiple trials with result aggregation, model leaderboards (e.g. ฯ0.5, GR00T, ฯ0). Modern systems use VLM (Vision-Language Models) for auto-evaluation of complex qualitative criteria inaccessible to simple numerical metrics.
The simulation role describes software that reproduces the physical behavior of a robot and its environment in a computer environment. The simulation component implements: deterministic integration of rigid-body and soft-body dynamics (typically via PhysX, Bullet, MuJoCo, Newton engines), photo-realistic scene rendering (RTX, ray-tracing), sensor models (RGB/D cameras, LiDAR, IMU, F/T), synthetic data generation for training machine learning policies, comparative evaluation of multiple solution variants under identical conditions. Simulation enables massively parallel RL training without physical hardware and is the foundation of the sim-to-real workflow.
The robot learning role describes software for training a robot's control policies and manipulation/locomotion skills using machine learning methods. It covers: reinforcement learning in simulation with sim-to-real transfer, imitation learning and learning from demonstration, training Vision-Language-Action (VLA) models, and fine-tuning robotics foundation models. It typically uses massively parallelized simulation environments (Isaac Lab, MuJoCo) to generate training data, then deploys the trained policies on a physical robot.
The synthetic data generation role describes software for automated production of large training datasets for robot perception and control models. The component implements: programmed simulation scenarios with parameterized objects and motions, automatic ground-truth labeling (segmentation, bounding boxes, 6-DoF poses, depth maps), demonstration trajectory collection via in-simulation teleoperation or expert automated policies, domain augmentation (lighting, textures, materials) for domain randomization, error-recovery mechanisms for producing correct trials despite failures. Generates data scale unattainable in the physical world โ typically 10,000+ hours of trajectories.
Used to evaluate task-generalist policies (including pi0.5 by Physical Intelligence). Presented at RSS 2026 (Sydney).
120 benchmark tasks, 312 asset objects, 3 competency axes (Visual/Relational/Procedural), 3 difficulty levels.
Ubuntu 24.04 LTS 'Noble Numbat' โ supported until April 2029. The host for ROS 2 Jazzy.
Requires NVIDIA Isaac Sim as the simulation foundation. A CUDA-capable GPU is mandatory for high-fidelity rendering and physics.
License family: Permissive
First public release with code (github.com/NVLabs/RoboLab), paper (arXiv 2604.09860), and leaderboard. Presented at RSS 2026.
