RPE
How it works
1. Define the task suite: sets of tasks (e.g. 'put onion in bowl', 'stack blocks') with success criteria (final-state predicates). 2. Scene curation: 3D scenes with objects placed in simulation - manually via USD/URDF or via LLM (agentic scene generation in RoboLab). 3. Perturbation configuration: define the range of variations (lighting, camera pose, backgrounds, textures, shadows, sensor noise) for systematic robustness testing. 4. Rollout execution: the policy is invoked step by step over each episode - in simulation the environment state evolves under physics (PhysX, MuJoCo). Modern platforms run thousands of episodes in parallel on GPUs (Isaac Sim + Isaac Lab). 5. Metrics capture: success/failure, time-to-completion, path efficiency, jerk, collision count, trajectory smoothness. 6. Statistics and insights: aggregate metrics across tasks + variations; sensitivity analysis (NPE in RoboLab) to weigh each environment parameter. 7. Sim-to-real transfer analysis: compare simulation results with real-world runs to estimate the reality gap.
Problem solved
Evaluating robot policies directly on a physical robot is expensive (equipment wear), unsafe (collisions with environment/humans), slow (one robot - one session at a time), non-reproducible (each physical experiment has a unique initial state), and hard to scale (thousands of scenario variants). RPE in high-fidelity simulation solves all these problems simultaneously, enabling tens of thousands of evaluation episodes in a short time.
Components
Sets of parameterised tasks with clear success criteria (final-state predicates). Examples: LIBERO (130 tasks), RoboCasa, Meta-World (50 tasks), RoboLab (120 tasks).
Official
Simulation engine (Isaac Sim/PhysX, MuJoCo, Genesis, Drake) with physics parameters, rendering, cameras, actuators.
Official
Ranges of environment parameter variations: lighting, camera pose, backgrounds, textures, shadows, sensor noise, object mass/friction. Critical for domain randomization.
Official
A layer connecting the tested policy (VLA/diffusion/RL) with the simulation - provides observations in the expected format and executes actions. Framework-agnostic (LeRobot, HuggingFace).
Official
Per-rollout recording: success/failure, time, jerk, collision count, trajectory. Export to dashboards (Weights & Biases, MLflow) and leaderboards.
Official
Aggregation of metrics across tasks + variations, statistics (bootstrap CI), sensitivity analysis (NPE - Neural Posterior Estimation) to identify parameters with the greatest impact.
Official
Implementation
High success rate in simulation does NOT guarantee real-world success. Dynamics, lighting, textures, and sensor noise can differ significantly.
When policies are trained on the same task suite, evaluation stops discriminating quality - all policies hit 95%+ success.
Publishing results with 10 episodes per task - differences between policies are within noise.
Success/failure hides partial progress - a policy may be 'close' to success without a completed task.
Reporting only success rate without understanding why the policy failed (grasp failure? planning failure? sensor failure?).
Evolution
OpenAI Gym provides a standard interface for RL evaluation. MuJoCo (physics) becomes the standard for robotics simulation research.
50 manipulation tasks with a Sawyer robot - the first widely adopted benchmark for multi-task RL policy evaluation.
100 tasks with Franka Panda in PyRep/CoppeliaSim - extends Meta-World with more variations and realistic kinematics.
130 language-conditioned tasks for lifelong robot learning - the first popular benchmark for language-conditioned imitation learning.
SimplerEnv (Li et al. 2024) provides realistic simulation for VLA policies (RT-2, OpenVLA). RoboCasa (Nasiriany et al.) - 100 kitchen tasks with LLM-generated variations.
NVIDIA publishes Isaac Lab-Arena (github.com/isaac-sim/IsaacLab-Arena) - a large-scale GPU-accelerated framework with Hugging Face LeRobot Hub integration. GR00T N evaluation time reduced from days to <1 hour.
NVIDIA SRL presents RoboLab at RSS 2026 - robot- and policy-agnostic benchmark with 120 tasks, NPE sensitivity analysis, agentic scene/task generation (code agents: /robolab-scenegen, /robolab-taskgen).
Hyperparameters (configurable axes)
How many times each task is executed (for statistical significance).
Number of perturbation dimensions: lighting, camera pose, textures, backgrounds, sensor noise, etc.
Final-state predicate defining when a task is considered complete.
Level of natural-language instruction detail. Critical for language-conditioned policies.
Computational complexity
Time complexity: O(N_tasks * N_variations * N_episodes * T_rollout). Space complexity: O(N_episodes * (obs_size + action_size + metadata)).
Execution paradigm
Evaluation is fully deterministic per seed (absent RL agent stochasticity). Each episode runs the full policy forward pass for every step.
Parallelism
Episodes are independent and ideal for embarrassingly parallel execution on multi-GPU clusters. Isaac Sim can render 4096 environments in parallel on a single H100 GPU.
Hardware requirements
GPU (NVIDIA RTX/A100/H100) mandatory for scene rendering + policy inference + physics simulation. Isaac Sim requires CUDA.
MuJoCo/Drake can run on CPU for small simulations, but without the scalability of modern platforms.