LIBERO
How it works
LIBERO specifies manipulation tasks through a procedural generator built on robosuite: the scene, objects and goal conditions are described in PDDL (BDDL files), which in principle allows generating arbitrarily many tasks. Four ready-made suites isolate different distribution shifts: LIBERO-Spatial changes spatial layout, LIBERO-Object changes the objects, LIBERO-Goal changes the task goal, and LIBERO-100 mixes multiple knowledge types. An agent (a behavior-cloning visuomotor policy) is trained sequentially on successive tasks with 50 demonstrations per task, then evaluated for forward transfer, forgetting and task success. The benchmark ships several encoder/policy architectures and anti-catastrophic-forgetting algorithms (Experience Replay, EWC, PackNet) compared against sequential finetuning and a multitask baseline. The authors' experiments found, among other things, that sequential finetuning can outperform dedicated lifelong methods in forward transfer, no single visual encoder excels at all transfer types, and naive supervised pretraining can hurt downstream performance.
Problem solved
Before LIBERO there was no standardized benchmark for studying knowledge transfer in lifelong robot manipulation that separated the effect of declarative knowledge (space, objects) from procedural knowledge (goals, behaviors). LIBERO enables controlled comparison of policies and lifelong learning algorithms on a shared task set, treating robustness to task ordering and the effect of pretraining as explicit research axes.
Components
LIBERO-Spatial, LIBERO-Object and LIBERO-Goal with 10 tasks each (each isolating one type of distribution shift), plus LIBERO-100 with 100 tasks requiring entangled knowledge transfer, split into LIBERO-90 (pretraining) and LIBERO-10/LIBERO-Long (evaluation).
50 high-quality human-teleoperated trajectories per task, enabling sample-efficient behavior-cloning training.
An extendable robosuite-based pipeline generating scenes, objects and goal conditions from PDDL descriptions (stored as BDDL files); in principle it can generate infinitely many tasks.
The robosuite simulation environment with the MuJoCo physics engine, in which manipulation tasks are executed and visual observations rendered.
Reference visuomotor policies: bc_rnn_policy (ResNet + RNN), bc_transformer_policy (ResNet + Transformer) and bc_vilt_policy (ViLT vision-language transformer).
A set of anti-catastrophic-forgetting algorithms: sequential finetuning (base), Experience Replay (ER), Elastic Weight Consolidation (EWC), PackNet and a multitask learning baseline.
Implementation
The authors' experiments showed that naive supervised pretraining can lower an agent's subsequent lifelong-learning (LLDM) performance instead of improving it.
No single visual encoder architecture excels at all types of knowledge transfer, complicating the choice of one default policy.
Lifelong-learning results depend on the order in which tasks are presented, which can confound comparisons between methods.
Evolution
Bo Liu et al. publish LIBERO at the NeurIPS 2023 Datasets and Benchmarks Track together with code (MIT) and demonstration data (CC BY 4.0).
LIBERO becomes one of the standard evaluation suites for Vision-Language-Action models and generalist manipulation policies (e.g. OpenVLA).
Hyperparameters (configurable axes)
Which LIBERO suite is used, determining the type of knowledge transfer studied.
The forgetting-mitigation method used during sequential training.
Choice of visual encoder and sequence head for the visuomotor policy.
Number of demonstration trajectories used for training per task.
The order in which tasks are presented in sequential training — affects transfer and forgetting.
Hardware requirements
Training and evaluating visuomotor policies and rendering observations from MuJoCo/robosuite benefit from GPU acceleration (including EGL GPU rendering).