1) Data is collected from many sources: teleoperation (an operator drives the robot), human/egocentric video, simulation/synthetic data and a robot's real interaction. 2) The data is curated, synchronized (observations-actions-proprioception) and (optionally) annotated. 3) Sources and embodiments are blended (cross-embodiment), and real data is mixed with simulation. 4) The dataset fuels model training (imitation/RL/pretraining), and the deployed robot generates new data, closing the data flywheel.
Robots need data about acting in the physical world which โ unlike internet text/images โ is scarce and expensive. Embodied AI data organizes the collection, curation and blending of such data sources.
Teleoperation, human/egocentric video, simulation/synthetic data and a robot's real interaction.
Official
Synchronized triples: observation (vision/sensors), action (motion command) and state (proprioception).
Official
Curation, synchronization, annotation and blending of sources and embodiments (cross-embodiment), mixing sim + real.
Official
Embodied data is hard to collect at scale โ teleoperation is slow and expensive.
Data from one embodiment/simulation transfers poorly to other robots and reality.
Inconsistent, biased or unsafe demonstrations degrade learned policies.
Large-scale human video becomes a data source for embodied AI.
Pooling data across many robots (cross-embodiment) demonstrates transfer and scaling of robot learning.
Mixing real data with simulation and data flywheels become standard for robot foundation models.
Time complexity: Zalezna od skali i modalnosci danych. Space complexity: O(epizody ร modalnosci ร dlugosc).
Embodied data is scarce and costly, strongly tied to embodiment, and simulation data does not always transfer to reality (sim-to-real).
Proportions of teleoperation, human video, simulation and real interaction.
Single- vs multi-embodiment (cross-embodiment) data.
From raw trajectories to rich labels (language, actions, success).
Concerns the data/training phase.
A data strategy, no routing.
Training and simulation data generation are highly parallel; real-robot collection is sequential.
Simulation data generation and training on multimodal data are compute-intensive (GPU/TPU); large storage and I/O bandwidth are also needed.
