Robots Atlas>ROBOTS ATLAS
Data

Embodied AI data

ActivePublished: 27 July 2026Updated: 27 July 2026Published
Key innovation
Treats data of acting in the physical world (observations + actions + proprioception) as the key, scarce resource for training robots, requiring dedicated collection, curation and blending of sources (teleoperation, video, simulation).
Category
Data
Abstraction level
Pattern
Operation level
DataTraining
Use cases
Training robot foundation models (VLA, video-action)Learning manipulation and locomotion from dataBlending data from many robots (cross-embodiment)Mixing real data with simulation (sim + real)Building a production data flywheelScaling robot learning

How it works

1) Data is collected from many sources: teleoperation (an operator drives the robot), human/egocentric video, simulation/synthetic data and a robot's real interaction. 2) The data is curated, synchronized (observations-actions-proprioception) and (optionally) annotated. 3) Sources and embodiments are blended (cross-embodiment), and real data is mixed with simulation. 4) The dataset fuels model training (imitation/RL/pretraining), and the deployed robot generates new data, closing the data flywheel.

Problem solved

Robots need data about acting in the physical world which — unlike internet text/images — is scarce and expensive. Embodied AI data organizes the collection, curation and blending of such data sources.

Components

Data sourcesAcquisition

Teleoperation, human/egocentric video, simulation/synthetic data and a robot's real interaction.

TeleoperationAn operator drives the robot to collect demonstrations.
Simulation / synthetic dataCheap, scalable simulator data.
Human/egocentric videoScalable demonstrations from a human viewpoint.

Official

Observation-action-state structureData format

Synchronized triples: observation (vision/sensors), action (motion command) and state (proprioception).

Official

Curation and blending pipelineData processing

Curation, synchronization, annotation and blending of sources and embodiments (cross-embodiment), mixing sim + real.

Official

Implementation

Reference implementations
Open X-Embodiment / RT-X (dataset)
Open X-Embodiment Collaboration
LeRobot datasets (Hugging Face)
Hugging Face
Implementation pitfalls
Data scarcity and costHigh

Embodied data is hard to collect at scale — teleoperation is slow and expensive.

Fix:Simulation/synthetic data, human video, cross-embodiment, data flywheel.
Embodiment gap and sim-to-realHigh

Data from one embodiment/simulation transfers poorly to other robots and reality.

Fix:Cross-embodiment learning, domain randomization, fine-tuning on target data.
Data quality, bias and safetyMedium

Inconsistent, biased or unsafe demonstrations degrade learned policies.

Fix:Curation, quality filtering, format standardization (e.g. dataset cards).

Evolution

2021
Human egocentric video as data (Ego4D)

Large-scale human video becomes a data source for embodied AI.

2023
Cross-embodiment: Open X-Embodiment / RT-X
Inflection point

Pooling data across many robots (cross-embodiment) demonstrates transfer and scaling of robot learning.

2024
Sim + real mix and data flywheels

Mixing real data with simulation and data flywheels become standard for robot foundation models.

Hyperparameters (configurable axes)

Data-source mixCritical

Proportions of teleoperation, human video, simulation and real interaction.

sim + realMixing synthetic and real data.
Embodiment coverageHigh

Single- vs multi-embodiment (cross-embodiment) data.

cross-embodimentData from many robots.
Annotation richnessMedium

From raw trajectories to rich labels (language, actions, success).

Computational complexity

Time complexity: Zalezna od skali i modalnosci danych. Space complexity: O(epizody × modalnosci × dlugosc).

Compute bottleneck

Data scarcity, embodiment gap and sim-to-real

Embodied data is scarce and costly, strongly tied to embodiment, and simulation data does not always transfer to reality (sim-to-real).

Depends on
Koszt teleoperacjiRoznorodnosc wcielen/domen

Execution paradigm

Primary mode
Dense

Concerns the data/training phase.

Activation pattern
All paths active
Routing mechanism

A data strategy, no routing.

Parallelism

Parallelism level
Fully parallel

Training and simulation data generation are highly parallel; real-robot collection is sequential.

Scope
Training
Constraints
!Collecting data from a real robot is inherently sequential (real time).

Hardware requirements

Primary

Simulation data generation and training on multimodal data are compute-intensive (GPU/TPU); large storage and I/O bandwidth are also needed.