RFM
How it works
An RFM typically couples a pretrained vision-language backbone (VLM), which contributes internet-scale semantic knowledge, with an action head that generates motor commands. The model takes observations (camera images, robot state) and an instruction (natural language or goal image) and produces actions. Different action representations are used: discretizing actions as text tokens with autoregressive decoding (RT-2, OpenVLA), predicting action chunks with a diffusion head (Diffusion Policy, GR00T N1 System 1), or via flow matching (π0). Training uses a heterogeneous data pyramid — real robot trajectories, human videos, and synthetic data — and cross-embodiment data from many robots at once, enabled by a shared observation/action space. Adaptation to a new robot or task is done through fine-tuning (often within hours on a consumer GPU) or by changing the language instruction. Some models (GR00T N1) use a dual-system architecture: a slower vision-language module (System 2) and a fast, real-time motor controller (System 1).
Problem solved
Classical robot policies are designed per task, trained on narrow datasets, and deployed in specific environments on a specific robot, so they generalize poorly and require expensive data collection for each new platform. RFMs address this by using large-scale pretraining and cross-embodiment data so that a single model generalizes to new objects, instructions, and robots with minimal fine-tuning.
Components
A pretrained vision-language model contributing internet-scale semantic knowledge; processes images and language instructions.
Official
The module generating motor commands: action tokens (autoregression), a diffusion head, or flow matching, often predicting action chunks.
Official
A shared observation/action representation allowing one model to train on data from many robots (e.g. Open X-Embodiment: 1M+ trajectories, 22 embodiments).
A data mixture: real robot trajectories, human demonstration videos, and synthetic simulation data, balancing scale and realism.
Official
Implementation
Different robots have different action spaces and sensors; without a shared normalization, cross-embodiment transfer fails.
Large models may fail to generate actions in real time for dexterous manipulation.
Synthetic simulation data does not always transfer to a real robot.
Robot data is expensive and unevenly distributed across platforms and tasks, biasing the model.
Evolution
Stanford CRFM (Bommasani et al.) coins the "foundation model" concept — pretraining at scale and broad adaptability.
Google scales a transformer as a robot policy trained on a large set of real-world demonstrations.
Google DeepMind shows a vision-language model with actions as tokens transfers web knowledge to robot control.
1M+ trajectories from 22 robots; cross-embodiment training yields positive transfer (RT-1-X +50% low-data, RT-2-X 3x on emergent skills).
A transformer trained on 800k Open X-Embodiment trajectories; fine-tunes to a new robot in hours on a consumer GPU.
Physical Intelligence couples a VLM with a flow-matching head for dexterous manipulation across platforms (single-arm, dual-arm, mobile).
An open humanoid foundation model with a dual-system architecture (System 2 VLM + System 1 diffusion transformer), trained on a heterogeneous data pyramid.
Hyperparameters (configurable axes)
Parameter count of the VLM/policy backbone.
Number of action steps predicted at once (action chunking).
Number and diversity of robots in the training data.
Rate of action generation (Hz); critical for dexterous, high-frequency control.
Proportions of real robot, human video, and synthetic data.
Execution paradigm
Most RFMs are dense transformer-based models. Dual-system variants (GR00T N1) separate slow reasoning (System 2) from fast control (System 1).
Parallelism
Pretraining parallelizes across many devices; on-robot inference is constrained by real-time control (autoregressive action-token decoding can be sequential).
Hardware requirements
Pretraining and inference of large transformers leverage GPU tensor cores.
Google models (RT-2) trained on TPU; good training scalability.
Smaller models (Octo, GR00T N1 ~1B) can be fine-tuned and deployed on consumer/edge GPU on the robot.