1) A base generative video model is pre-trained at scale on footage, learning the dynamics and behavior of the world. 2) For a given task and observation, the model predicts future frames/video sequences (how the scene should unfold). 3) An action decoder converts these predicted frames into robot motion commands (e.g. trajectories, manipulator control). 4) The model is fine-tuned to a specific task on a small amount of robot data (demonstrations), because most of the dynamics knowledge comes from video pre-training.
Vision-Language-Action models trained on static images require very large amounts of robot data and generalize poorly to new scene dynamics. VAM reduces data requirements and improves dynamics understanding by transferring knowledge from generative video models.
A pre-trained video-generation model that understands world dynamics and behavior and predicts future scene frames.
Official
A module converting predicted frames/video sequences into robot motion commands (trajectories, manipulator control).
Official
Frame generation can be too slow for real-time control.
Predicted frames may be physically infeasible for the robot (the decoder must ground them).
mimic robotics introduces the Video-Action Model concept, combining video generation with robot action prediction.
mimic robotics and Black Forest Labs present FLUX-mimic; task fine-tuning from ~30 min of robot data, deployed at Audi Production Lab.
Time complexity: Zalezna od bazowego modelu wideo. Space complexity: O(rozmiar modelu wideo + dekoder).
Predicting video frames/sequences at control time is the most compute-intensive part; achieving latency sufficient for real-time robot control is a key challenge.
Choice of the generative video model that forms the core.
How predicted frames are mapped to motion commands.
Amount of robot data needed to fine-tune a task.
Prediction depends on the observation and task.
No routing โ a unified video model + decoder.
Training (video pre-training) is highly parallel; control-time inference is limited by the sequential loop.
Generative video models (diffusion/flow-matching transformers) are compute-intensive and benefit from GPU accelerators.
