ViLLA
How it works
Stage 1 (Latent Action Model): an inverse-dynamics encoder I(z_t | I_t, I_{t+H}) and a forward-dynamics decoder F(I_{t+H} | I_t, z_t) model the dynamics between consecutive frames. The encoder is a spatial-temporal transformer with causal temporal masks; the decoder is a spatial transformer that takes the initial frame and discretized latent action tokens z_t = [z_t^0, ..., z_t^{k-1}] (k=4), quantized with a VQ-VAE objective and a codebook of size |C|. Stage 2 (Latent Planner): a web-scale pre-trained VLM uses the latent actions as pseudo-labels for embodiment-agnostic long-horizon planning. Stage 3 (Action Expert): the Action Expert is introduced and jointly trained with the Latent Planner, turning the latent plan into concrete, dexterous robot control actions.
Problem solved
The volume of action-labeled robot data is small relative to web-scale data, limiting the generalization of classic VLA. ViLLA addresses this data bottleneck by introducing latent actions as an intermediate representation learned from action-unlabeled human videos and cross-embodiment data, and it decouples long-horizon planning from low-level control.
Components
An encoder-decoder trained on internet-scale heterogeneous data. The inverse-dynamics encoder I(z_t|I_t,I_{t+H}) is a spatial-temporal transformer with causal temporal masks; the forward-dynamics decoder F(I_{t+H}|I_t,z_t) is a spatial transformer. Latent action tokens z_t=[z_t^0..z_t^{k-1}] (k=4) are quantized with a VQ-VAE objective and a codebook of size |C|.
Leverages a web-scale pre-trained VLM and uses latent actions as pseudo-labels, combining the generalizability of the VLM with embodiment-agnostic planning over sequences of latent actions.
Introduced in Stage 3 and jointly trained with the Latent Planner; it turns the latent plan into concrete low-level control actions enabling dexterous manipulation.
Implementation
Evolution
AgiBot (Zhiyuan) publishes AgiBot World Colosseo (arXiv:2503.06669), defining the hierarchical ViLLA framework and implementing it in the GO-1 (Genie Operator-1) generalist policy.
Execution paradigm
The core components (VLM Latent Planner, LAM spatial/spatial-temporal transformers, Action Expert) are dense transformer architectures; processing proceeds through three stages with distinct roles.
Hardware requirements
Training and inference of the transformer components (VLM, LAM, Action Expert) at large data scale benefit from GPU acceleration.