VDA
How it works
The policy receives two kinds of input: the current visual observation (e.g. an RGB or RGB-D image, a point cloud) and a context of one or a few demonstrations of the same task. Acting like in-context learning, the model relates the current observation to the demonstrations and predicts the robot’s next actions (e.g. gripper poses). In the representative instance (Instant Policy), demonstrations, observations and actions are encoded as a graph and actions are generated by a learned diffusion process over that graph; the model is trained on simulated pseudo-demonstrations. Key property: no weights are fine-tuned at inference — the new task is "loaded" purely through the in-context demonstrations.
Problem solved
It addresses the difficulty of quickly specifying new tasks to a robot: language instructions can be ambiguous and hard to ground in the scene, while classical imitation learning requires training a separate per-task policy. Conditioning on a demonstration lets a task be specified by showing it and generalized without retraining.
Implementation
Evolution
Introduction of learning a policy that infers the task from a single demonstration — the basis for conditioning a policy on a demonstration.
A representative method realizing the observation + demonstration -> action pattern: learns a new task from 1-2 demonstrations without further training, modelling reasoning over demonstrations, observations and actions as graph diffusion (Vosylius and Johns, Imperial College London; accepted at IROS 2025).