UMI
How it works
A human performs the task while holding a handheld parallel-jaw gripper with a mounted GoPro camera (fisheye lens), an IMU and side mirrors that widen the field of view. The gripper motion is recovered via visual-inertial tracking (SLAM) into a 6-DoF trajectory, and the grasp width from the image. The collected demonstrations are used to train a diffusion policy via imitation learning. At deployment, inference-time latency matching of observations and actions plus a relative-to-current-trajectory action representation let the same policy run on different robot arms.
Problem solved
Collecting manipulation-learning data usually requires an expensive robot and teleoperation, which is slow and limits environment diversity. UMI removes the need for a robot during data collection — a human records demonstrations with a low-cost gripper (~30 s per demo, over 3× faster than teleoperation) while keeping the data deployable on a real robot.
Components
A portable, low-cost handheld device mirroring a robot gripper, used to record demonstrations.
A wide-angle wrist-mounted camera capturing the scene from the gripper’s viewpoint.
An inertial sensor aiding recovery of the gripper motion (visual-inertial tracking).
Mirrors that widen the camera’s effective field of view with side views of the scene.
A learning-and-deployment layer based on a diffusion policy, with latency matching and a relative-trajectory action representation.
Implementation
Low light, motion blur or textureless scenes can cause drift or loss of gripper-trajectory tracking.
The latency gap between recording and the target robot degrades the policy if uncompensated.
Kinematic and dynamic differences between the handheld gripper and a robot arm can limit transfer.
Evolution
UMI presented at RSS 2024; Best Systems Paper Award finalist. Open-sourced hardware and software.
An extension of UMI to mobile manipulation with legged/quadruped robots.
Hyperparameters (configurable axes)
Number of recent observation frames fed to the policy.
Number of action steps the diffusion policy predicts at once.
Compensation for the observation/actuation latency difference between recording and the target robot.
Compute bottleneck
Recovering the gripper’s 6-DoF trajectory from the fisheye image and IMU is the key, sensitive data-processing step; its quality determines whether demonstrations are usable.
Hardware requirements
Training and inference of the diffusion policy require GPU acceleration.