Robots Atlas>ROBOTS ATLAS
Robotics

ViTacFormer

2025ResearchPublished: 21 October 2026Updated: 21 October 2026Published
Key innovation
Fuses vision and touch into a single representation space via a cross-attention encoder with an autoregressive head that predicts future tactile signals, driving imitation learning for multi-fingered hands and enabling precise, long-horizon dexterous manipulation.
Category
Robotics
Abstraction level
System
Operation level
Robot controlModelTraining
Use cases
Dexterous manipulation with multi-fingered anthropomorphic handsContact-rich and visually occluded manipulationLong-horizon, multi-stage manipulation tasksImitation learning for robotic handsVisuo-tactile sensor fusion for robot control

How it works

High-resolution vision and tactile observations are encoded and fused by a cross-attention encoder into a shared latent representation. An autoregressive head predicts future tactile signals (contact anticipation), forcing the latent space to encode information about upcoming contact. This representation feeds an imitation-learning policy trained on demonstrations that outputs actions for a multi-fingered hand. Training follows an easy-to-challenging curriculum that gradually increases task difficulty, refining the visual-tactile space and improving policy accuracy and robustness.

Problem solved

Vision-only methods struggle with precise, contact-rich manipulation, especially when the contact area is occluded or the setting is unstructured. ViTacFormer provides a unified representation combining touch with vision to enable fine-grained control and long-horizon, multi-stage dexterous manipulation tasks.

Components

Cross-attention encoder (vision-touch fusion)Cross-modal fusion of inputs

Fuses high-resolution vision and touch into a shared latent representation using a cross-attention mechanism.

Autoregressive tactile prediction headAnticipation of tactile signals

Predicts future contact signals, forcing a latent representation that encodes upcoming contact.

Imitation-learning policy headAction generation

Generates control actions for the multi-fingered hand from the shared visuo-tactile representation.

Implementation

Evolution

Original paper · 2025 · arXiv (preprint) · Liang Heng
ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation
Liang Heng, Haoran Geng, Kaifeng Zhang, Pieter Abbeel, Jitendra Malik
2025
ViTacFormer published (arXiv)
Inflection point

Introduction of the visuo-tactile fusion architecture with autoregressive tactile prediction; ~50% higher success rates and first autonomous long-horizon dexterous manipulation (up to 11 stages).

Execution paradigm

Primary mode
Dense

Dense transformer architecture; the tactile prediction head operates autoregressively.

Activation pattern
All paths active

Hardware requirements

Primary

Transformer architectures benefit from massively parallel matrix multiplication on GPU tensor cores.