A team from Stanford University, UC Berkeley and NVIDIA — with Li Fei-Fei among 34 co-authors — has published T-Rex, a manipulation model that reacts to touch in real time. The work, on arXiv since June 2026, adds a high-frequency tactile signal to vision-language-action (VLA) models and lifts average success by more than 30 percentage points over the strongest baseline.
Key takeaways
- T-Rex is a touch-reactive dexterous manipulation model — arXiv 2606.17055, with code released on GitHub.
- 34 co-authors, including Li Fei-Fei, Jitendra Malik, Pieter Abbeel, Ken Goldberg and Trevor Darrell.
- More than 30 percentage points higher average success than the strongest baseline — 65% on average across 12 tasks.
- Tasks include turning book pages, picking up an egg, squeezing toothpaste, pulling out thin cards and screwing in a light bulb.
- Trained on a 100-hour tactile-rich dataset.
- Architecture: a variable-rate Mixture-of-Transformers with a temporal tactile VQ-VAE encoder.
Touch as a correction loop
Contemporary vision-language-action models usually overlook touch or treat it as a secondary signal. T-Rex flips that logic: vision sets the plan, while touch corrects the motion in a high-frequency loop during contact with the object. That lets it react to slippage, resistance or changing grip force faster than the visual pathway alone could.
Vision is responsible for answering what I am supposed to do, while touch is tasked with continuously correcting whether what I am doing right now is correct.
The authors put this idea at the center of the paper, arguing that many abilities we call common sense come not from how many images we have seen, but from the body learning to adapt its actions to the world through long-term contact.
How the architecture works
At its core is a variable-rate Mixture-of-Transformers. A slower pathway plans motion from image and language, while a faster pathway closes the tactile correction — the two are joined by asynchronous cascaded flow matching, which separates slow visuomotor planning from fast tactile refinement. The touch stream is encoded by a separate temporal VQ-VAE?VQ-VAE: an encoder that turns a continuous signal into a sequence of discrete tokens encoder, so the model processes touch as a sequence rather than isolated readings.
Three parts of the T-Rex architecture:
Results and comparison
Across 12 dexterous manipulation tasks T-Rex reaches 65% success on average — more than 30 percentage points above the strongest baseline that ignores touch. The gap widens precisely in contact-dense tasks where vision alone falls short: pulling out a thin card or screwing in a light bulb demands a feel for resistance that a camera cannot capture. The training data is 100 hours of tactile-rich recordings collected around elementary motor primitives.
Why it matters
T-Rex shows that the bottleneck in dexterous manipulation is not visual perception itself but the absence of a fast tactile loop coupled to planning. Treating touch as a first-class modality rather than an add-on moves VLA models toward tasks that until now required manual tuning. For industrial and home robotics it makes handling delicate or slippery objects — where the error margin is minimal and vision fails — a more realistic prospect.
What's next?
- Code and architecture details are available in a public GitHub repository, letting teams reproduce the results on their own tactile sensors.
- An open question is transferring the 100-hour dataset to other hand designs and sensors — the authors flag data, not model size, as the main constraint.
