Hand Tracking
How it works
A typical pipeline runs in two stages: a detector first localizes the hand in the frame (a palm/hand detector), then a landmark regression model reconstructs joint coordinates (21 landmarks with x, y and relative depth in MediaPipe Hands). Between consecutive frames, temporal tracking and filtering reduce full re-detection and stabilize the output. In devices such as Leap Motion the 3D hand geometry is reconstructed from a pair of infrared images, while XR headsets use their built-in cameras with machine learning. The resulting hand skeleton is then interpreted as gestures or mapped onto the joints of a gripper or manipulator for teleoperation.
Problem solved
Interacting with digital systems and robots used to require physical controllers, sensor gloves or markers. Hand tracking captures rich, high-dimensional finger-motion information without contact, using inexpensive vision hardware — which is essential for natural XR interaction and for intuitive teleoperation of dexterous robot hands.
Components
Detects the presence and region of a hand in the frame, returning a bounding box that initializes the landmark model.
Official
Regresses hand joint coordinates within the detected region; 21 landmarks (x, y and relative depth) in MediaPipe Hands.
Official
Uses the prediction from the previous frame to skip full detection and smooth jitter in the landmarks.
Official
Implementation
Fingers occluding each other or folded into the palm cause erroneous joint estimates.
Fast hand motion blurs the image and degrades landmark accuracy.
A single RGB camera yields only relative depth, hindering accurate 3D pose reconstruction.
Touching or overlapping hands can be confused or merged by the detector.
Poor or variable lighting and external infrared sources (e.g. sunlight) degrade tracking.
Evolution
First shipment of the optical Leap Motion Controller (two monochromatic infrared cameras) for contact-free hand tracking, July 2013.
Introduction of experimental controller-free hand tracking on Quest (December 2019) from the headset's built-in cameras using machine learning.
Release of MediaPipe Hands — on-device, real-time hand tracking from a single RGB camera (21 landmarks).
A general vision-based system for dexterous robot hand teleoperation driven by human hand tracking.
Hyperparameters (configurable axes)
Number of joints/keypoints reconstructed per hand.
How many hands are tracked simultaneously.
Type of sensor providing the input.
Confidence threshold above which a hand detection is accepted.
Model variant trading accuracy for performance (e.g. lite vs full).
Execution paradigm
Reflects typical CNN-based implementations (detector + landmark regressor) where all network paths are active for each frame.
Parallelism
The two-stage pipeline (detection -> landmarks) introduces a sequential dependency within a frame, but CNN inference is highly parallel on GPU/NPU.
Hardware requirements
CNN inference for detection and landmark regression maps well onto GPUs.
Lightweight models (e.g. MediaPipe Hands) run in real time on mobile and edge CPUs.