XR
How it works
XR rests on three pillars. First — tracking: sensors and cameras determine, in real time, the position and orientation of the head, hands and controllers (typically with 6 degrees of freedom). Second — rendering: the system generates a stereoscopic 3D image matched to the user's motion, with the lowest possible motion-to-photon latency to avoid simulator sickness. Third — scene composition: in VR the image fully replaces the surroundings, in AR digital content is overlaid on the view of the world, and in MR virtual objects are anchored to room geometry and can interact with physical objects. In robotic applications these three pillars are coupled with the robot's state: the operator's hand motion maps to manipulator commands, and the robot's sensor data is rendered as objects in the scene.
Problem solved
Traditional 2D interfaces (screen, mouse, keyboard) poorly represent the three-dimensional space in which robots and machines operate. Operators struggle to judge depth, orientation and spatial relations, which slows down and introduces errors into remote control and data analysis. XR provides a spatial, intuitive sense of presence and direct 3D manipulation, shortening the distance between human intent and robot action.
Components
The headset occludes the real surroundings and displays a fully generated 3D scene. In robotics it is used for teleoperation in a reconstructed remote environment and for training in simulation.
The user still sees the surroundings while the system adds an information layer (labels, arrows, telemetry). In robotics it supports maintenance, assembly and on-site visualisation of machine state.
Virtual objects are anchored to real room geometry and respond to it (occlusion, collisions). In robotics it allows overlaying a robot's digital twin onto its physical counterpart.
Evolution
Ivan Sutherland builds the "Sword of Damocles", a precursor of today's headsets.
Tom Caudell at Boeing names the digital overlay assisting aircraft assembly.
Milgram and Kishino arrange VR, AR and MR on a single axis — the conceptual foundation of XR.
The launches of Oculus Rift, HTC Vive and Microsoft HoloLens bring XR to mass consumers and industry.
Apple Vision Pro popularises high-resolution MR and the "spatial computing" framing.
Hyperparameters (configurable axes)
Time from the user's movement to the corresponding change on screen. Critical for comfort — excessive latency causes simulator sickness.
Angular extent of the image seen by the user; a wider FOV increases the sense of immersion.
Number of independent motion axes tracked by the system. 6DoF covers rotation and translation of the head and controllers.