In a loop: 1) perception - the agent receives a screenshot (and optionally an accessibility tree/DOM) and interprets it with a vision-language model; 2) reasoning/planning - it decides the next step toward the goal; 3) action - it performs a GUI operation (click at specific coordinates, type text, scroll, shortcuts); 4) observe the effect and repeat until the task is complete. The agent often runs in an isolated environment (VM/browser), can use the user's logged-in accounts, and combines GUI control with tool/API calls where available.
Automating computer tasks usually required API integrations or brittle RPA scripts that break when interfaces change. A CUA operates at the GUI layer like a human, so it can handle apps without APIs and adapt to interface changes.
Analyzing screenshots (and optionally DOM/accessibility tree) with a vision-language model to understand the interface.
Deciding the next steps toward the goal based on the screen state.
The action set: click (coordinates), typing, scrolling, keyboard shortcuts.
Official
An isolated environment with a browser and file system, often with access to user accounts.
Official
A CUA acts with the user's privileges (logged-in accounts), so mistakes or hijacking can cause real harm.
Malicious page content can try to hijack the agent and induce unwanted actions.
Multi-step GUI tasks are prone to compounding errors; a single wrong click can derail the whole flow.
In October 2024 Anthropic released Computer Use, in which Claude controls a computer via screenshots and GUI actions - a breakthrough in popularizing CUAs.
In January 2025 OpenAI introduced the Operator agent based on a CUA model, performing browser tasks; Google was developing Project Mariner.
Specialized, commercial CUAs such as Hark Handoff emerged, competing on benchmarks (Online-Mind2Web) and lowering cost per token.
