Computer Use Agent
A computer-use agent is an AI system that operates a general-purpose computer through its graphical interface — looking at the screen, moving a cursor, clicking, typing and using keyboard shortcuts — rather than calling purpose-built tools or APIs. Because the interface is the universal layer that every desktop and mobile application already exposes to people, such an agent can in principle work with any software: a spreadsheet, a legacy accounting program, an image editor, a settings dialog. Browser agents are the web-only special case; computer use widens the scope to the entire operating system, including files, terminals and native applications.
The defining mechanism is the screenshot-action loop. The harness captures an image of the screen and passes it to a multimodal model together with the goal and recent history; the model replies with one action such as move to a coordinate, click, type a string, press a key combination or scroll; the harness executes it with a virtual mouse and keyboard, waits briefly for the interface to settle, takes a fresh screenshot and repeats until the model declares the task finished or a step budget runs out. Nothing in the loop is magic: the model has no hidden access to the application's state, so it must infer from pixels where a button is, whether a dialog opened and whether the last click worked. Hard problems therefore include precise visual grounding on high-resolution screens, slow latency per step, loss of context over long sessions and recovering from unexpected pop-ups.
Anthropic made the pattern a public product category in October 2024 by releasing computer use in public beta with an upgraded Claude 3.5 Sonnet. The API exposed a computer tool plus text-editor and bash tools, and Anthropic was explicit that the capability was experimental, error-prone and slow, recommending a sandboxed virtual machine or container, no access to sensitive accounts and human confirmation for consequential steps. OpenAI shipped its own Computer-Using Agent model through the Operator research preview in January 2025, and Google and others have released comparable systems, so the idea is now a cross-vendor interface standard rather than one lab's feature. Earlier research on GUI agents, including work on screen understanding and web interaction, supplied the ingredients; what changed was that general-purpose vision-language models became good enough to drive an arbitrary desktop with only a prompt.
OSWorld (Xie and colleagues, 2024) is the benchmark most often used to track progress. It provides real virtual machines running Ubuntu, Windows and macOS with several hundred tasks across applications such as a browser, office suite, file manager and code editor, each paired with an execution-based script that checks the resulting state rather than the sequence of clicks. At publication human participants completed over 70 percent of tasks while the best models managed only low double digits, and the screenshot-only result Anthropic reported for its October 2024 model was in the mid-teens, then considered state of the art. Scores have climbed considerably since, so quoted numbers age quickly, but the benchmark's lasting lesson is where failures cluster: imprecise clicking, misreading small interface elements, failing to notice an error and giving up or looping on a stuck state.
Security is the defining risk, because an agent with a mouse and keyboard inherits every permission of the account it runs under. The most discussed attack is prompt injection, in which text placed on a web page, in a document, in an email or even in an image tells the agent to abandon its task and instead copy files, open a terminal or send data elsewhere; the model cannot reliably distinguish instructions from the user from instructions embedded in content it merely looked at. Compounding this are ordinary mistakes with real consequences — deleting the wrong folder, submitting a form twice, buying the wrong item — and the possibility of a screen showing passwords or personal data being sent to a model provider. Standard defenses are isolation in a disposable VM, narrow credentials, network allowlists, logging every action, and requiring a person to approve irreversible or financial steps, which connects this topic directly to sandboxing and permission models.
A frequent misconception is that computer use is just a better version of robotic process automation. Traditional scripts record exact clicks and break when a button moves; a computer-use agent reads the screen and adapts, which makes it flexible but also non-deterministic, harder to audit and sometimes slower than a script by orders of magnitude. For a stable, repeated task with an API the script or API call remains the cheaper and more reliable answer. The agent earns its place on the long tail of one-off, messy, API-less jobs where a human would otherwise do the clicking, and where an occasional wrong turn is caught by a verification step or a person.
Read computer-use agent as a multimodal model sitting at a screen with a virtual mouse and keyboard, looping through see, act and look again: its reach is every application a person could use, OSWorld-style benchmarks show it still trails patient humans, and since it holds the full authority of its account while reading untrusted pixels, it belongs inside a sandbox with approvals for anything that cannot be undone.