Browser Agent
A browser agent is an AI system that carries out tasks by operating a web browser the way a person would: reading pages, clicking links and buttons, typing into forms, scrolling, switching tabs and deciding what to do next from what it sees. The model is not handed a clean API for each website; it is handed the same messy, visual, stateful interface that human users face, and must turn a goal such as "find the cheapest refundable flight and put it in the cart" into a sequence of low-level browser actions. That makes it a specialization of the broader agent idea, distinguished by its environment — the open web — rather than by any particular model architecture.
The lineage starts with models that browsed only to answer questions. WebGPT (OpenAI, Nakano and colleagues, 2021) fine-tuned GPT-3 to drive a text-based browser backed by a search engine, issuing commands such as search, click a link, scroll and quote, then composing a long-form answer with references; it was trained with imitation of human demonstrations and then with human preference feedback, an early example of the reward-model recipe applied to tool use. Earlier simulated shopping environments such as WebShop (Yao and colleagues, 2022) showed that language models could act in web-like worlds at all. The step from answering questions to completing transactions, where actions change state, is what turned browsing into agency and what makes errors expensive.
Benchmarks are what made the field measurable, and they consistently showed a large gap between machines and people. WebArena (Zhou and colleagues, 2023) built self-hosted, reproducible websites — an online store, a Reddit-style forum, a GitLab instance and a content-management system — with several hundred long-horizon tasks checked by whether the final state is correct; at release the best GPT-4-based agent solved only a low-teens percentage of tasks against roughly four-fifths for humans. Mind2Web (Deng and colleagues, 2023) took the complementary route of collecting thousands of tasks across over a hundred real websites in dozens of domains, testing whether an agent generalizes to unseen sites and domains rather than one fixed sandbox. VisualWebArena (2024) later added tasks that cannot be solved without looking at images. Scores have risen quickly since, yet exact figures depend heavily on the scaffold, so a headline number is best read as a snapshot of one agent configuration rather than of a model.
The central design choice is how the page is represented: as structured text from the DOM or accessibility tree, or as raw pixels from a screenshot. DOM-based agents receive the document tree or an accessibility snapshot with element identifiers, which is compact, precise about what is clickable and cheap to process, but pages can contain thousands of nodes, hide meaning in scripts and canvases, and differ between what the tree says and what is visually rendered. Screenshot-based agents see what a human sees — layout, icons, charts, overlapping pop-ups — and output coordinates or element descriptions, which is more general but demands strong visual grounding and is vulnerable to small misclicks. Many production systems combine both, for example the SeeAct line of work (Zheng and colleagues, 2024) that grounds a model's visual plan onto DOM elements. A further hybrid, "set-of-marks" prompting, overlays numbered boxes on a screenshot so the model can name an element rather than guess pixels.
Browser agents became consumer products in 2025, and with them the security problem became concrete. OpenAI's Operator arrived in January 2025 as a research preview that controlled a remote browser, and other vendors followed with agentic browsers and browser extensions. A page the agent reads is untrusted input that sits in the same context window as the user's instructions, so hidden text on a web page can attempt prompt injection — telling the agent to exfiltrate data, buy something or ignore its task. Mitigations include confirmation before purchases and logins, restricting which sites and credentials an agent can touch, running the browser in isolation, and separating trusted instructions from page content, but none of these is a complete fix. Other practical hurdles are CAPTCHAs, bot detection, paywalls, session expiry and the sheer slowness of an agent that takes a screenshot or snapshot at every step.
A common misconception is that a high benchmark score means the agent can be left unattended on the real web. Benchmarks use fixed, cooperative sites with unambiguous success checks; live sites change layout, throw cookie banners, behave differently per region and contain adversarial content. Reliability also compounds: a task of thirty steps with a 95 percent per-step success rate succeeds only about one time in five, so long workflows depend on verification, retries and human checkpoints. Where a site offers an API, calling it is almost always faster and safer than clicking through the interface, so browsing is best seen as the fallback for the long tail of software with no API at all.
Read browser agent as a language model given a web browser and a goal: the page representation (DOM, screenshot or both) decides what it can perceive, benchmarks such as WebArena and Mind2Web show how far it still trails a patient human, and because everything on the page is untrusted input, how much it may do unsupervised is a security decision as much as a capability one.