This is the paper introducing WebArena, by Shuyan Zhou and Frank F. Xu (lead contributors), Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon and Graham Neubig of Carnegie Mellon University. It was posted to arXiv as 2307.13854 on 25 July 2023 and last revised on 16 April 2024 (v4, summarised here). Code, data, environment reproduction resources and video demonstrations are at webarena.dev.
Summary
The authors observe that agents were "primarily created and tested in simplified synthetic environments" and set out to build one that is both realistic and reproducible: standalone rather than dependent on live websites, which avoids CAPTCHAs, unpredictable content changes and configuration drift, and realistic because it uses the open-source libraries behind popular sites and imports sampled data from their real-world counterparts.
Environment
Site categories were chosen by analysing about 200 entries from the authors' own browser histories. The environment contains one instance in each of four categories — an e-commerce store (OneStopShop), a Reddit-style social forum, a GitLab instance built from the actual GitLab project, and an online-store content-management system — plus three utilities (a map, a calculator, a scratchpad) and knowledge resources including an English Wikipedia and site user manuals. It is delivered as Docker containers with reset scripts, and each platform has generated user profiles so that agents act under particular roles and permissions. The observation is a page URL, the open tabs and the focused page's content as raw HTML, a screenshot or an accessibility tree, optionally limited to the viewport; the paper describes WebArena as the first web environment to support multi-tab tasks. The action space covers element operations (click, hover, type, key presses, scroll), tab operations and URL navigation, with elements addressed by coordinates or by an ID prepended during DOM or accessibility-tree traversal.
Benchmark
The benchmark has 812 tasks instantiated from 241 templates, about 3.3 per template. Authors wrote intents to be abstract and high-level, creative (adding constraints such as "create a Reddit account identical to my GitLab one"), and templated with variables. Tasks fall into information seeking (a textual answer, often requiring navigation across pages or user-specific content), site navigation, and content and configuration operations. Some tasks are deliberately unachievable, such as asking for a store's contact number the site does not provide, with "N/A" as the expected response. Information-seeking answers are scored by exact_match, must_include or fuzzy_match, the last using gpt-4-0613 to judge semantic equivalence; the other categories use programmatic reward functions with a locator (a database query, API call or JavaScript selection) and keyword checks on intermediate states. Reference answers were annotated twice with a third annotator resolving disagreements, and evaluation programs were written by three JavaScript-proficient authors.
Human performance
Five computer-science graduate students each attempted one task from each of 170 templates, completing 78.24% overall (74.68% on information seeking, 81.32% on the rest) in 110 seconds on average. Half of the failures were misinterpretations of the intent, incomplete answers or incomplete executions; the remainder were off-target. The authors note that many tasks require domain knowledge, such as what a merge request is.
Results
Three models (GPT-4, GPT-3.5, text-bison-001) were run with two in-context examples, an accessibility-tree observation with element IDs, with or without chain-of-thought reasoning, and with or without an "unachievable hint" telling the agent to stop if a task is impossible. With CoT and the hint, GPT-4 reached 11.70% (8.63% on achievable tasks, 77.78% on unachievable), GPT-3.5 8.75% and text-bison-001 5.05%; CoT added 2.34 points for GPT-3.5. Error analysis found GPT-4 declaring 54.9% of feasible tasks impossible; removing the hint raised GPT-4 to 14.41% (13.02% achievable) while it still recognised 44.44% of unachievable tasks by generating reasons of its own, whereas GPT-3.5 tended to hallucinate answers, repeat invalid actions or exceed step limits. Of the 61 templates on which the GPT models succeeded at least once, GPT-4 achieved 100% on only four and GPT-3.5 on none, which the authors present as evidence that tasks from one template still present distinct challenges and as a motivation for agents with memory of past successes.
Key claims
- Self-hosted, reproducible environment: OneStopShop, Reddit-style forum, GitLab, CMS, map, calculator, scratchpad, Wikipedia; multi-tab; Docker delivery.
- 812 tasks from 241 templates (3.3 per template); three task categories plus unachievable tasks; functional-correctness scoring with
exact_match,must_include,fuzzy_match(GPT-4 judge) and programmatic state checks. - Human baseline 78.24% (74.68% information seeking, 81.32% other), 110 seconds average, five CS graduate students on one task from each of 170 templates.
- GPT-4 with CoT: 11.70% with the unachievable hint, 14.41% without; GPT-3.5 8.75%; text-bison-001 5.05%.
- GPT-4 wrongly declared 54.9% of feasible tasks unachievable when prompted with the hint.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 4 (arxiv.org); the abstract page (arxiv.org) records submission on 25 July 2023 and revision on 16 April 2024.
Relationships
- supports: WebArena — the primary text for the environment, task suite, human baseline and launch results.
- related: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (Xie et al., 2024) — the desktop successor from an overlapping author group, which re-measured humans on a WebArena sample.
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI, AI Agentic Browsers, Carnegie Mellon University