This is the paper introducing the OSWorld environment and benchmark. Its seventeen authors are led by Tianbao Xie, Danyang Zhang and Jixuan Chen with Tao Yu as senior author, at the University of Hong Kong, with co-authors affiliated to Carnegie Mellon University, Salesforce Research and the University of Waterloo. It was posted to arXiv as 2404.07972 on 11 April 2024 and revised on 30 May 2024 (v2, the version summarised here). Code, environment, baselines and data are released at os-world.github.io.
Summary
The paper argues that earlier agent benchmarks fall into two groups, neither of which reflects real computer use: datasets of demonstrations without an executable environment, whose non-execution-based evaluation "assumes a single solution for each task and wrongfully penalizes alternative correct solutions," and executable environments that simplify the observation and action spaces and confine tasks to one application or domain such as web navigation or coding. OSWorld is presented as "the first-of-its-kind scalable, real computer environment for multimodal agents," supporting task setup, execution-based evaluation and interactive learning on Ubuntu, Windows and macOS.
Environment
A task is formalised as a partially observable Markov decision process. At each step the agent receives a natural-language instruction plus a screenshot, an accessibility (a11y) tree, or both, and emits an executable action such as a pyautogui mouse click at pixel coordinates or a key combination. The loop ends when the agent outputs DONE or FAIL or reaches the step limit, which was 15 in the paper's experiments. The reward function is execution-based and returns 1, or a positive decimal below 1 for partial achievement, if the final state meets the task objective or if the agent correctly predicts that an infeasible task cannot be done.
Benchmark construction
The benchmark comprises 369 tasks on Ubuntu, with a supplementary 43 Windows tasks that require user activation because of copyright restrictions. Tasks centre on eight applications — Chrome, VLC, Thunderbird, VS Code, LibreOffice Calc, Writer and Impress, and GIMP — plus basic OS tools such as the terminal, file manager, image viewer and PDF viewer. Task ideas were drawn from official guidelines and tutorials, video tutorials, how-to sites, Q&A forums, video courses and personal blogs, selected by popularity and diversity; because multi-application tasks were hard to find online, the authors also brainstormed them. Nine computer-science student authors spent over three months and roughly 1,800 person-hours (650 on single-app tasks, 750 on workflow tasks, 400 on double-checking).
Of the 369 tasks, 268 (72.6%) are single-application and 101 (27.4%) are multi-app workflows; 84 (22.8%) are integrated from NL2Bash, Mind2Web, SheetCopilot, PPTC and GAIA; and 30 (8.1%) are infeasible, built from deprecated or hallucinated features that real users had asked about. The benchmark has 302 distinct initial states and 134 unique evaluation functions, a count the authors describe as "orders of magnitude larger than prior work"; the comparison table lists WebArena with 5, VisualWebArena with 6, and MiniWoB++ with 125. Each task carries an initial-state configuration (about one person-hour to build) and an evaluation script pairing getter functions that extract evidence from the final state with an evaluator function (about two person-hours). Quality control included cross-checking by two other authors, an attempt by two authors who had not annotated the task, and four rounds of fixes totalling more than 400 person-hours.
Human performance
Annotators were computer-science students with basic software skills who had not seen the tasks or the software. Their median completion time was 111.94 seconds, against 35.38 seconds on 100 WebArena tasks sampled under the same setup, with a substantial number of OSWorld tasks taking 900 seconds or more. Human accuracy was approximately 72.36%, against 88% on the WebArena sample. By category the human figures were 75.00% (OS), 71.79% (Office), 70.51% (Daily), 73.47% (Professional) and 73.27% (Workflow).
Results
Baselines covered Mixtral-8x7B, Llama-3-70B, GPT-3.5, GPT-4, Gemini-Pro, Gemini-Pro-1.5, Qwen-Max and GPT-4o with accessibility-tree input, and CogAgent, GPT-4V, Gemini-ProV, Gemini-Pro-1.5, Claude-3-Opus and GPT-4o under screenshot-only, screenshot-plus-a11y-tree and Set-of-Mark settings. Overall success rates ranged from 0.99% to 12.24%. The best result was GPT-4 with the accessibility tree at 12.24% (OS 20.83%, Office 3.58%, Daily 25.64%, Professional 26.53%, Workflow 2.97%); GPT-4V with screenshot plus a11y tree reached 12.17%, GPT-4V with Set-of-Mark 11.77%, GPT-4o with a11y tree 11.36%. Screenshot-only agents scored between 5.26% (GPT-4V) and 5.80% (Gemini-ProV), and Claude-3-Opus reached 2.42% on screenshots alone, 4.41% with the a11y tree added and 6.72% with Set-of-Mark. The authors note that Claude-3 Opus, "reported to be competitive with GPT-4V on common benchmarks," fell far behind as a digital agent, trailing GPT-4V by 2.84% to 7.76% on average; qualitatively, Claude produced satisfactory high-level plans but its grounding "contains hallucinations in detail," such as treating column B in Calc as column C.
Under the GPT-4V Set-of-Mark configuration used for the analysis, success fell with human completion time — 16.78% on tasks under 60 seconds, 13.12% for 60–180 seconds, 4.59% above 180 seconds, against human rates of 84.91%, 81.08% and 49.57% — and single-app tasks (13.74%) were solved at more than twice the rate of multi-app workflows (6.57%). Agents performed slightly better on infeasible tasks (16.67%) than feasible ones (13.34%), though the authors observed that some configurations, such as Gemini-Pro on screenshots, tended to output FAIL and stop, producing false positives on infeasible tasks. Performance was better on CLI-oriented OS tasks than on GUI-heavy Office tasks, with LibreOffice Calc subsets often scoring zero, whereas human performance stayed around 70% with under 5% variance across categories.
Further analyses found that higher screenshot resolution improved screenshot-only performance; that longer text-based trajectory history helped under Set-of-Mark but image-only history did not; that a single a11y-tree observation needs roughly 6,000 tokens of context to cover about 90% of cases; and that perturbing window position or size or cluttering the screen with irrelevant applications cut success on a 28-task subset from 50.79% by "over 60% to even 80%." A GPT-4V screenshot-only run on the 43 Windows tasks scored 2.55% against 4.88% on the matching Ubuntu tasks, with a correlation coefficient of 0.7. Among 550 failed GPT-4V episodes, more than 75% involved mouse-click inaccuracies, which cascade into repeated clicks and pop-ups; agents also lacked prior knowledge of professional software and sometimes completed tasks by code in ways that ignored the instruction (using ffmpeg where the task said to use GIMP).
Future directions and safety
The conclusion lists four directions: VLMs with longer context and more robust GUI grounding, agent architectures for exploration, memory and reflection, safety, and broader data and environments. On safety the authors write that they "have not observed any harmful and damaging behaviors during our experiments" but that a universal agent could bypass CAPTCHAs, abuse accounts or exploit software vulnerabilities; the evaluators check task completion only and "pay little attention to potential unnecessary damaging actions," and the authors "didn't work out an efficient way to detect the latent side effects of the agent."
Key claims
- 369 Ubuntu tasks (268 single-app, 101 multi-app workflow, 84 integrated from prior benchmarks, 30 infeasible) plus 43 Windows tasks; 302 initial states; 134 unique execution-based evaluation functions.
- Human success 72.36% (CS students unfamiliar with the software), median 111.94 seconds per task; 88% and 35.38 seconds on a 100-task WebArena sample.
- Best model 12.24% (GPT-4, a11y tree, 15 steps); screenshot-only 5.26%–5.80%; best multi-app workflow result 6.57%; Claude-3-Opus 2.42%–6.72% depending on input.
- Main failure sources: GUI grounding (mouse-click inaccuracy in over 75% of 550 failed GPT-4V episodes) and operational knowledge of applications.
- Higher resolution and longer text history help; Set-of-Mark reduced GPT-4V's score relative to screenshot-plus-a11y-tree, which the authors attribute to element density on desktop screens.
- Stated limitations: false positives and negatives remain despite four rounds of checks; evaluators do not detect side effects; a11y-tree quality varies across applications.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org); the abstract page (arxiv.org) records submission on 11 April 2024 and revision on 30 May 2024, 51 pages and 21 figures. The paper is licensed CC BY 4.0.
Relationships
- supports: OSWorld — the primary text for the benchmark's design, human baseline and launch results.
- supports: WebArena — supplies the comparative human study (88%, 35.38 seconds) on a WebArena sample.
- depends-on: GAIA — 84 integrated tasks include items adapted from GAIA.
- related: OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks (Yuan et al., 2026) — the long-horizon successor from the same lab.
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI, Agentic harnesses and capability elicitation