This is the paper by Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang and Tat-Seng Chua of the National University of Singapore, East China Normal University and Hong Kong Baptist University that introduces the ScreenSpot-Pro benchmark. It was submitted to arXiv as 2504.07981 on 4 April 2025. The paper addresses GUI grounding, the task of locating the on-screen element an instruction refers to, in professional software with high-resolution displays, small targets and complex interfaces, and reports that existing grounding models perform poorly there.
Summary
Benchmark
ScreenSpot-Pro contains 1,581 instructions, each with its own screenshot, drawn from 23 applications in five professional categories plus common tasks in three operating systems. The application genres are development and programming, creative software, CAD and engineering, scientific and analytical tools, office software, and operating-system commons; Table 1 lists the specific software, including AutoCAD, Word and Excel, with versions and platforms.
Experts with at least five years' experience using the relevant applications recorded the data while performing their regular work. A silently running capture tool, triggered by a shortcut, took a screenshot and overlaid it so the expert could draw the target bounding box and write the instruction in real time. Screens above 1080p were prioritised, monitor scaling was disabled, and dual-monitor setups were captured across both displays. Targets are typed as text or icon, with a target counted as an icon only when no text label is present; text targets make up 62.6% of elements. Each instance was reviewed by at least two annotators to ensure a single unambiguous target with a precise interactable bounding box. Targets occupy 0.07% of the screenshot area on average, against 2.01% in the earlier ScreenSpot benchmark. An appendix adds a Chinese instruction for every task, translated by GPT-4 and reviewed by bilingual authors.
Scoring checks whether a model's predicted point falls inside the ground-truth box; for models that output boxes, the box centre is used.
Results for end-to-end models
Ten models that support GUI grounding were tested. OS-Atlas-7B led at 18.9%, followed by UGround (7B) at 16.5% and Aria-UI at 11.3%; no other model exceeded 10%. ShowUI (2B) and CogAgent (18B) scored 7.7%, OS-Atlas-4B 3.7%, MiniCPM-V 3.0%, Qwen2-VL-7B 1.6%, SeeClick 1.1%, GPT-4o 0.8% and QwenVL-7B 0.1%. Icons were much harder than text for every model: OS-Atlas-7B scored 28.1% on text targets and 4.0% on icons. The authors attribute the icon difficulty to the number of functions in professional applications, the assumption that users already know the icons, and icon meanings rarely seen in web training data. With Chinese instructions most models fell further, OS-Atlas-7B to 16.8% and UGround from 16.4% to 7.7%.
Search-based methods
The paper proposes that reducing the search area improves accuracy, and compares four multi-round methods using OS-Atlas-7B as the grounding model. Planner-free methods, Iterative Focusing, Iterative Narrowing and ReGround (a crop-and-reground baseline), reached 31.0%, 31.9% and 40.2%; ReGround's best crop size for OS-Atlas-7B was 1,024 × 1,024 pixels. ScreenSeekeR, an agentic framework in which GPT-4o acts as a planner that proposes likely regions, infers neighbouring elements and scores patches, raised OS-Atlas-7B from 18.9% to 48.1% without additional training, even though GPT-4o's own direct grounding accuracy is 0.8%. Ablations show accuracy falling to 41.9% without recursive search, 46.4% without neighbour inference and 46.8% when patch scoring is replaced by majority vote.
Limitations stated by the authors
The benchmark is confined to grounding and excludes agent planning and execution tasks of the kind in OSWorld, a choice the authors say was made to mitigate legal risks from software licensing restrictions.
Key claims
- 1,581 instructions, each with a unique high-resolution screenshot; 23 applications, 5 industries, 3 operating systems; targets average 0.07% of screen area; 62.6% text targets.
- Best end-to-end model OS-Atlas-7B 18.9% (28.1% text, 4.0% icon); GPT-4o 0.8%.
- ReGround 40.2% and ScreenSeekeR 48.1% with OS-Atlas-7B, training-free.
- Data recorded by experts with five or more years' experience during normal work; every instance reviewed by at least two annotators.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 1 (arxiv.org); abstract page arxiv.org. Kaixin Li is the contact author.
Relationships
- supports: ScreenSpot-Pro — the primary text for the benchmark's construction and release-time results.
- related: OSWorld — the agent-execution benchmark the authors distinguish their scope from.
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI, Multimodality