This is the paper introducing the GAIA benchmark, by Grégoire Mialon (FAIR, Meta), Clémentine Fourrier and Thomas Wolf (Hugging Face), Craig Swift (AutoGPT), Yann LeCun (FAIR, Meta) and Thomas Scialom (GenAI, Meta). It was posted to arXiv as 2311.12983 on 21 November 2023. The benchmark and leaderboard are hosted at huggingface.co/gaia-benchmark.
Summary
The authors present GAIA as a benchmark "that, if solved, would represent a milestone in AI research": real-world questions requiring "reasoning, multi-modality handling, web browsing, and generally tool-use proficiency." They set it against the trend of benchmarks that grow ever harder for humans, arguing that questions "conceptually simple for humans yet challenging for most advanced AIs" better test the robustness they associate with general intelligence, and report that human respondents obtain 92% against 15% for GPT-4 with plugins.
Design principles
Four principles are stated. First, questions should be conceptually simple though possibly tedious for people, rooted in the real world and varied, so that they exercise fundamental abilities rather than specialist skill; most consist in finding and transforming information from documents or the open web. Second, interpretability: a small, curated set whose reasoning traces a user can follow. Third, robustness to memorisation: answers are absent by design from pre-training text, the required accuracy and the ability to inspect traces mitigate contamination, and new questions can be written from the published guidelines if memorisation occurs. Fourth, ease of use: zero-shot prompts, sometimes with an attached file, and factoid answers that allow "simple, fast and factual evaluation."
Composition and levels
GAIA has 466 questions, text-based and sometimes accompanied by a file (image, spreadsheet, PDF, audio or video). A prefix prompt instructs the system to end with FINAL ANSWER: followed by a number, as few words as possible, or a comma-separated list, under formatting rules; scoring is quasi-exact match after type-dependent normalisation, with a single correct answer per question. Questions are sorted into three levels by the number of steps and tools the annotator used: Level 1 (146 questions) generally needs no tool or one tool and at most five steps; Level 2 (245) roughly five to ten steps combining tools; Level 3 (75) "a near perfect general assistant" taking arbitrarily long action sequences. A developer set of 166 annotated questions is released with answers; the remaining 300 are released without annotations to power a leaderboard.
Construction and validation
Questions were written by the authors and compensated annotators from Surge AI, each grounded in one or more stated sources of truth — stable web pages such as Wikipedia, Papers With Code or arXiv, an attached document, or a self-contained puzzle — with the creator recording the answer, tools, steps and time. Two further annotators then answered each question independently; a question was validated when all three agreed, fixed when the disagreement was simple, and otherwise removed. Of 623 newly crafted questions, both new annotators agreed with the original answer on 55%, one on 27% and neither on 18%; 68% were valid as written (75% of Level 1, 68% of Level 2, 47% of Level 3). The human score of 92% is the fraction of correct answers by validating annotators on valid questions (94%, 92% and 87% by level in the validation table; 93.9%, 91.8% and 87.3% in the results table, taking on average 6.8, 10.5 and 17.7 minutes). Creating a question, including validation and repair, is estimated at two hours of annotator time. Web-hosted evidence may change or be blocked by robots.txt, so questions specify page versions where needed and the authors verified that evidence pages were not bot-restricted.
Results
GPT-4 with plugins was queried by hand through ChatGPT, with plugins chosen per question by the authors, so its score is described as an oracle that "cannot be reproduced exactly"; models with API access were run three times and averaged. Scores on Levels 1, 2 and 3: GPT-4 9.1, 2.6 and 0; GPT-4 Turbo 13.0, 5.5 and 0; AutoGPT with a GPT-4 backend 14.4, 0.4 and 0; GPT-4 with plugins 30.3, 9.7 and 0; a search engine (answer deducible from the first results page) 7.4, 0 and 0; human annotators 93.9, 91.8 and 87.3. The authors read the correlation between levels and model performance as validating the difficulty definition. GPT-4 without tools solved some browsing questions by having memorised the intermediate facts, and AutoGPT underperformed plain GPT-4 on Levels 1 and 2, which the authors attribute to how it calls the GPT-4 API.
Discussion and limitations
The paper notes that closed models and plugin sets change over time, making evaluations hard to reproduce; that GAIA "is indeed likely to decay over time" through contamination or disappearance of web evidence, so it should evolve year by year; that solving GAIA "requires full automation since no approximation is allowed in the answer"; and, in a stated argument for open source, that the value of automation risks being captured by technology owners. Listed limitations are that GAIA does not evaluate the reasoning trace, so a correct answer reached by a flawed path counts; that unambiguous questions are costly to write and sometimes carry unnatural detail; and that all questions are in standard English and rely largely on English web pages, so the benchmark says nothing about assistants for the roughly 80% of the world that does not speak English.
Key claims
- 466 questions; Levels 1/2/3 of 146/245/75; 166-question developer set with annotations, 300 held for the leaderboard.
- Human validation score 92% aggregated (93.9/91.8/87.3 by level); 68% of drafted questions valid as written.
- Release results: GPT-4 with manually chosen plugins 30.3/9.7/0; GPT-4 Turbo 13.0/5.5/0; AutoGPT 14.4/0.4/0; GPT-4 9.1/2.6/0; search engine 7.4/0/0.
- Scoring is quasi-exact match on a single reference answer; no trace evaluation, no partial credit.
- Stated limitations: trace not evaluated, question design cost, English-only.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); the abstract page (arxiv.org) records a single version submitted 21 November 2023.
Relationships
- supports: GAIA — the primary text for the benchmark's design, levels, human baseline and release results.
- supports: OSWorld — GAIA items were among the 84 tasks integrated into OSWorld (OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (Xie et al., 2024)).
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI, Meta AI, Hugging Face