GDPval

GDPval is a benchmark from OpenAI that tests whether an AI model can produce the kind of work product a professional is paid for, such as a legal brief, a sales forecast spreadsheet, a nursing care plan or a slide deck. OpenAI announced it on September 25, 2025 and posted the paper to arXiv on October 5, 2025 (GDPval Paper (OpenAI, Oct 2025); Source: openai.com). A model's output is placed next to the deliverable an experienced professional wrote for the same assignment, and other professionals from that occupation, not told which is which, say which they prefer. The headline score is the share of comparisons the model wins or ties. At release the best model tested reached 47.6%; OpenAI reported 84.9% for GPT-5.5 in April 2026 (Source: openai.com). OpenAI's hosted leaderboard has since been closed and points readers to GDPval-AA, a third-party implementation by Artificial Analysis that uses model judges and reports an Elo rating (Source: evals.openai.com).

What it measures

The paper presents GDPval as a way to gauge the economic relevance of AI ahead of adoption statistics, which the authors call lagging indicators: instead of counting how many firms use AI, it asks directly whether model output is as good as expert output on tasks with a known time and wage cost (GDPval Paper (OpenAI, Oct 2025)). The name refers to the selection method, which starts from the industries that contribute most to U.S. gross domestic product, and not to any estimate of GDP effects (Source: openai.com).

The construct is narrow by design. Each task is a single, fully specified assignment with its reference files attached, completed in one pass. OpenAI's announcement says the benchmark does not capture work in which a model would have to build context or revise through several drafts, and that most jobs are more than a list of tasks that can be written down (Source: openai.com).

Design

Occupation selection

The authors selected the nine sectors that each contribute more than 5% of U.S. GDP, using Federal Reserve Bank of St. Louis value-added data for the second quarter of 2024: real estate and rental and leasing (13.8%), government (11.3%), manufacturing (10.0%), professional, scientific and technical services (8.1%), health care and social assistance (7.6%), finance and insurance (7.4%), retail trade (6.3%), wholesale trade (5.8%) and information (5.4%). Within each sector they took the five occupations with the largest total wage bill, from May 2024 Bureau of Labor Statistics data, among occupations judged to be predominantly digital. That judgment was made by having GPT-4o classify each of an occupation's ONET tasks as digital or non-digital, weighting by ONET's relevance, importance and frequency ratings, and keeping occupations at 60% or above. The result was 44 occupations that the paper says collectively earn $3 trillion a year, ranging from software developers, lawyers and registered nurses to order clerks, concierges and film and video editors (GDPval Paper (OpenAI, Oct 2025)).

Task authorship and review

Tasks were written by professionals with at least four years in the occupation and an average of 14; fewer than 10% of applicants were accepted, and each occupation had at least five contributors. Each task has a request, usually with reference files, and the expert's own deliverable. An example OpenAI published asks a manufacturing engineer to design a jig so that one person rather than two can run a cable-spooling test on a mining truck, and to deliver a PDF summarizing the 3D design (Source: openai.com). Deliverables include documents, spreadsheets, slide decks, diagrams, CAD files, audio and video. In the 220-task gold subset the average task took the expert 9.49 hours (median 5) and was valued at $398.46, computed as completion time multiplied by the occupation's median hourly wage; 67.7% of tasks required working with at least one reference file. Every task passed through automated screening and at least three rounds of human review, five on average, and graders rated 89.07% of gold-subset prompts as well specified (GDPval Paper (OpenAI, Oct 2025)).

Scoring

Graders are professionals in the same occupation. They see the request, the reference files and two or more unlabeled deliverables, rank them, and label the model's as better than, as good as, or worse than the human's. Grading one comparison took over an hour on average. For the paper's headline figures each model was sampled three times per task and each sample was graded by three experts, giving nine comparisons per task. OpenAI reports three variants of the metric: wins or ties, clear wins only, and a no-ties figure; for GPT-5.2 Thinking these were 70.9%, 49.8% and 61.0% (Source: openai.com). The paper notes that blinding was imperfect because model outputs carry stylistic tells, such as frequent em dashes in OpenAI outputs and first-person phrasing in Claude outputs (GDPval Paper (OpenAI, Oct 2025)).

OpenAI also trained an experimental automated grader, based on GPT-5 at high reasoning effort, that agreed with human graders 65.7% of the time against 70.8% agreement between two humans. Twelve of the 220 public tasks could not be graded automatically, for reasons including internet access requirements and non-Python code. The paper reports that the grader correlated less well with humans on outputs from OpenAI's own stronger models, which it links to evidence that models favor their own responses (GDPval Paper (OpenAI, Oct 2025)). The grading service at evals.openai.com has since been withdrawn; OpenAI released the rubrics and gold deliverables and states that pairwise human expert preference remains the standard, with a model judge giving only a rough estimate (Source: evals.openai.com).

Speed and cost analysis

The paper compares model and expert time and cost under three scenarios. A naive ratio of expert time to API response time made GPT-5 90 times faster and 474 times cheaper than an expert. When an expert must review each output and redo the task if it falls short, the advantage for GPT-5 shrank to 1.12 times faster and 1.18 times cheaper with one attempt, and 1.39 times faster and 1.63 times cheaper with repeated attempts; GPT-4o was slower and more expensive than an unaided expert under both. The authors add that the analysis does not price catastrophic errors. A separate review of GPT-5's losing outputs found most were acceptable but weaker than the human's, roughly 29% were bad or catastrophic, and about 3% catastrophic (GDPval Paper (OpenAI, Oct 2025)).

Versions and variants

• Full set and gold subset. The full set has 1,320 tasks, 30 per occupation. The public gold subset has 220, five per occupation, with prompts and reference files on Hugging Face; published scores from OpenAI and third parties are on the gold subset (GDPval Paper (OpenAI, Oct 2025)).
• GDPval-AA. Artificial Analysis runs the 220 public tasks through its own open-source agent harness, Stirrup, which gives every model a sandbox with shell access, web search and web fetch. Pairs of model submissions for the same task are compared blind by a model judge, and the results are fitted to a Bradley-Terry model to give an Elo rating (a scale on which a 400-point gap corresponds to 10:1 odds of being preferred). It was added to the Artificial Analysis Intelligence Index in January 2026 with a single Gemini judge. GDPval-AA v2, introduced in June 2026, moved to a panel of three judges from different developers, anchored the scale so that the human expert deliverables sit at 1,000, raised the turn limit to 250 and let models abandon a task. In September 2026 the judge panel was changed to Claude Opus 5, GPT-5.6 Sol and Gemini 3.8 Flash (Source: artificialanalysis.ai).
• GDPval Extended. Mercor publishes a leaderboard that scores models on 213 public GDPval tasks and on 132 additional tasks written by its own experts and held out from the public; the page reports a mean score and does not describe the grading method (Source: mercor.com).
• Under-contextualized GDPval. An experiment in the paper cut prompts to 42% of their original length by removing guidance on where to find data and how to format the result; GPT-5 performed worse, which the authors attribute to difficulty working out context (GDPval Paper (OpenAI, Oct 2025)).

Development and maintainers

The paper lists 19 OpenAI authors led by Tejal Patwardhan, with Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh and Phoebe Thacker as equal contributors; its acknowledgements thank the vendor partners and the industry experts who wrote and graded the tasks (GDPval Paper (OpenAI, Oct 2025)). OpenAI said in the announcement that it planned further versions with more occupations and interactive tasks and invited professionals and customers to contribute (Source: openai.com). OpenAI publishes human-graded results in its own launch posts, including figures for competitors' models that it ran itself (Source: openai.com). OpenAI's hosted leaderboard now states that it is no longer active and directs readers to the Artificial Analysis leaderboard (Source: evals.openai.com).

Results

In the paper, GPT-4o won or tied 12.5% of comparisons and Claude Opus 4.1, the best model tested, 47.6%; OpenAI described the progression across its own models as roughly linear over time (GDPval Paper (OpenAI, Oct 2025)). OpenAI reported 70.9% for GPT-5.2 Thinking in December 2025, calling it the company's first model at or above expert level on the benchmark, then 83.0% for GPT-5.4 in March 2026 and 84.9% for GPT-5.5 in April 2026 (Source: openai.com; openai.com; openai.com). On GDPval-AA, Anthropic reported Claude Opus 4.6 about 144 Elo points ahead of GPT-5.2 in February 2026, which its system card translates to a pairwise win rate near 70%, and Opus 4.7 about 79 points ahead of GPT-5.4 in April 2026, a win rate near 61.2% (Claude Opus 4.6 System Card; Claude Opus 4.7 System Card; Source: anthropic.com). Elo figures quoted on different dates differ for the same model, and over the period Artificial Analysis added models, moved from v1 to v2 and changed the judge panel: Claude Fable 5 appears as 1932 on v1 in early June 2026, 1783 on v2 in mid-June, 1723 in Anthropic's September 1 table and 1631 on the leaderboard on September 19.

Wins or ties vs. industry expert

Human expert graders, 220-task gold subset. All figures are reported by OpenAI, including those for other developers' models.

GDPval-AA v2 Elo

Artificial Analysis's implementation: common Stirrup harness, three-model judge panel, human expert deliverables fixed at 1,000. Leaderboard rows are as retrieved on the date shown; rows reported by developers reproduce the leaderboard as it stood when they published.

GDPval-AA v1 Elo

The original Artificial Analysis implementation (January to June 2026): single Gemini judge, no human anchor. Ratings are not comparable with v2.

Mercor GDPval Extended

Mercor's mean score on its 132 held-out tasks; the grading method is not described on the leaderboard page. The same page gives each model's score on the 213 public tasks, quoted in the Setting column.

GDPval-AA

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GDPval-AA — Elo (version not stated)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GDPVal-AA v2 — General agentic

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GDPVal (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

A wins-or-ties rate counts the share of pairwise judgments, not the share of tasks a model can do unaided, and the comparison deliverable is one expert's attempt and not a perfect answer. Epoch AI analysts Florian Brand and Greg Burnham treat wins plus ties as the most informative variant because the ceiling on outright wins is unclear: on some tasks there may be little room to improve on the human reference What do 'economic value' benchmarks tell us? (Brand and Burnham, Epoch AI, 2026). The paper reports human graders agreeing with each other 70.8% of the time, which limits how finely close scores can be separated (GDPval Paper (OpenAI, Oct 2025)).

Scores depend on the harness. In the paper, OpenAI models ran with web search and a code interpreter, Claude was sampled through its consumer interface to use its file-creation feature, and the setup for Gemini and Grok is not described; a prompt telling GPT-5 to render and inspect its own files raised its score by about five percentage points, and more reasoning effort raised scores for o3 and GPT-5 (GDPval Paper (OpenAI, Oct 2025)). OpenAI's GPT-5.3-Codex figure was obtained with custom skills (Source: openai.com).

A GDPval-AA Elo is a different quantity. It compares models with each other under one harness and is judged by models, not by professionals; in v1 it had no human anchor, and Epoch AI noted that it therefore could not yield an absolute wins-or-ties figure What do 'economic value' benchmarks tell us? (Brand and Burnham, Epoch AI, 2026). Version 2 places the human deliverables at 1,000. A model's rating moves when the pool or the judges change, so a figure in a launch post describes the leaderboard on that day (Source: artificialanalysis.ai).

The speed and cost ratios in OpenAI's posts ("roughly 100x faster and 100x cheaper" at release; more than 11 times the speed and under 1% of the cost for GPT-5.2) use inference time and API prices only. OpenAI states that they exclude the oversight, iteration and integration that real use requires, and the paper's own review-and-fix scenarios reduce the gain to between 1.1 and 1.6 times for GPT-5 (GDPval Paper (OpenAI, Oct 2025); Source: openai.com; openai.com).

Limitations and criticism

The paper lists its limits: 44 occupations at 30 tasks each is a first cut; physical work, tacit knowledge, proprietary software, personal data and communication between people are excluded; tasks are one-shot and fully specified; the automated grader falls short of human graders; and expert grading is expensive (GDPval Paper (OpenAI, Oct 2025)).

Epoch AI's review of GDPval, the Remote Labor Index and APEX-Agents describes GDPval's realism as doubly limited, because tasks are self-contained and there is no cluttered working environment to navigate, and calls it the closest of the three to saturation. It also identifies a reliability problem specific to GDPval: more than half the tasks depend on live web research, two public tasks already pointed to sources that no longer resolved, and about 5% are framed from a point in time that has since passed, which may make them easier in hindsight. The authors conclude that a high score would indicate a shift toward delegating discrete tasks to AI, comparable to the effect of coding agents on software engineering, and not the automation of whole occupations What do 'economic value' benchmarks tell us? (Brand and Burnham, Epoch AI, 2026). Epoch AI's Anson Ho wrote in February 2026 that models had passed the human baseline within months of release while measured economic effects remained muted, so the benchmark was falling short as a leading indicator of automation (Source: epoch.ai).

Ethan Mollick, writing about the release results, reported that the models beat the human experts in software development and financial advice and lost in pharmacology and real estate, and that the main reason models lost was a failure to format results well or follow instructions exactly, not hallucination (Source: oneusefulthing.org; oneusefulthing.org). The Center for AI Safety cites the five-point gap between GDPval's automated grader and human agreement in arguing that human judges remain necessary for this class of benchmark (Source: safe.ai). The Remote Labor Index paper characterizes GDPval as showing near-parity on kinds of tasks shared across many professions, which indicates potential for augmentation but does not measure end-to-end project automation (Remote Labor Index: Measuring AI Automation of Remote Work (Mazeika et al., 2025)).

Epoch AI's review adds that little is known about how the head-to-head comparisons are carried out after the original paper: the number of judges used in later evaluations is not stated, nor what is done to have experts judge as an economically motivated customer would and not on surface characteristics What do 'economic value' benchmarks tell us? (Brand and Burnham, Epoch AI, 2026).

Use in policy and safety frameworks

GDPval is reported as a capability measure. GDPval-AA appears in the capabilities chapters, not the safety assessments, of Anthropic's system cards from Claude Opus 4.6 onward (Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card; Claude Opus 4.7 System Card; System Card: Claude Opus 4.8 (Anthropic, May 2026); System Card: Claude Fable 5 & Claude Mythos 5 (Anthropic, June 2026); System Card: Claude Sonnet 5 (Anthropic, June 2026); System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026)) and carries 10% of the weight of the Artificial Analysis Intelligence Index, itself quoted in OpenAI's and xAI's launch posts (Source: artificialanalysis.ai; openai.com; x.ai). In labor-market debate the benchmark is cited as evidence of task-level capability, set against findings that enterprise deployments rarely show measurable returns (Enterprise AI Deployment Gap) and against the low end-to-end automation rates on the Remote Labor Index (AI Labor Disruption).

Relationships

• depends-on: GDPval Paper (OpenAI, Oct 2025) — the introducing paper.
• instance-of: AI Benchmarks and Evaluation — an expert-graded, occupation-based benchmark.
• supports: AI and Productivity — task-level speed, cost and quality comparisons with professionals.
• related: AI Labor Disruption, Enterprise AI Deployment Gap, Jagged Frontier — how task-level results are read in labor-market arguments.
• related: Remote Labor Index, APEX-Agents, SWE-Lancer — other benchmarks that price or professionally grade work.
• related: Artificial Analysis Intelligence Index — includes GDPval-AA v2 as a component.
• related: OpenAI, Epoch AI, Ethan Mollick.

See also

• METR Time Horizons — the paper finds that win rates are highest on tasks of up to two hours and fall as expert completion time grows (GDPval Paper (OpenAI, Oct 2025)); METR's time-horizon work measures that relationship directly (Measuring AI Ability to Complete Long Software Tasks).
• MarketBench: Evaluating AI Agents as Market Participants — Fradkin & Krishnan (April 2026) — MarketBench, which tests whether agents can price and bid for work and not only complete it.
• Finance Agent, HealthBench — single-profession counterparts.