GDPval is a benchmark introduced in an OpenAI technical report released on October 5, 2025 that evaluates frontier AI models on 1,320 real-world, economically valuable tasks spanning 44 occupations across 9 GDP-weighted U.S. sectors. Tasks are built by industry professionals averaging 14 years of experience and graded via blinded expert pairwise comparison. The paper reports that frontier model performance improves roughly linearly over successive generations, with Claude Opus 4.1 and GPT-5 approaching expert quality at roughly 100× the speed and 100× the cost reduction under a naive comparison that does not adjust for quality.
Overview
The report was authored by a large OpenAI team (Patwardhan, Dias, Proehl et al., including Glaese and Tworek). It is framed as a response to the rapid saturation of traditional benchmarks (AI Benchmarks and Evaluation) and to the observation that few benchmarks measure "writing, business judgment, or empathy" — the work people actually do (Source: oneusefulthing.org).
The benchmark targets knowledge work — computer-based tasks that produce concrete deliverables such as briefs, blueprints, care plans, analyses, spreadsheets, and presentations. It is explicitly scoped to knowledge work and does not cover manual labor, tacit knowledge, or interactive/iterative workflows.
Methodology
Task selection and GDP weighting
The 9 sectors were chosen as the largest contributors to U.S. GDP, covering roughly $3 trillion in annual earnings: Real Estate & Rental/Leasing (13.8%), Government (11.3%), Manufacturing (10.0%), Professional/Scientific/Technical Services (8.1%), Health Care & Social Assistance (7.6%), Finance & Insurance (7.4%), Retail Trade (6.3%), Wholesale Trade (5.8%), and Information (5.4%). Within these sectors the paper covers 44 occupations (roughly 5 per sector), aligned with ONET and BLS Work Activities. It comprises 1,320 tasks total (≥30 per occupation), with a 220-task "gold subset" released open-source. Coverage spans 208 unique ONET tasks, 25 of 35 ONET skills, and 26 of 41 ONET work activities.
Reference deliverables
Tasks are constructed by industry experts (average 14 years' experience; under 10% of applicants selected) and include a request, optional reference files, and a reference deliverable the expert produced. Gold-subset tasks average 9.49 hours of human completion time and $398 in dollar value; 67.7% require interaction with reference files.
Quality control
Each task received an average of five human expert reviews (minimum three) across three stages: generalist initial review, occupation-specific expert review, and final iterative review. Automated model-based screening flagged issues but experts retained final authority. The paper reports 89.07% of gold-subset tasks were judged well-specified.
Occupation coverage
The 44 occupations span familiar knowledge-work targets (Software Developers, Lawyers, Accountants, Financial Analysts, Project Managers, Editors, Registered Nurses) and also include operational and frontline-supervisory roles (First-Line Supervisors of Production, First-Line Police Supervisors, Pharmacists, Real Estate Brokers, Order Clerks, Audio & Video Technicians). This breadth distinguishes GDPval from prior coding-heavy benchmarks and partially addresses the "jagged frontier" concern that benchmarks skew toward easily automated knowledge work (AI Benchmarks and Evaluation).
Grading protocol
The primary grading method is blinded expert pairwise comparison: experts rank two or more unlabeled deliverables for the same task, scored as {0 = human preference, 0.5 = tie, 1 = model preference}. Average grading time per comparison exceeds 1 hour, with 3 graders per sample per model.
An automated grader, based on GPT-5-high, was also used. Human-automated agreement was 66% against human inter-rater agreement of 71% — within 5 percentage points of human inter-rater agreement — and is available publicly at evals.openai.com. Of the 220 gold-subset tasks, 12 were ungradable owing to constraints such as required internet access, a Python-only environment, font packages, and speech-to-text limits.
Results by model
Win rates on the gold subset (higher means more often preferred to the human expert deliverable):
| Model | Win rate | Notes |
|---|---|---|
| Claude Opus 4.1 | 47.6% | Best overall; effectively tied with human experts. Excels at aesthetics; best on .pdf, .xlsx, .ppt. |
| GPT-5 | 39.0% | Best on instruction-following and calculations; fewest instruction-following failures. Best on pure text outputs. |
| o3 | 35.2% | |
| o4-mini | 29.1% | |
| GPT-4o | 12.5% | |
| Gemini 2.5 Pro, Grok 4 | lower | Frequently promised but failed to provide deliverables, ignored reference data, used wrong formats. |
OpenAI frontier model performance on the gold subset improves roughly linearly over successive generations. Win rates are highest on 0–2 hour tasks and decline as task duration grows, consistent with METR's finding that the time horizon of reliably autonomous AI work is finite and expanding (Source: Measuring AI Ability to Complete Long Software Tasks). Near-parity with humans is observed in Government, Retail Trade, and Wholesale Trade.
Common failure modes across all models are data hallucination and miscalculation. Claude, Grok, and Gemini most often lose on instruction-following failures.
Cost and time analysis
The paper reports three scenarios. Under the naive ratio (no quality adjustment), GPT-5 is 90× faster and 474× cheaper than an expert; o3 is 161× faster and 480× cheaper; o4-mini is 186× faster and 1,265× cheaper; and GPT-4o is 327× faster and 5,172× cheaper. The headline "~100× speed and ~100× cost" summarizes this naive comparison for the top frontier models.
The paper is explicit that quality-adjusted scenarios — in which humans finish tasks the model gets wrong — shrink the gains substantially. Under "try once, then fix," GPT-5 gives 1.12× faster and 1.18× cheaper, while GPT-4o is slower and more expensive than unaided experts. Under "try N times, then fix," GPT-5 gives 1.39× faster and 1.63× cheaper, and GPT-4o remains a net negative.
The naive ratios coexist with these quality-adjusted figures: once win rate and the need for human repair are accounted for, the realized advantage in the paper's scenarios narrows to roughly a 10–60% speed and cost improvement (AI and Productivity). The contrast bears on policy framing that positions AI adoption as pure substitution — see (Source: epoch.ai) and AI Labor Disruption.
Performance enhancement experiments
Several experiments tested how performance responds to additional effort and scaffolding. Varying reasoning effort for o3 and GPT-5 at Low, Medium, and High settings produced predictable improvements, a real-world replication of the test-time compute scaling signal (Scaling Laws). A custom multimodal prompt eliminated black-square PDF artifacts from GPT-5 (previously over 50% of PDFs), cut egregious PowerPoint formatting errors from 86% to 64%, raised multimodal inspection usage from 15% to 97%, and added roughly 5 percentage points to win rates. Best-of-4 sampling with GPT-5 as judge further improved performance.
Limitations
The paper notes several limits. It covers only 44 occupations and 30 tasks per occupation, a limited initial slice of knowledge work rather than a comprehensive labor-market evaluation. It is restricted to computer-based knowledge work and excludes manual labor, physical tasks, extensive tacit knowledge, PII-handling, proprietary software, and inter-individual communication. Tasks are precisely specified and one-shot rather than interactive, whereas real work requires iterative context discovery; GDPval hands models the full context upfront. The automated grader has meaningful gaps relative to human experts, and construction is expensive, with industry-expert graders limiting scalability.
These limitations echo concerns in (Source: oneusefulthing.org) and (Source: oneusefulthing.org) that benchmark performance on precisely-specified tasks overstates real-world readiness.
Relationships
- supports: AI Benchmarks and Evaluation — adds a real-world, economically weighted benchmark to the account of post-saturation evaluation.
- supports: AI and Productivity — provides quantitative, occupation-level productivity multiples for frontier models on expert-graded deliverables, while also documenting how quality-adjusted gains narrow to single-digit multipliers.
- related: (Source: oneusefulthing.org) — Mollick's commentary on what AI agents can actually deliver in professional workflows.
- related: (Source: oneusefulthing.org) — Mollick's earlier commentary on benchmark displacement and GDPval framing.
- related: (Source: epoch.ai) — Moravec's-Paradox framing of where AI displaces knowledge work first.
- related: Measuring AI Ability to Complete Long Software Tasks — the time-horizon finding complements GDPval's observation that win rates fall with task duration.