ARC-AGI — the Abstraction and Reasoning Corpus for Artificial General Intelligence — is a series of benchmarks designed to measure how efficiently a system acquires skills on tasks it has not seen before, rather than how well it performs tasks it has been trained on. It was introduced by François Chollet in the 2019 paper "On the Measure of Intelligence" and has since been maintained by the ARC Prize Foundation, which publishes the benchmark datasets, runs verification of reported scores, and administers the annual ARC Prize competitions (Source: arcprize.org; On the Measure of Intelligence (Chollet, 2019)). The first two generations present static puzzles on coloured grids; the third, released in March 2026, places an agent in interactive environments with no instructions. Each generation was built after frontier systems began scoring well on the one before, and by September 2026 the foundation had verified scores above 95% on the first two generations and 62.7% to 99.9%, depending on the test harness, on the third (Source: arcprize.org; arcprize.org).
What it measures
Definition of intelligence
The series rests on a specific definition drawn from the 2019 paper: "The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty." The ARC Prize Foundation restates this as a working definition of artificial general intelligence — "a system that can match the learning efficiency of humans" — and sets it against the alternative formulation of AGI as a system able to automate the majority of economically valuable work, which the foundation describes as a useful goal but an incorrect measure of intelligence (Source: arcprize.org).
The reasoning given for the distinction is that task-specific skill is heavily influenced by prior knowledge and training data, so unlimited priors allow developers to buy levels of skill for a system and mask its own generalization power. On that account a benchmark that measures skill measures the developer's encoding of knowledge rather than the system's capacity to learn (Source: arcprize.org).
Core knowledge priors
ARC-AGI tasks are restricted to what the foundation calls core knowledge priors — cognitive building blocks present at birth or acquired very early in human development with minimal explicit instruction, following the Core Knowledge theory associated with Elizabeth Spelke. The stated rationale is twofold: restricting the inputs to universally available cognitive primitives isolates generalization from domain-specific knowledge, and it puts human and machine test-takers on comparable footing. A benchmark including tasks in written English, by this argument, would advantage systems pre-trained on large text corpora and disadvantage humans who do not read English, because language is a cultural artifact rather than a measure of cognitive ability (Source: arcprize.org). The ARC Prize 2024 technical report lists the assumed priors as concepts such as objectness, basic topology and elementary integer arithmetic, which it says children acquire typically before age four ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024).
The design principle the foundation states for the series is "easy for humans, hard for AI" — a deliberate contrast with benchmarks built around graduate-level or specialist problems such as GPQA or Humanity's Last Exam, on the reasoning that tasks humans solve effortlessly and machines do not are the ones that expose gaps in reasoning and adaptability (Source: arcprize.org). The foundation describes the purpose of the series as measuring the "residual gap" between current AI and AGI, which it defines as "a system's ability to acquire any skill a human can, as efficiently as a human can" (Source: arcprize.org). Chollet has said that the name does not make the benchmark a test for AGI: "ARC-AGI is not an acid test for AGI…It's a research tool designed to focus attention on the most challenging unsolved problems in AI" (Source: aiguide.substack.com).
Design
Task format in ARC-AGI-1 and ARC-AGI-2
Each task shows the test-taker a small number of demonstration pairs, typically two to five, in which an input grid of coloured cells is transformed into an output grid according to an unstated rule. The test-taker must infer the rule and produce the correct output grid for one or more new inputs. Grids are at most 30×30 cells and use up to 10 colours, and every task follows a different rule, so that it "should not be possible to prepare for any of the tasks in advance" (Source: arxiv.org; ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024)). Melanie Mitchell's description of a typical task is a challenge "to figure out the common abstract transformation rule from a small number of demonstrations of transformations between grids, and to apply that rule to a new grid" (Source: aiguide.substack.com). Language models receive the grids as arrays of integers rather than images (Source: arcprize.org).
ARC-AGI-2 tasks were written to require what the foundation calls symbolic interpretation (treating a symbol as having meaning beyond its visual pattern), compositional reasoning (applying several interacting rules at once) and contextual rule application (applying a rule differently depending on context) (Source: arcprize.org).
Scoring in ARC-AGI-1 and ARC-AGI-2
A task counts as solved only if the submitted output grid matches the answer exactly for every test input, and the test-taker is allowed two attempts per task, a rule often written pass@2. The score is the percentage of tasks solved (Source: arxiv.org; arcprize.org; aiguide.substack.com). Since ARC-AGI-2 every verified result is reported together with its cost per task, on the foundation's stated view that "intelligence is about finding the solution efficiently, not exhaustively"; verified runs are capped at $10,000 per run and use a single run without averaging (Source: arcprize.org; arcprize.org).
Scoring in ARC-AGI-3
ARC-AGI-3 scores agents on action efficiency against a human baseline rather than on raw solve rate, which distinguishes it from the pass/fail scoring of the earlier generations. The metric, Relative Human Action Efficiency (RHAE), scores each level as min(1.15, (h/a)²) where a is the agent's action count and h the upper-median best first-run human count, weights levels linearly within an environment so the tutorial contributes least, and caps the environment score at the weighted fraction of levels completed. Squaring the ratio means an agent taking ten times a human's actions receives 1% rather than 10% credit (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence). An action is one turn that changes the environment; a model's internal reasoning steps and tool calls are not counted, and agents are stopped after five times the human-baseline action count on a level (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence).
An environment is a small turn-based game on a 64×64 grid in 16 colours, with at least six levels and a deliberately easy first level, in which the agent is never told the objective (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence). ARC Prize describes one semi-private environment, tu93, as "a maze-like game with guards and moving patrols" (Source: arcprize.org).
Public and held-out sets
Each generation is divided into public sets and held-out sets, because, in the foundation's words, "the test taker must not know what the test will be." A semi-private set is used to test commercial models through their developers' APIs; the foundation requires zero-data-retention agreements from providers but calls the set semi-private because tasks sent to external APIs may leak over time. A fully private set is used only for the Kaggle competitions, where submissions run offline. The foundation's stated defences against accumulated exposure are those agreements, the release of a new benchmark version roughly every year, and monitoring of the gap between public and semi-private scores (Source: arcprize.org).
| Generation | Public | Semi-private | Private |
|---|---|---|---|
| ARC-AGI-1 | 400 training tasks and 400 evaluation tasks | 100 tasks, introduced mid-2024 | 100 tasks |
| ARC-AGI-2 | 1,000 training tasks and 120 evaluation tasks | 120 tasks | 120 tasks |
| ARC-AGI-3 | 25 demonstration environments | 55 environments | 55 environments |
(Source: arcprize.org; arcprize.org; ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence.) ARC-AGI-3's split inverts the roughly 10:1 public-to-private ratio of ARC-AGI-2, and the report states that its public set is a demonstration of the format rather than a training resource and that the private sets are deliberately out-of-distribution relative to it (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence).
Human baselines
The original ARC-AGI-1 private tasks were checked by two people who scored 97% and 98% and between them solved every task, but no first-party human baseline under consistent conditions existed for the first generation, which the ARC-AGI-2 paper lists among its limitations (Source: arxiv.org; ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024)). An independent study at New York University tested 1,729 people on the 800 public tasks and estimated average accuracy at 76.2% on the training set and 64.2% on the public evaluation set, with 790 of the 800 tasks solved by at least one person within three attempts H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark (LeGris et al., 2024). The ARC Prize leaderboard lists 98% for a human panel, 98% for STEM graduates and 77% for average Mechanical Turk workers on ARC-AGI-1 (Source: arcprize.org).
For ARC-AGI-2 the foundation tested candidate tasks on 407 members of the public in 515 sessions between November 2024 and May 2025, paying $115–150 per 90-minute session plus $5 per correct task. A task was kept only if at least two participants solved it within two attempts; across all attempts 66% of test pairs were solved, and no recorded demographic factor, including programming or mathematics background, showed a clear relationship with performance (Source: arxiv.org). For ARC-AGI-3, ten members of the public were tested on each candidate environment and an environment was kept only if at least two solved it in full on first exposure; the report records 486 participants and 2,893 attempts (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence).
Versions and variants
ARC-AGI-1 (2019)
The original corpus of grid-puzzle tasks, presented in "On the Measure of Intelligence" as a test of fluid intelligence rather than domain knowledge, with public training and public evaluation sets and a held-out private evaluation set On the Measure of Intelligence (Chollet, 2019); the semi-private set used for verification was added by the ARC Prize competition (ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024)). The benchmark was originally called the Abstraction and Reasoning Corpus (ARC) and was renamed ARC-AGI to avoid collision with other benchmarks of the same name ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024).
ARC-AGI-2 (2025)
Described in a 2025 paper by Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers and Henry Pinkard (arXiv 2505.11831, revised January 2026), which retains the grid-puzzle format while curating a new dataset intended to resist the pattern-matching and brute-force approaches that had begun scoring well on ARC-AGI-1. Human baselines were established for the new set. The paper gives the reasons for replacement as the susceptibility of nearly half the ARC-AGI-1 private tasks to brute-force program search, the lack of first-party human data, uneven difficulty between subsets, and information leakage from an estimated 10,000 scores disclosed against the same 100 private tasks over four competitions; tasks solved in the 2020 contest were removed and the evaluation sets were calibrated so that human accuracy differs by no more than one percentage point between them (Source: arxiv.org; arcprize.org).
ARC-AGI-3 (2026)
An interactive benchmark, credited to the ARC Prize Foundation and posted as arXiv 2603.24621 on March 24, 2026 and revised April 17, 2026, alongside a public launch on March 25, 2026. It departs from the static grid-puzzle format of the first two generations: agents are placed in novel, abstract, turn-based environments and must explore, infer goals without explicit instruction, build internal models of environment dynamics, and plan action sequences. Like its predecessors it uses only Core Knowledge priors and avoids language and external knowledge, and its environments are difficulty-calibrated through testing with human subjects. The paper reports that humans solve 100% of the environments while frontier AI systems, as of March 2026, scored below 1%, and describes the benchmark as the only unsaturated general agentic-intelligence benchmark known to its authors at that date (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence; Source: arcprize.org).
Development and maintainers
Chollet created the first corpus while a researcher at Google (Source: arcprize.org). The ARC Prize Foundation, co-founded by Chollet and Mike Knoop in 2024 and led by president Greg Kamradt, is a non-profit funded by donations from individuals, foundations and AI laboratories. Its testing policy states that sponsors receive no access to held-out data and no influence over what is tested or when results are published, that board members and staff with an interest in a laboratory recuse themselves from decisions about it, and that the methodology is reviewed by an independent academic panel of Todd Gureckis of New York University, Melanie Mitchell of the Santa Fe Institute and Vishal Misra of Columbia University, which reviews the approach but not individual results (Source: arcprize.org). The organization's history and staff are set out at ARC Prize Foundation.
Verified and community leaderboards
The foundation runs frontier-model evaluations itself, using open-source benchmarking code, against the semi-private sets, and publishes model outputs, costs and per-task scores. It tests models from commercial APIs with more than $10 million in monthly gross revenue, a threshold it justifies as ensuring "sufficient commercial generalization pressure against benchmark overfitting," and publishes results no later than 30 days after a model's public release. Scores that developers quote as "ARC Prize Verified" come from this process. A separate community leaderboard carries self-reported results that are not verified by default; an open-source submission may be verified if it runs within $10,000 and 12 hours and its public and semi-private scores agree within a stated tolerance (±10 percentage points for ARC-AGI-1, ±3 for ARC-AGI-2, ±15 for ARC-AGI-3) (Source: arcprize.org).
The harness question in ARC-AGI-3
A harness is the software placed around a model to turn it into an agent: it decides what the model sees each turn, what it remembers and which tools it can call. The ARC-AGI-3 technical report states the foundation's position on harness-assisted and public-set results directly: it "will never report public set scores of any system on the official leaderboard," and evaluating on the public set "is emphatically not a valid measure of progress towards AGI." To show that a perfect public score is achievable by construction, the foundation released an open-source harness that scores 100% on all public environments using human replay. The official leaderboard therefore uses no harness at all. The report also records a bimodality result from early testing: in a variant of environment TR87, Opus 4.6 scored 0.0% with no harness and 97.1% with a harness built by a Duke University team, while in environment BP35 it scored 0.0% under both — which the authors read as evidence that harness performance on seen environments does not transfer to unseen ones (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence).
The question of which settings count as a harness was reopened in July 2026. OpenAI published a post on July 29 attributing GPT-5.6 Sol's 7.8% leaderboard result to configuration rather than capability: running the official harness on the public set it measured 13.3%, and with retained reasoning and compaction enabled through its Responses API, 38.3%, which it described as roughly three times the score at six times fewer output tokens (Source: openai.com). Chollet responded on July 30 that harnesses "custom-made to solve the benchmark" are off limits while general-purpose API settings "available to all API users" are permissible, acknowledging a "potential parity issue" he considers acceptable "as long as the settings and the cost are clearly reported" (Source: the-decoder.com).
Since September 2026 the foundation has reported ARC-AGI-3 results under two labelled conditions. The Standard harness is a minimal, provider-neutral interface that lets a model carry forward notes it chooses to keep. The Provider Adapter harness uses context-management features designed by the model's provider, such as preserving opaque reasoning state between requests and compacting long conversations. The foundation says the two "answer different evaluation questions" and that it believes "a future AGI should be able to solve ARC-AGI-3" under the Standard conditions (Source: arcprize.org; arcprize.org).
ARC Prize competitions
The ARC Prize competitions, run annually since 2024, attach prize money to progress on the current-generation benchmark and publish verified results. Entries run on Kaggle without internet access against the private set, and prize eligibility requires open-sourcing the solution; the grand prize requires 85% and has not been claimed (Source: ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024); arcprize.org). ARC Prize 2026 was launched on March 25, 2026 with more than $2 million in prizes, with the ARC-AGI-3 competition hosted on Kaggle (Source: arcprize.org; arcprize.org).
Results
The first Kaggle competition in 2020 was won with a score of about 20% on the ARC-AGI-1 private set using brute-force program search, and the best competition score had reached only 33% by the end of 2023. During ARC Prize 2024 the team MindsAI reached 55.5% and the open-source winner, the ARChitects, 53.5%, both by fine-tuning a model on each task at test time ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024). Outside the competition, Ryan Greenblatt reported 50% on the public evaluation set in 2024 by having GPT-4o write thousands of candidate Python programs per task, an approach that scored 43% on the ARC Prize public leaderboard (Source: blog.redwoodresearch.org; arcprize.org). General-purpose language models scored far lower: ARC Prize lists GPT-4o at 4.5% and measured o1-preview at 21.2% on the public evaluation set in September 2024 (Source: arcprize.org; arcprize.org). On December 20, 2024 ARC Prize reported that a preview of OpenAI's o3, trained on 75% of the public training set, scored 75.7% on the semi-private set within the $10,000 limit and 87.5% in a configuration using roughly 172 times the compute (Source: arcprize.org). ARC-AGI-2 launched in March 2025 with every frontier model below 5%, a level the paper treats as noise (Source: arxiv.org); ARC Prize 2025 was won by NVIDIA's NVARC team at 24.03% on the private set at $0.20 per task, and by December 2025 the best verified commercial model stood at 37.6% (Source: arcprize.org). Verified ARC-AGI-2 scores then rose to 77.1% in February 2026, 85.0% in April, 92.5% in July and 95.0% in September 2026 (Source: deepmind.google; openai.com; arcprize.org; arcprize.org). ARC-AGI-3 followed a shorter course: official semi-private scores at release in March 2026 were 0.50% for Anthropic's Opus 4.6 (Max), 0.40% for Gemini 3.1 Pro Preview, 0.20% for GPT 5.4 (High) and 0.10% for Grok-4.20, all run without a harness under a single shared system prompt (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence); the first scores above 1% came in mid-2026, and on September 3, 2026 ARC Prize reported 62.7% for GPT-6 Astra under its Standard harness and 99.9% under the Provider Adapter harness (Source: arcprize.org).
In the tables below, scores and costs are as listed on the ARC Prize verified leaderboard on 2026-09-19 unless another source is given. The label in parentheses after a model name in the Setting cell is the reasoning-effort or thinking-budget setting ARC Prize ran. The Date is that of the ARC Prize or developer post reporting the result where one exists, and otherwise the model's release date.
ARC-AGI-1 semi-private
Percentage of the 100 semi-private ARC-AGI-1 tasks solved with two attempts per task.
| Model | Score | Setting | Date | Source |
|---|---|---|---|---|
| GPT-6 Astra | 98.5% | xhigh reasoning; $0.347 per task; ARC Prize verified | 2026-09-03 | (Source: arcprize.org) |
| Claude Fable 5.1 | 97.5% | max effort; $1.40 per task; ARC Prize verified | 2026-09-01 | (Source: arcprize.org); System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026) |
| Claude Fable 5 / Mythos 5 | 98.5% | Table 8.1.A column headed "Claude Fable 5/Mythos 5" (Figure 8.16.A labels the series Claude Fable 5); ARC Prize Foundation; effort not stated (card §8.1 Table 8.1.A) | 2026-09-01 | System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026) |
| GPT-5.6 Sol | 96.5% | ARC Prize Foundation; reported by Anthropic in the Claude Fable 5.1 & Mythos 5.1 system card; effort not stated (card §8.1 Table 8.1.A) | 2026-09-01 | System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026) |
| DeepSeek V4 Pro 0813 | 90.0% | max reasoning; $0.296 per task; ARC Prize verified | 2026-08-13 | (Source: arcprize.org) |
| Inkling-Small | 84.0% | evaluation set not stated in the card; the Inkling and Inkling-Small values equal the ARC Prize verified semi-private figures on evaluations/arc-agi; developer-reported (card §5. Evaluations) | 2026-07-30 | Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| Claude 4.5 Haiku | 47.7% | evaluation set not stated in the card; the Inkling and Inkling-Small values equal the ARC Prize verified semi-private figures on evaluations/arc-agi; reported by Thinking Machines Lab in the Inkling-Small model card (card §5. Evaluations) | 2026-07-30 | Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| GPT 5.6 Luna | 87.7% | evaluation set not stated in the card; the Inkling and Inkling-Small values equal the ARC Prize verified semi-private figures on evaluations/arc-agi; reported by Thinking Machines Lab in the Inkling-Small model card (card §5. Evaluations) | 2026-07-30 | Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| Claude Opus 5 | 97.5% | high effort; $0.470 per task; ARC Prize verified | 2026-07-24 | (Source: arcprize.org); System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026); System Card: Claude Opus 5 (Anthropic, July 2026) |
| Kimi K3 | 94.5% | max reasoning; $0.770 per task; ARC Prize verified | 2026-07-16 | (Source: arcprize.org) |
| Inkling | 79.5% | $0.300 per task; ARC Prize verified | 2026-07-15 | (Source: arcprize.org); Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| GPT-5.6 Sol | 97.5% | xhigh reasoning; $0.400 per task; ARC Prize verified | 2026-07-09 | (Source: arcprize.org); System Card: Claude Opus 5 (Anthropic, July 2026) |
| GLM-5.2 | 77.0% | $0.192 per task; ARC Prize verified | 2026-06-16 | (Source: arcprize.org) |
| Claude Fable 5 | 98.5% | xhigh effort; $1.02 per task; ARC Prize verified | 2026-06-09 | (Source: arcprize.org) |
| Claude Opus 4.8 | 92.5% | max effort; $2.33 per task; ARC Prize verified | 2026-05-28 | (Source: arcprize.org); System Card: Claude Opus 5 (Anthropic, July 2026) |
| GPT-5.5 | 95.0% | xhigh reasoning; $0.730 per task; ARC Prize verified, reported in OpenAI's launch table | 2026-04-23 | (Source: openai.com) |
| Claude Opus 4.7 | 93.5% | high effort; $1.41 per task; ARC Prize verified | 2026-04-16 | (Source: arcprize.org); Claude Opus 4.7 System Card |
| Claude Opus 4.7 | 92.0% | private validation set; summary-table value; developer-reported (card §8.1) | 2026-04-16 | Claude Opus 4.7 System Card |
| Claude Opus 4.6 | 93.0% | summary-table value; developer-reported (card §8.1) | 2026-04-16 | Claude Opus 4.7 System Card |
| GPT-5.4 Pro | 94.5% | xhigh reasoning; $7.21 per task; ARC Prize verified, reported in OpenAI's launch table | 2026-03-05 | (Source: openai.com); Claude Opus 4.7 System Card |
| GPT-5.4 | 93.7% | xhigh reasoning; $0.620 per task; ARC Prize verified, reported in OpenAI's launch table | 2026-03-05 | (Source: openai.com); Claude Opus 4.7 System Card |
| Gemini 3.1 Pro | 98.0% | preview; $0.522 per task; ARC Prize verified | 2026-02-19 | (Source: arcprize.org); Claude Opus 4.7 System Card |
| Claude Sonnet 4.6 | 86.50% | 120k thinking tokens, High effort; private dataset, reported by the ARC Prize Foundation (card §2.7) | 2026-02-17 | Claude Sonnet 4.6 System Card |
| Claude Opus 4.6 | 94.0% | 120K thinking budget, high effort; $1.39 per task; ARC Prize verified | 2026-02-05 | (Source: arcprize.org); Claude Opus 4.6 System Card; Claude Opus 4.7 System Card |
| GPT-5.2 Pro | 90.5% | xhigh reasoning; $11.65 per task; ARC Prize verified, reported in OpenAI's launch table | 2025-12-11 | (Source: openai.com) |
| GPT-5.2 Thinking | 86.2% | xhigh reasoning; $0.960 per task; ARC Prize verified, reported in OpenAI's launch table | 2025-12-11 | (Source: openai.com) |
| GPT-5.1 Thinking | 72.8% | high reasoning; ARC Prize verified, reported in OpenAI's GPT-5.2 launch table | 2025-12-11 | (Source: openai.com) |
| Claude Opus 4.5 | 80.0% | 64K thinking budget; $1.47 per task; ARC Prize verified | 2025-11-24 | (Source: arcprize.org); System Card: Claude Opus 4.5 (Anthropic, November 2025) |
| Gemini 3 Pro | 75.0% | $0.493 per task; ARC Prize verified | 2025-11-18 | (Source: arcprize.org) |
| GPT-5 | 65.7% | high reasoning; $0.509 per task; ARC Prize verified | 2025-08-07 | (Source: arcprize.org) |
| Grok 4 | 66.7% | thinking; $1.01 per task; ARC Prize verified | 2025-07-09 | (Source: arcprize.org) |
| The ARChitects (ARC Prize 2024 winner) | 56.0% | purpose-built competition system; $0.200 per task | 2025-05-14 | (Source: arxiv.org) |
| o3 (released model) | 53.8% | medium reasoning; $0.288 per task; ARC Prize verified. ARC Prize's April 2025 post gave 53% | 2025-04-22 | (Source: arcprize.org) |
| o4-mini | 41.8% | medium reasoning; $0.150 per task; ARC Prize verified. ARC Prize's April 2025 post gave 42% | 2025-04-22 | (Source: arcprize.org) |
| DeepSeek R1 | 15.8% | $0.060 per task; ARC Prize verified | 2025-01-20 | (Source: arcprize.org) |
| o3 (preview) | 87.5% | 1,024 samples per task, about 172 times the compute of the 75.7% run and above the $10,000 limit; model trained on 75% of the public training set | 2024-12-20 | (Source: arcprize.org) |
| o3 (preview) | 75.7% | 6 samples per task, within the $10,000 limit; model trained on 75% of the public training set. ARC Prize's post, repriced in December 2025, gives $26 per task; the leaderboard lists $200 | 2024-12-20 | (Source: arcprize.org) |
| GPT-4o | 4.5% | $0.050 per task; ARC Prize verified | 2024-12 | (Source: arcprize.org) |
| Human panel | 98.0% | $17.00 per task, as listed by ARC Prize | 2024-12 | (Source: arcprize.org) |
| Average Mechanical Turk worker | 77.0% | $3.00 per task, as listed by ARC Prize | 2024-12 | (Source: arcprize.org) |
ARC-AGI-2 semi-private
Percentage of the 120 semi-private ARC-AGI-2 tasks solved with two attempts per task.
| Model | Score | Setting | Date | Source |
|---|---|---|---|---|
| GPT-6 Astra | 95.0% | max reasoning; $1.12 per task; ARC Prize verified | 2026-09-03 | (Source: arcprize.org) |
| Claude Fable 5.1 | 90.0% | max effort; $4.49 per task; ARC Prize verified | 2026-09-01 | (Source: arcprize.org); System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026) |
| Claude Fable 5 / Mythos 5 | 89.2% | Table 8.1.A column headed "Claude Fable 5/Mythos 5" (Figure 8.16.B labels the series Claude Fable 5); ARC Prize Foundation; effort not stated (card §8.1 Table 8.1.A) | 2026-09-01 | System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026) |
| Claude Opus 5 | 90.42% | ARC Prize Foundation; effort not stated (card §8.1 Table 8.1.A) | 2026-09-01 | System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026); System Card: Claude Opus 5 (Anthropic, July 2026) |
| DeepSeek V4 Pro 0813 | 61.3% | max reasoning; $0.598 per task; ARC Prize verified | 2026-08-13 | (Source: arcprize.org) |
| Inkling-Small | 40.1% | xhigh effort; $0.233 per task; ARC Prize verified | 2026-07-30 | (Source: arcprize.org); Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| Claude 4.5 Haiku | 4.0% | evaluation set not stated in the card; the Inkling and Inkling-Small values equal the ARC Prize verified semi-private figures on evaluations/arc-agi; reported by Thinking Machines Lab in the Inkling-Small model card (card §5. Evaluations) | 2026-07-30 | Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| GPT 5.6 Luna | 47.6% | evaluation set not stated in the card; the Inkling and Inkling-Small values equal the ARC Prize verified semi-private figures on evaluations/arc-agi; reported by Thinking Machines Lab in the Inkling-Small model card (card §5. Evaluations) | 2026-07-30 | Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| Claude Opus 5 | 90.4% | max effort; $2.06 per task; ARC Prize verified | 2026-07-24 | (Source: arcprize.org) |
| Kimi K3 | 60.4% | max reasoning; $1.59 per task; ARC Prize verified | 2026-07-16 | (Source: arcprize.org) |
| Inkling | 36.5% | $0.642 per task; ARC Prize verified | 2026-07-15 | (Source: arcprize.org); Inkling-Small Model Card (Thinking Machines Lab, July 2026) |
| GPT-5.6 Sol | 92.5% | max reasoning; $1.44 per task; ARC Prize verified | 2026-07-09 | (Source: arcprize.org); System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026); System Card: Claude Opus 5 (Anthropic, July 2026) |
| GLM-5.2 | 22.8% | $0.250 per task; ARC Prize verified | 2026-06-16 | (Source: arcprize.org) |
| Claude Fable 5 | 89.2% | max effort; $5.45 per task; ARC Prize verified | 2026-06-09 | (Source: arcprize.org) |
| Claude Opus 4.8 | 72.1% | high effort; $2.74 per task; ARC Prize verified | 2026-05-28 | (Source: arcprize.org); System Card: Claude Opus 5 (Anthropic, July 2026) |
| GPT-5.5 | 85.0% | xhigh reasoning; $1.87 per task; ARC Prize verified, reported in OpenAI's launch table | 2026-04-23 | (Source: openai.com) |
| Claude Opus 4.7 | 75.8% | max effort; $7.43 per task; ARC Prize verified | 2026-04-16 | (Source: arcprize.org) |
| Claude Opus 4.7 | 75.83% | max thinking; private validation set; described as a new high for Opus-class models; developer-reported (card §8.1, 8.11) | 2026-04-16 | Claude Opus 4.7 System Card; System Card: Claude Opus 5 (Anthropic, July 2026) |
| GPT-5.4 Pro | 83.3% | xhigh reasoning; $16.41 per task; ARC Prize verified, reported in OpenAI's launch table | 2026-03-05 | (Source: openai.com); Claude Opus 4.7 System Card |
| GPT-5.4 | 73.3% | xhigh reasoning; labelled ARC Prize verified in OpenAI's launch table. The ARC Prize leaderboard lists 74.0% at $1.52 per task | 2026-03-05 | (Source: openai.com); Claude Opus 4.7 System Card |
| Gemini 3.1 Pro | 77.1% | Thinking (High); $0.962 per task; ARC Prize verified, reported in Google DeepMind's launch table | 2026-02-19 | (Source: deepmind.google); Claude Opus 4.7 System Card; Gemini 3.1 Pro Model Card (Google DeepMind, February 2026) |
| Claude Sonnet 4.6 | 60.4% | high effort, 120K thinking budget; $2.70 per task; reported by Anthropic and listed on the ARC Prize leaderboard | 2026-02-17 | (Source: anthropic.com) |
| Claude Sonnet 4.6 | 60.42% | 120k thinking tokens, High effort; private dataset, reported by the ARC Prize Foundation (card §2.7 / 2.1 n.4) | 2026-02-17 | Claude Sonnet 4.6 System Card |
| Claude Sonnet 4.6 | 58.3% | max effort (summary-table value); developer-reported (card §2.1) | 2026-02-17 | Claude Sonnet 4.6 System Card; Gemini 3.1 Pro Model Card (Google DeepMind, February 2026) |
| Claude Opus 4.6 | 69.17% | reported by the ARC Prize Foundation on their private dataset, with 120k thinking tokens and High effort; no training specifically for ARC-AGI-2 (card §2.9) | 2026-02-15 | Claude Opus 4.6 System Card |
| Claude Opus 4.6 | 68.8% | figure in summary table 2.3.A, labelled "ARC-AGI-2 (Verified)"; differs from the 69.17% stated in Section 2.9 (see Notes) (card §2.3) | 2026-02-15 | Claude Opus 4.6 System Card; Claude Opus 4.7 System Card; Claude Sonnet 4.6 System Card; Gemini 3.1 Pro Model Card (Google DeepMind, February 2026) |
| Gemini 3 Pro (Deep Thinking) | 45.1% | reported by Anthropic in the Claude Opus 4.6 System Card summary table; the table marks this column entry "(Deep Thinking)" (card §2.3) | 2026-02-15 | Claude Opus 4.6 System Card |
| GPT-5.2 | 54.2% | reported by Anthropic in the Claude Opus 4.6 System Card summary table; setting not stated (card §2.3) | 2026-02-15 | Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card |
| GLM-5 | 4.9% | $0.270 per task; ARC Prize verified | 2026-02-11 | (Source: arcprize.org) |
| Claude Opus 4.6 | 69.2% | 120K thinking budget, high effort; $3.47 per task; ARC Prize verified. Google DeepMind's table gives 68.8% for the max setting | 2026-02-05 | (Source: arcprize.org); Claude Sonnet 4.6 System Card |
| Gemini 3 Deep Think (February 2026) | 84.6% | $13.62 per task; ARC Prize verified | 2026-02 | (Source: arcprize.org) |
| GPT-5.2 Pro | 54.2% | high reasoning; $15.72 per task; ARC Prize verified, reported in OpenAI's launch table | 2025-12-11 | (Source: openai.com) |
| GPT-5.2 Thinking | 52.9% | xhigh reasoning; $1.90 per task; ARC Prize verified, reported in OpenAI's launch table | 2025-12-11 | (Source: openai.com); Gemini 3.1 Pro Model Card (Google DeepMind, February 2026) |
| GPT-5.1 Thinking | 17.6% | high reasoning; ARC Prize verified, reported in OpenAI's GPT-5.2 launch table | 2025-12-11 | (Source: openai.com); System Card: Claude Opus 4.5 (Anthropic, November 2025); Gemini 3 Pro Model Card (Google DeepMind, November 2025) |
| Poetiq refinement system on Gemini 3 Pro | 54% | open-source refinement loop around the model; $31 per task; ARC Prize verified | 2025-12-05 | (Source: arcprize.org) |
| Claude Opus 4.5 | 37.6% | 64K thinking budget; $2.20 per task in ARC Prize's December 2025 analysis ($2.40 on the leaderboard); ARC Prize verified | 2025-12-05 | (Source: arcprize.org); System Card: Claude Opus 4.5 (Anthropic, November 2025); Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card |
| Claude Sonnet 4.5 | 13.6% | reported by Anthropic in the Claude Opus 4.5 system card (card §2.3) | 2025-11-24 | System Card: Claude Opus 4.5 (Anthropic, November 2025); Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card; Gemini 3 Pro Model Card (Google DeepMind, November 2025) |
| Gemini 3 Deep Think (preview) | 45.1% | with code execution; ARC Prize verified, reported by Google; the leaderboard gives a provisional $77.16 per task | 2025-11-18 | (Source: blog.google) |
| Gemini 3 Pro | 31.1% | $0.811 per task; ARC Prize verified | 2025-11-18 | (Source: arcprize.org); System Card: Claude Opus 4.5 (Anthropic, November 2025); Claude Sonnet 4.6 System Card; Gemini 3.1 Pro Model Card (Google DeepMind, February 2026); Gemini 3 Pro Model Card (Google DeepMind, November 2025) |
| Gemini 2.5 Pro | 4.9% | ARC Prize Verified, semi-private set; sourced from the ARC Prize website; developer-reported (Google DeepMind, in the Gemini 3 Pro model card) (card §Evaluation – Results (table, PDF p. 5)) | 2025-11-18 | Gemini 3 Pro Model Card (Google DeepMind, November 2025) |
| GPT-5 | 9.9% | high reasoning; $0.730 per task; ARC Prize verified | 2025-08-07 | (Source: arcprize.org) |
| Grok 4 | 16.0% | thinking; $2.17 per task; ARC Prize verified. xAI's launch post gave 15.9% | 2025-07-09 | (Source: arcprize.org) |
| o3 | 3.0% | medium reasoning; baseline from the ARC-AGI-2 paper, as of May 14, 2025 | 2025-05-14 | ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems |
| The ARChitects (ARC Prize 2024 winner) | 2.5% | purpose-built competition system; baseline from the ARC-AGI-2 paper | 2025-05-14 | ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems |
| Claude 3.7 Sonnet | 0.9% | 8K thinking budget; baseline from the ARC-AGI-2 paper | 2025-05-14 | ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems |
| Human panel | 100.0% | $17.00 per task, as listed by ARC Prize; every task was solved by at least two people within two attempts | 2025-03-24 | (Source: arcprize.org) |
ARC-AGI-3 semi-private
Relative Human Action Efficiency on the 55 semi-private environments, run by ARC Prize. Results at the March 2026 launch were run with no harness under one shared system prompt; ARC Prize now labels this series "Standard". The cost is the total for the run.
| Model | Score | Setting | Date | Source |
|---|---|---|---|---|
| GPT-6 Astra | 62.7% | Standard harness; max reasoning; $26,098 for the run; ARC Prize verified | 2026-09-03 | (Source: arcprize.org) |
| Grok 4.6 | 2.1% | Standard harness; xhigh reasoning; $5.6K for the run; ARC Prize verified | 2026-08-12 | (Source: arcprize.org) |
| Claude Opus 5 | 30.2% | Standard harness; high effort; $20.7K for the run; ARC Prize verified | 2026-07-24 | (Source: arcprize.org) |
| Claude Opus 5 | 30.16% | verified Relative Human Action Efficiency score at high effort on the semi-private game set; max-effort results unavailable at release; reported by the ARC Prize Foundation (Table 8.1.A prints 30.2 (high)) (card §8.1; 8.14.2) | 2026-07-24 | System Card: Claude Opus 5 (Anthropic, July 2026) |
| Claude Opus 4.8 | 1.52% | high effort; reported by the ARC Prize Foundation via the Claude Opus 5 system card (Table 8.1.A prints 1.5) (card §8.1; 8.14.2) | 2026-07-24 | System Card: Claude Opus 5 (Anthropic, July 2026) |
| GPT-5.6 Sol | 7.78% | Standard harness; max reasoning; $25.1K for the run; ARC Prize verified | 2026-07-09 | (Source: arcprize.org); System Card: Claude Opus 5 (Anthropic, July 2026) |
| Grok 4.5 | 0.3% | Standard harness; high reasoning; $6.9K for the run; ARC Prize verified | 2026-07-08 | (Source: arcprize.org) |
| Claude Opus 4.8 | 1.5% | Standard harness; high effort; $10.0K for the run; ARC Prize verified | 2026-05-28 | (Source: arcprize.org) |
| GPT-5.5 | 0.4% | Standard harness; high reasoning; $10.0K for the run; ARC Prize verified | 2026-04-23 | (Source: arcprize.org) |
| Claude Opus 4.7 | 0.2% | Standard harness; high effort; $10.0K for the run; ARC Prize verified | 2026-04-16 | (Source: arcprize.org) |
| Claude Opus 4.6 | 0.50% | no harness, shared system prompt; max effort; launch result. The leaderboard lists $8.9K for the run | 2026-03-25 | ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence |
| Gemini 3.1 Pro Preview | 0.40% | no harness, shared system prompt; launch result. The leaderboard lists $2.2K for the run | 2026-03-25 | ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence |
| GPT-5.4 | 0.20% | no harness, shared system prompt; high reasoning; launch result. The leaderboard lists $5.2K for the run | 2026-03-25 | ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence |
| Grok-4.20 (Beta 0309 Reasoning) | 0.10% | no harness, shared system prompt; launch result. The leaderboard lists $3.8K for the run | 2026-03-25 | ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence |
ARC-AGI-3 semi-private (Provider Adapter harness)
The same 55 environments run by ARC Prize with the provider's own context-management features enabled. ARC Prize reports these separately from the Standard series.
| Model | Score | Setting | Date | Source |
|---|---|---|---|---|
| GPT-6 Astra | 99.9% | Provider Adapter harness; high reasoning; $18,817 for the run; ARC Prize verified | 2026-09-03 | (Source: arcprize.org) |
| GPT-6 Astra | 98.6% | Provider Adapter harness; max reasoning; $17,332 for the run; ARC Prize verified | 2026-09-03 | (Source: arcprize.org) |
| GPT-6 Astra | 96.7% | Provider Adapter harness; reasoning effort none; $23,457 for the run; ARC Prize verified | 2026-09-03 | (Source: arcprize.org) |
ARC-AGI-3 public set (self-reported and harness-assisted)
Relative Human Action Efficiency on the 25 public demonstration environments. ARC Prize does not place public-set scores on its official leaderboard, and none of the agent-system results here is ARC Prize verified. The first three rows measure a purpose-built agent system or a developer's own configuration rather than a model under the common test conditions.
| Model | Score | Setting | Date | Source |
|---|---|---|---|---|
| NVIDIA AVO agent system with Claude Opus 5 | 100.00% | self-reported by NVIDIA; all 183 levels completed in 6,624 environment actions; public set only | 2026-08-21 | (Source: developer.nvidia.com) |
| GPT-5.6 Sol | 38.3% | self-reported by OpenAI; official harness plus retained reasoning and compaction through the Responses API | 2026-07-29 | (Source: openai.com) |
| Schema harness with Claude Opus 4.8 and Claude Fable 5 fallback | 98.98% | self-reported by Impossible Research with UC Berkeley and Carnegie Mellon co-authors; not ARC Prize verified | 2026-07-15 | (Source: schema-harness.github.io) |
| GPT-5.6 Sol | 13.33% | measured by ARC Prize on the public set; max reasoning; official harness. OpenAI's own run of the official harness gave 13.3% | 2026-07-09 | (Source: arcprize.org) |
ARC-AGI-3 Axis Reflect (ar25) — Single-game transcript analysis
Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.
| Model | Score | Setting | Date | Source |
|---|---|---|---|---|
| Claude Opus 5 | 100 | completed the game with a score of 100, clearing all eight levels in 294 actions; analysis produced by an LLM judge for the ARC Prize Foundation (card §8.14.2) | 2026-07-24 | System Card: Claude Opus 5 (Anthropic, July 2026) |
ARC-AGI (as reported in developer documents)
Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.
| Model | Score | Setting | Date | Source |
|---|---|---|---|---|
| Seed1.5-Thinking | 39.9% | version (ARC-AGI-1 vs 2) and split not stated; single response; thinking model; developer-reported (card §6.1, Table 2) | 2025-04-29 | Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning (ByteDance Seed, April 2025) |
| DeepSeek R1 | 18.3% | version (ARC-AGI-1 vs 2) and split not stated; single response; thinking model; reported by ByteDance Seed in the Seed1.5-Thinking report; source of the comparison figure not stated (card §6.1, Table 2) | 2025-04-29 | Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning (ByteDance Seed, April 2025) |
| OpenAI o3-mini | 25.8% | version (ARC-AGI-1 vs 2) and split not stated; single response; thinking model; reported by ByteDance Seed in the Seed1.5-Thinking report; source of the comparison figure not stated (card §6.1, Table 2) | 2025-04-29 | Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning (ByteDance Seed, April 2025) |
| Grok 3 Beta | 31.9% | version (ARC-AGI-1 vs 2) and split not stated; single response; thinking model; reported by ByteDance Seed in the Seed1.5-Thinking report; source of the comparison figure not stated (card §6.1, Table 2) | 2025-04-29 | Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning (ByteDance Seed, April 2025) |
| Gemini 2.5 Pro | 27.6% | version (ARC-AGI-1 vs 2) and split not stated; single response; thinking model; reported by ByteDance Seed in the Seed1.5-Thinking report; source of the comparison figure not stated (card §6.1, Table 2) | 2025-04-29 | Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning (ByteDance Seed, April 2025) |
Interpreting scores
A verified ARC-AGI score describes a model under one configuration at one cost. The leaderboard plots score against cost per task because the same model scores very differently at different reasoning settings: GPT-6 Astra scored 59.6% on ARC-AGI-2 with reasoning disabled and 95.0% at maximum, and Claude Opus 4.5 ranged from 7.8% with no thinking budget to 37.6% at 64K tokens (Source: arcprize.org). The o3 preview's 87.5% used 1,024 samples per task at a retail cost ARC Prize put in the thousands of dollars per task, which is why the foundation made efficiency "a required metric when reporting performance"; it noted that a person could be paid about $5 per task to solve the same puzzles (Source: arcprize.org). OpenAI stated a year later that GPT-5.2 Pro crossed 90% on ARC-AGI-1 at roughly 390 times lower cost than the o3 preview (Source: openai.com).
On the two static generations there is no chance-level floor, since the answer is a grid that must be produced exactly, and the ARC-AGI-2 paper says accuracies below 5% "are generally not treated as meaningful" because they "likely result from noise-level heuristics or incidental pattern fits" (Source: arxiv.org). The ceiling is set by human solvability: every ARC-AGI-2 task was solved by at least two people, though individual participants solved about two-thirds of the test pairs they attempted (Source: arxiv.org). On ARC-AGI-3 a score of 100% means matching the upper-median human's action count on every level, not merely completing every environment; OpenAI estimates that the average human tester would score 48% on the metric (Source: openai.com), and ARC Prize reported that under the Provider Adapter harness Astra used fewer actions than the median tested human on 96.0% of levels and 51.7% fewer actions per level on average (Source: arcprize.org).
Reported ARC-AGI-3 results in 2026 have diverged sharply depending on the surrounding agent system. On July 15, 2026, Impossible Research and co-authors at UC Berkeley and Carnegie Mellon posted the Schema harness, self-reporting 98.98% on the ARC-AGI-3 public set using Claude Opus 4.8 with Claude Fable 5 fallback, against an official leaderboard best of 13.33% at the time; those figures are not ARC Prize-verified (Source: schema-harness.github.io). On August 21, 2026, NVIDIA reported that its AVO agent system scored 100.00 on the Relative Human Action Efficiency metric across the 25-environment ARC-AGI-3 public set, completing all 183 levels in 6,624 environment actions using Claude Opus 5, against the 7,542 actions VISTA reported with the same model; ARC Prize separately reports approximately 30% for Claude Opus 5 at high reasoning effort. NVIDIA noted that the two systems differ in agent backend, observation representation and memory, and that the figures cover the public set rather than the semi-private or private competition sets (Source: developer.nvidia.com). Anthropic's launch material for Opus 5 described its ARC-AGI-3 score as three times as high as the next-best model's (Source: anthropic.com).
The gap between harness-assisted results on the public set and unassisted verified results on the held-out sets is the reason ARC Prize maintains semi-private and private evaluation sets and a verification process, and such figures are better understood as measurements of a system than of a model, a distinction developed at AI benchmarks and evaluation. The same applies within the verified results: for Astra the difference between the two official harnesses was 37 points at best, and with reasoning disabled the model scored 96.7% under the Provider Adapter against 62.7% at maximum reasoning under the Standard harness. Across the 167 game and reasoning-level pairs that both harnesses solved, ARC Prize measured the adapter runs at 49% fewer tokens and roughly 3.66 times faster. Knoop wrote that "we lack evidence to call this AGI yet," and Chollet gave the standard-harness figure as 66% in a post against 62.7% in the published table (Source: arcprize.org; thenextweb.com). See Agentic harnesses and capability elicitation and Context Compaction.
The foundation has been explicit about what high scores do not show. Of the o3 preview, Chollet wrote that "passing ARC-AGI does not equate to achieving AGI," adding that the model "still fails on some very easy tasks" (Source: arcprize.org). Of Astra, ARC Prize wrote that it is "not claiming that it is AGI," because ARC-AGI-3 "has a tightly bounded scope and format, and its environments have deterministic, closed-ended mechanics and goals. It does not represent the complexity and open-endedness of the real world" (Source: arcprize.org).
Limitations and criticism
Training on the benchmark
The o3 preview was trained on 75% of the ARC-AGI-1 public training set, and ARC Prize's April 2025 retest of the released o3, which OpenAI said was a different model not directly trained on ARC-AGI, measured 41% and 53% at low and medium reasoning effort against the preview's 75.7% (Source: arcprize.org; arcprize.org). Anthropic corrected its Claude Opus 4.5 system card in November 2025 to disclose that it had trained on a reshuffled split of both the public training and public evaluation sets of ARC-AGI-1, not the training set alone, and replaced public-set scores with semi-private ones. The foundation itself states its belief that ARC-AGI-1 and ARC-AGI-2 data has entered frontier training corpora "either incidentally or intentionally," citing a Gemini 3 Deep Think reasoning trace that applied the correct ARC colour mapping although the verification prompt mentioned neither ARC nor the mapping; it calls this a new kind of overfitting that arises when public and private sets are similarly distributed, and says it is "not precisely sure how much" this helps models (ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence; Source: arcprize.org).
Whether solutions reflect the intended ability
Melanie Mitchell, who has described ARC as a benchmark she values and who now sits on the foundation's academic panel, wrote after the o3 result that the winning 2024 methods and o3 each ran against the assumptions that made the benchmark interesting: that a solver should need little training on the domain, only a few demonstrations per task, and little computation. She called the 2024 results "the bitter lesson of ARC," and argued that it remained open whether systems solving the tasks had formed the abstractions the tasks were designed to probe or would generalize them to variations (Source: aiguide.substack.com). Other critics argue that the series tests a narrow cognitive style (ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems).
Replacement by successive versions
The ARC-AGI-2 paper acknowledges that ARC-AGI-1 "saturated well before it could capture the full spectrum of human fluid intelligence," and the foundation describes ARC-AGI-1 as solved (Source: arxiv.org; arcprize.org). A July 2026 Federal Reserve Board staff note cites the series as an example of a recurring pattern, writing that early standardized-test benchmarks "were quickly saturated, with subsequent expert level reasoning benchmarks (e.g. GPQA and ARC-AGI) following a similar pattern" (The AI Buildout and the Economy: Publicly Available Data to Assess AI's Impact (FEDS Notes, July 2026)). The foundation's response is that replacement is by design: Kamradt has said that the project ends when "we can no longer come up with quantifiable problems that are 'feasible for humans and hard for AI'" (ARC Prize Foundation), and after the Astra result the foundation said it is exploring how to evaluate "recursive self-improvement and open-ended innovation" in a next generation (Source: arcprize.org).
Discrepancies in published figures
Reported numbers for the same result sometimes differ between ARC Prize pages and developer tables, as the Setting cells above record: GPT-5.4's ARC-AGI-2 score appears as 73.3% in OpenAI's launch table and 74.0% on the leaderboard, and the o3 preview's cost per task appears as $26 in the repriced blog post and $200 on the leaderboard. OpenAI also changed Astra's ARC-AGI-3 figure on its own announcement page from 98.6% in a pre-publication draft to 99.99% in the live post (Astra).
Use in policy and safety frameworks
ARC-AGI is reported as a capability measure in developer documentation rather than used as a safety threshold. Anthropic's Claude Opus 4.6 and Claude Sonnet 4.6 system cards list it as their measure of novel reasoning (Claude Sonnet 4.6), and OpenAI and Google DeepMind print "ARC Prize Verified" figures in launch tables (Source: openai.com; deepmind.google). The foundation's testing policy is addressed in part to policymakers, stating that it is published "so that labs, researchers, policymakers, and the public can audit our methods," and the foundation has said it wants ARC-AGI to be "an academic, policy, and industry standard for measuring and guiding AGI" (Source: arcprize.org; ARC Prize Foundation). The Federal Reserve staff note cited above uses the series when cautioning that strong benchmark performance "does not necessarily translate into the ability to complete actual on-the-job tasks" (The AI Buildout and the Economy: Publicly Available Data to Assess AI's Impact (FEDS Notes, July 2026)). In public argument about whether AGI has arrived, ARC-AGI results have been invoked on both sides: after Nvidia's Jensen Huang declared the race to AGI over in September 2026, Gary Marcus pointed to Chollet's statement of September 3, 2026 that saturating ARC-AGI-3 is not a claim of AGI (Artificial General Intelligence (AGI)). "A Definition of AGI" offers a separate operationalization based on psychometric batteries rather than abstraction tasks.
Relationships
- instance-of: AI Benchmarks and Evaluation — ARC-AGI is one benchmark family within the broader evaluation landscape.
- depends-on: François Chollet — originator of the benchmark and of the underlying definition of intelligence.
- depends-on: ARC Prize Foundation — maintainer, verifier and competition organizer.
- supports: ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems — the second-generation benchmark paper.
- supports: ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence — the third-generation benchmark paper.
- related: Agentic AI — ARC-AGI-3 evaluates agentic exploration and planning rather than static puzzle-solving.
- related: Jagged Frontier — the human-easy / machine-hard design principle is one source of evidence for uneven capability profiles.
- related: Agentic harnesses and capability elicitation, Context Compaction — the harness and API-settings disputes over ARC-AGI-3 scores.
- related: Artificial General Intelligence (AGI) — the definitional dispute in which ARC-AGI results are cited.
- related: OpenAI o-series (o1 → o4-mini), Astra — the models whose results ended the first and third generations' period of low scores.
See also
- GPQA, Humanity's Last Exam — knowledge-intensive benchmarks built on the opposite design principle.
- FrontierMath
- METR Time Horizons
- Sapient Intelligence — developer of the Hierarchical Reasoning Model, a small model evaluated on ARC-AGI tasks.
- Ryan Greenblatt — author of the 2024 program-sampling approach.