This is the technical report on the first ARC Prize competition, written by François Chollet, Mike Knoop, Gregory Kamradt and Bryan Landers of the ARC Prize Foundation and posted to arXiv as 2412.04604, dated 5 December 2024. It describes the ARC-AGI-1 dataset as used in competition, reports the 2024 results, surveys the approaches that produced them, and sets out the flaws in ARC-AGI-1 that led to ARC-AGI-2.
Summary
The benchmark as of 2024
The report restates the benchmark's premise: ARC-AGI (renamed from ARC "to avoid name collisions with other AI benchmarks") measures the ability to solve novel tasks that cannot be prepared for in advance, with every task following a different human-authored logic, requiring no world knowledge or language, and assuming only Core Knowledge priors such as objectness, basic topology and elementary integer arithmetic, which the report describes as acquired by children typically before age four. A task has two or more demonstration pairs (median three) and one or more test inputs on grids of up to 30 by 30 cells with ten values; the test-taker has two attempts per test input.
ARC-AGI-1's 1,000 tasks are split four ways: 400 public training tasks (easy), 400 public evaluation tasks (hard), 100 semi-private evaluation tasks (hard, unreleased but exposed to commercial APIs and so at risk of leakage) and 100 private evaluation tasks (hard, fully private). State-of-the-art scores are reported only on the private set. On human performance, the report states that the original private tasks were tested by two people who scored 97% and 98% and together solved 100%, and cites a New York University study showing that 99% of public evaluation tasks were solved by at least one of ten Mechanical Turk workers (H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark (LeGris et al., 2024)).
Before 2024 the benchmark had been the subject of a 2020 Kaggle competition ($20,000 in prizes) and two ARCathons in 2022 and 2023 ($100,000 each, run with the Davos non-profit Lab42). The top score rose from 20% in 2020 to 33% by 2024; no deep-learning approach scored above 1% in the 2020 competition and GPT-3 scored 0% on the public evaluation set by direct prompting. The authors read this stall as evidence that "progress towards AGI had stalled" while systems grew and memorized more data.
ARC Prize 2024 results
The competition ran from 11 June to 10 November 2024 on Kaggle and arcprize.org, with a $600,000 grand prize for the first team to reach 85% on the private set, $50,000 in progress prizes and $75,000 in paper prizes. The grand prize was not claimed. Kaggle entries ran on a single P100 GPU for under 12 hours with no internet access, and only open-sourced solutions could be named winners. The winners were the ARChitects (53.5%), Guillermo Barbadillo (40%), alijs (40%), William Wu (37%) and PoohAI (37%). MindsAI achieved the highest score of the competition, 55.5%, but declined to open-source its solution and was ineligible for a prize. In total 1,430 teams submitted 17,789 entries.
A secondary ARC-AGI-Pub leaderboard allowed internet access and roughly 1,000 times more compute (up to $10,000 in API credits against the equivalent of $10 per Kaggle entry), verified on the semi-private set with the public set reported alongside to detect overfitting (a gap above 10 points was treated as overfit). Its final standings were Jeremy Berman 53.6% semi-private / 58.5% public, Akyürek et al. 47.5% / 62.8%, Ryan Greenblatt 43% / 42%, o1-preview 18% / 21%, Claude 3.5 Sonnet 14% / 21%, GPT-4o 5% / 9% and Gemini 1.5 4.5% / 8%, the model results using a common direct-prompting approach at pass@1. The authors note that the compute-limited and compute-rich leaderboards "tracked closely," which they take to mean that "massive compute may not be necessary in order to beat ARC-AGI."
Paper awards went to Li et al. ("Combining Induction and Transduction for Abstract Reasoning"), Akyürek et al. ("The Surprising Effectiveness of Test-Time Training for Abstract Reasoning") and Bonnet and Macfarlane ("Searching Latent Program Spaces"), with runners-up including the ARChitects, Barbadillo, Fletcher-Hill, Ouellette and Puget.
Approaches
Until 2024 all top approaches used discrete program search, beginning with icecuber's brute-force search over a domain-specific language that won the 2020 competition at 20%; the report notes that ensembling every 2020 entry would have solved 49% of the private set, and that the best single brute-force submission reached 40%. Three families drove the 2024 gains. Deep-learning-guided program synthesis includes Greenblatt's method of prompting GPT-4o to generate thousands of Python programs per task and debug the most promising, and Ouellette's LLM-guided search over a DSL; the authors estimate that Greenblatt's approach could reach 85% by generating and debugging around 100 million programs per task at a multi-million-dollar compute cost, and observe that LLM-guided synthesis "does not currently decisively beat DSL-based brute-force program search," both scoring around 40% at comparable budgets. Test-time training (TTT), pioneered on ARC by Jack Cole and Mohamed Osman in 2023 and used by MindsAI, fine-tunes a model on each task's demonstration pairs at test time before predicting the output directly (transduction); the report states that no static-inference transduction solution scores above 11%, and lists OmniARC (a Qwen2.5-0.5B model), Akyürek et al. (an 8B model reaching 53% on the public set), MindsAI (a Salesforce T5 model) and the ARChitects (NeMo-Minitron-8B with augmentation and a stability-based selection criterion). The third family combines program synthesis with transduction, since the two solve substantially different tasks; every top 2024 score used such a combination, with the best single-method submissions around 40%.
The report forecasts that program synthesis and test-time search "will be adopted by every frontier AI system over the next 12 to 24 months," which will require reporting compute alongside scores since "any search-based approach can always score higher if provided with more compute," and that test-time training or a derivative "should become popular from 2026 onwards."
Flaws in ARC-AGI-1 and the case for ARC-AGI-2
The authors list three problems with the unchanged 2019 private set: with only 100 tasks and on the order of 10,000 scores reported across four competitions, there is "a significant risk of overfitting"; the 49% solvable by brute-force search in 2020 suggests a large fraction of tasks "does not carry a useful signal towards AGI"; and the evaluation sets do not appear to share a consistent human difficulty distribution. ARC-AGI-2, with larger semi-private and private sets, was planned for the 2025 competition. The report also records that at least seven funded organisations, including Basis AI, Tufa Labs, Agemo and Symbolica, had reoriented toward ARC-AGI, and commits to running ARC Prize annually until the benchmark is beaten and a public reference solution is shared.
Key claims
- ARC-AGI-1 is split 400 training / 400 public evaluation / 100 semi-private / 100 private; official scores use the private set; two attempts per test input.
- The private-set state of the art rose from 33% to 55.5% (MindsAI) during ARC Prize 2024; the open-source winner scored 53.5%; the 85% grand prize was unclaimed.
- Frontier models by direct prompting scored 18% (o1-preview), 14% (Claude 3.5 Sonnet), 5% (GPT-4o) and 4.5% (Gemini 1.5) on the semi-private set.
- Ensembling all 2020 brute-force entries would have solved 49% of the private set; no static transduction approach exceeds 11%.
- The authors cite overfitting risk on the 100-task private set and brute-force susceptibility as reasons for ARC-AGI-2.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2412.04604 (arxiv.org); the abstract page is arxiv.org. Results are also published at arcprize.org.
Relationships
- supports: ARC-AGI — the source for the dataset split, competition rules, 2024 results and approach taxonomy.
- depends-on: On the Measure of Intelligence (Chollet, 2019) — the definition and dataset the competition is built on.
- related: ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems — the successor dataset this report announces.
- related: H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark (LeGris et al., 2024) — the human study the report cites.
- related: ARC Prize Foundation, François Chollet, Ryan Greenblatt