Authors: François Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, Henry Pinkard Published: arXiv 2505.11831, May 2025 (revised January 2026)
ARC-AGI-2 is the second-generation benchmark in the ARC-AGI family, presented in a 2025 paper by François Chollet and colleagues. It retains the grid-puzzle task format of the original ARC-AGI, introduced by François Chollet in 2019 as a test of general fluid intelligence rather than domain knowledge, while curating a new dataset designed to resist the pattern-matching and brute-force approaches that had begun scoring well on ARC-AGI-1.
Summary
In each ARC-AGI task, a system is shown a few input-output pairs of colored grids, must infer the transformation rule that maps inputs to outputs, and must then apply that rule to a new input. ARC-AGI-2 preserves this input-output pair format from ARC-AGI-1 (Chollet 2019, arXiv:1911.01547) for research continuity. Chollet originally positioned ARC-AGI as a measure of general fluid intelligence, the ability to infer novel rules from few examples, rather than of accumulated domain knowledge.
The redesign responds to results on the first benchmark, which had become partially saturable by scaled test-time compute, most notably OpenAI o3's reported result in late 2024. ARC-AGI-2's tasks are curated to be resistant to memorization and to require minimal domain knowledge, while remaining feasible for untrained humans to solve. The paper reports extensive human testing to establish baseline performance, and states that current AI systems lag significantly behind humans on ARC-AGI-2 despite their progress on ARC-AGI-1.
Key claims
- ARC-AGI-2 preserves the input-output pair task format of ARC-AGI-1 for research continuity. (high)
- Tasks are designed to be resistant to memorization and minimal in required domain knowledge, while remaining feasible for untrained humans. (high)
- Human testing establishes baseline performance, and current AI systems lag significantly behind humans on ARC-AGI-2 despite progress on ARC-AGI-1. (high)
- The benchmark targets generalization from few examples, in contrast with capability benchmarks such as GPQA and SWE-bench that measure knowledge or large-scale reasoning. (high)
Positioning and reception
The ARC-AGI family sits at the center of debate over how to measure progress toward general intelligence. Chollet has publicly argued that passing ARC-AGI is necessary, if not sufficient, for genuine general intelligence; critics argue that it tests a narrow cognitive style. The release of ARC-AGI-2 is itself an indication that ARC-AGI-1 had been partially saturated by scaled test-time compute in o3-class systems, prompting the benchmark's own authors to reset the bar. The sequence illustrates a recurring pattern relevant to AI Benchmarks and Evaluation, in which benchmark saturation drives benchmark replacement.
Relationships
- supports: AI Benchmarks and Evaluation, François Chollet
- supersedes: ARC-AGI-1 (Chollet 2019, arXiv:1911.01547)
- related: AI Software Progress, Scaling Laws
- instance-of: AI Benchmarks and Evaluation