ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence

The technical report introducing the third generation of the ARC-AGI benchmark series, credited to the ARC Prize Foundation and posted to arXiv as 2603.24621 (v1 March 24, 2026; v2 April 17, 2026; DOI 10.48550/arXiv.2603.24621). The report names a development team of Hunter Henry as lead environment designer; David Wexler and Derek Smith on engineering; Bryan Landers on design and communications; Gregory Kamradt as president and board member; Mike Knoop as co-founder and board member; and François Chollet as benchmark designer, co-founder and board member.

ARC-AGI-3 shifts the series from static grid puzzles to interactive, turn-based environments in which an agent must explore, infer goals, model environment dynamics and plan action sequences without being told the objective. The abstract states that human testing shows people solve 100% of the environments "in contrast to frontier AI systems which, as of March 2026, score below 1%," and the conclusions describe ARC-AGI-3 as "the only unsaturated general agentic intelligence benchmark as of March 2026."

Position in the series

The report recounts the series history: ARC-AGI-1 accompanied Chollet's 2019 "On the Measure of Intelligence," testing rule inference from a handful of input-output grid pairs of up to 30×30 cells in 10 colors, grounded in Core Knowledge priors. ARC-AGI-2, introduced March 2025, kept the grid format while requiring multi-step reasoning, sequential rule application and symbolic interpretation, and was calibrated with over 400 untrained participants (ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems). The report gives median human solve times of about 30 seconds per ARC-AGI-1 task against about 300 seconds per ARC-AGI-2 task.

On competitions, it records the 2020 Kaggle Abstraction and Reasoning Challenge ($20,000, 913 teams, winning solution about 20% via brute-force program search), the Lab42 ARCathon events of 2022 and 2023 ($100,000 each), ARC Prize 2024 (over $1 million, 1,430 teams, 47 paper submissions, test-time training reaching 53.5% on the private ARC-AGI-1 test set), and ARC Prize 2025 on ARC-AGI-2 (1,455 teams, 90 papers, NVIDIA's NVARC team first at 24% using synthetic data generation and test-time training on a 4-billion-parameter model). The 85% grand-prize threshold went unclaimed in both years.

Argument for a new generation

The report gives two reasons for moving beyond static tasks. The first concerns what reasoning models can and cannot do: it states that modern large reasoning models enable automation only where the base model already has sufficient domain knowledge and the domain supplies an exact correctness signal, which the report calls verifiable domains. It draws the inference that "AI reasoning capability is tied to LRM knowledge," and adds that this "leads to imprecise descriptions of LLMs as 'jagged intelligence', when in reality LLMs remain bound to task-specific training, albeit now over task-specific reasoning chains instead of the literal task data" — a qualification of the framing described on Jagged Frontier.

The second concerns benchmark contamination. The report argues that a benchmark resistant to direct memorization can still be attacked at a higher level when the public training set and private test set are similarly distributed and a developer trains on millions of automatically generated in-domain tasks, which reduces or removes the need for test-time adaptation. It states the authors believe this has happened to ARC-AGI-1 and ARC-AGI-2, "either incidentally or intentionally," and offers as evidence a Gemini 3 Deep Think reasoning trace that used the correct ARC-AGI integer-to-color mapping despite a verification prompt that mentioned neither ARC-AGI nor the mapping. The stated lesson is that private datasets must be steered out-of-distribution from public demonstration data to test generalization.

Benchmark design

ARC-AGI-3 evaluates four components: exploration (information must be actively obtained), modeling (turning observations into a predictive world model), goal-setting (identifying desirable future states without instruction), and planning and execution (mapping and course-correcting an action path). The agent is never told the objective and must infer the mechanics and win conditions itself.

The observation space is a 64×64 grid in 16 colors, called a frame; frame sequences allow non-interactive animation between turns. The action space is a subset of five key actions plus Undo, and one action selecting a grid cell by coordinates. An action is a turn that affects environment state; internal tool calls, reasoning steps and retries are not counted.

Environments were produced by an in-house game studio through a four-stage pipeline (specification, internal testing, external human testing, done), implemented on a custom Python engine built to reach 1,000 frames per second after Unity was judged too slow to iterate on. Design constraints restrict environments to Core Knowledge priors — objectness, basic geometry and topology, basic physics, agentness — and bar language, numerals, letters, real-world clip-art and cultural conventions such as green meaning "go". Each environment must have at least six levels, a deliberately easy tutorial first level, and multiple mechanics; difficulty is meant to come from composition rather than obscurity. Novelty was tested by asking whether one program could solve two environments while being at least 50% shorter than two independent solutions, which would indicate insufficient distinctness. Environment IDs are four characters, with longer informal names withheld to avoid leaking semantic information.

Validation ran in two layers. Qualification exercised each environment under random play — a 50,000-step regime asserting no level can be beaten by accident, a 1,000,000-step regime for non-tutorial levels, and a third 1,000,000-step sweep serving as a fuzzing harness — plus recording-based playback for reproducibility. State-space analysis built an explicit directed graph over reachable states with hash-based node merging, producing bounds on the probability a random policy solves a level; the acceptance threshold was that random play should succeed no more than 1 in 10,000 times. The report gives the first level of environment ls20 as having a win probability of exactly 1 in 355.

Dataset composition

This inverts ARC-AGI-2's roughly 10:1 public-to-private ratio. The report states the public set is a demonstration interface rather than a training resource, deliberately does not represent the mechanics of the private set, and that the private set is out-of-distribution relative to it.

Scoring: Relative Human Action Efficiency

Scoring is by action efficiency against a human baseline rather than solve rate, under a metric the report calls RHAE (Relative Human Action Efficiency), pronounced "Ray". Intelligence is defined as efficiency across the four components, with data, time, compute and risk collapsed into the single scalar of action count on first contact.

The level efficiency score is S = min(1.15, (h/a)²), where a is the agent's action count and h the human baseline. Four design decisions shape it:

• Human baseline is the upper-median best first-run human. Exactly 10 members of the public were tested per environment; participants completing a level are ranked by action count and the upper-median performer selected — the report gives the third-place finisher among four or five completions as the example.
• Power-law rather than linear. Efficiency is squared, so an agent taking 100 actions where a human took 10 scores 1% for that level rather than the 10% a linear formulation would give. The stated purpose is to distinguish near-human from materially suboptimal performance.
• Per-level cap of 1.15×. Credit for beating the human baseline is allowed but bounded, so that a single exploit — the report's example is a 2-action solution to a 20-action level — cannot dominate an environment score.
• Linear level weighting and a per-environment cap. In a five-level environment level l carries weight l, so the tutorial contributes 1/15th and the final level 5/15th; the environment score is additionally capped at the weighted fraction of levels completed (100% for five of five, about 66.7% for four, 40% for three), so completing more levels is always rewarded.

The total benchmark score is the mean of environment scores across the dataset. The formulation is credited as inspired by the robotics Success weighted by Path Length metric. Because full-set evaluation of high-reasoning frontier models runs to tens of thousands of dollars in API costs, agents are terminated after five times the human-baseline median action count per level.

Leaderboards and the harness question

The report splits reporting into an official and a community leaderboard, and states the official leaderboard exists to help the public judge how close frontier models are to human-level general intelligence while discounting gains from targeting the benchmark. It names two forms of overfitting it seeks to exclude: task-specific overfitting, where an agent is built or configured with knowledge of the public environments, and domain-specific overfitting, where an agent is built to play ARC-AGI-3 environments in general through synthetic lookalikes or benchmark-specific harness strategies.

Three consequences follow, each stated directly in the text. First, the foundation "will never report public set scores of any system on the official leaderboard," and evaluating on the public set "is emphatically not a valid measure of progress towards AGI." Second, to demonstrate that a perfect public score is achievable by construction, the foundation released an open-source harness that scores 100% on all public environments using human replay. Third, the official leaderboard uses no harness at all, on the stated position that "future AGI systems will not need task-specific external handholding to approach new tasks," and runs every model through the same system prompt: "You are playing a game. Your goal is to win. Reply with the exact action you want to take. The final action in your reply will be executed next turn. Your entire reply will be carried to the next turn."

The report supports this with a bimodality result from early testing, in which researchers built general harnesses targeting the public environments ls20, ft09 and vc33 and were then tested on the full public set. In a variant of environment TR87, Opus 4.6 scored 0.0% with no harness and 97.1% with the Duke harness; in environment BP35 it scored 0.0% under both. The authors read this as evidence that frame perception and API format are not the limiting factors, and that engineered harnesses do not measure AGI progress because performance on seen environments does not transfer to unseen ones.

Official semi-private leaderboard scores at release:

The community leaderboard is public, open to anyone, self-reported and unverified by default; the report states it specifically cautions against reading community scores as evidence of AGI progress, while acknowledging harness research as economically valuable. It predicts that sufficiently general harness ideas migrate behind the model API, citing chain-of-thought's path from a third-party DeepMind harness over GPT-3 to OpenAI's internal Q*/Strawberry work and then to o1 as a first-party model.

Human calibration

An environment was included only if at least two of the ten participants independently solved it in full on first exposure; many were solved by six or more. The report records 486 unique participants across 414 candidate environments, producing 2,893 attempts and 427.9 hours of recorded play. Median attempt duration was 7.4 minutes — 8.1 minutes for successful attempts against 5.9 for unsuccessful ones.

Testing moved from ARC-AGI-2's large batched cohorts to continuous sessions three times a week at a dedicated testing center in San Francisco. Participants received no task-specific instructions, worked 90-minute sessions with a 20-minute soft and 30-minute hard limit per environment, and completed about nine environments per session. Compensation was a fixed fee of $115–$140 plus $5 per environment solved. An attempt was defined as a session of more than 30 actions and under 30 minutes; participants had one attempt per environment, could reset a level but not revisit completed ones. Sessions showing low-effort cycling through easy environments were excluded as a misreading of the incentive structure. Participants were drawn from the general public and not selected for training or ability.

Pre-launch testing

The report describes releasing previews before full launch. The ARC-AGI-3 Preview Agent Competition ran 30 days from July 18 to August 19, 2025, with three public and three hidden environments and scoring on the hidden set only. StochasticGoose of Tufa Labs took first at 12.58%, completing 18 levels with a four-layer convolutional network and reinforcement learning predicting which actions change the frame; Blind Squirrel took second at 6.71% using a directed state graph. The report characterizes both as informed search over the action space.

Two harness efforts are described as solving all three public environments. A Duke University team, working under a sponsored collaboration, addressed context management — 64×64 frames exhaust a context budget under a naive rolling window — by letting the model execute arbitrary Python to retrieve and transform its own action history, reaching action counts comparable to human performance. Symbolica AI's Arcgentica used an orchestrator that never touches the environment directly and delegates to subagents returning compressed textual summaries.

ARC Prize 2026

The competition continues across two tracks on Kaggle, with a total prize pool of $2 million and an open-sourcing requirement for prize money. 2026 is the final year of the ARC-AGI-2 track, whose grand prize is guaranteed to be paid to the best team; the primary focus moves to ARC-AGI-3.

Provenance

Fetched August 23, 2026 from arxiv.org/pdf/2603.24621, 23 pages, HTTP 200. Default text extraction de-spaced the document; x_tolerance=1.0 restores word boundaries. Figure plot areas extract as scattered axis labels, so figure-only data — the participant demographics of Figure 5 and the efficiency distributions of Figures 6 and 8 — is not fully recoverable from the capture; both data tables and all prose are intact.

Three dates attach to the document and are recorded rather than reconciled: the arXiv v2 revision stamp of April 17, 2026, the cover date of April 20, 2026 printed on the arXiv PDF itself, and the April 22, 2026 cover date the queue task recorded on the publisher copy at arcprize.org. The arXiv dates are used in frontmatter. The submission history lists François Chollet as submitting author.

The report's acknowledgments credit Surge AI with early support in brainstorming environment concepts, and name seventeen contributing environment developers.

Relationships

• supersedes: none — extends rather than replaces ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems.
• depends-on: ARC-AGI — the benchmark series this generation belongs to.
• supports: AI Benchmarks and Evaluation — supplies the benchmark authors' own position on harness-assisted and public-set scores.
• contradicts: Jagged Frontier — qualifies the "jagged intelligence" description as imprecise, attributing uneven performance to task-specific training over reasoning chains.
• related: François Chollet, Agentic AI, AGI Timelines.