This is a paper by Solim LeGris, Wai Keen Vong, Brenden M. Lake and Todd M. Gureckis of New York University's Department of Psychology and Center for Data Science, posted to arXiv as 2409.01374 in September 2024. It reports the first human-performance estimate covering every task in the public ARC training and evaluation sets, and releases the resulting behavioural dataset, H-ARC, at arc-visualizations.github.io.
Summary
Motivation
The authors note that the only prior estimate of human accuracy on ARC, 83.8%, came from a semi-randomly selected subset of 40 training tasks, leaving unknown how robust that figure was and whether it held on the evaluation set, "which is believed to be much harder." Since comparing human and machine performance "is important for the validity of the benchmark," they set out to measure human accuracy on all 400 training and 400 evaluation tasks.
Method
Participants were recruited on Amazon Mechanical Turk through CloudResearch: 784 for the training set (60.2% male, 37.3% female, 2.5% other) and 948 for the evaluation set (51.1% male, 46.1% female, 2.8% other), aged 18 to 77 (mean 39.8), paid $10 with a $1 bonus for a correct solution to a randomly selected task accompanied by an adequate written description. Each participant received five randomly selected tasks from one split (reduced from ten after 241 participants to limit attrition), with no time limit; sessions averaged 23 minutes on the training set and 29 on the evaluation set. On average 11.8 participants attempted each training task and 10.3 each evaluation task, for 15,744 attempts in total.
The procedure followed the original paper: three attempts per task, with only correct/incorrect feedback after each, and only the first test example used for tasks with several. The interface mirrored the original ARC interface, with cell editing, flood fill, copy and paste, grid resizing and an added undo tool; participants completed a tutorial task and comprehension quiz first. Repeat submission of an identical incorrect grid was blocked after the first 340 training-set participants, since about 8% of second and third attempts had been repeats. Participants also wrote natural-language descriptions of their inferred rule.
Data were incomplete for 90 of 783 training-set and 242 of 946 evaluation-set participants, leaving 7.5% and 13.3% of expected task data missing (10.3% overall). To bound the effect, the authors report pessimistic and optimistic estimates from 1,000 simulations that impute failure or success, respectively, on the tasks dropped participants did not complete; they describe both bounds as "somewhat unrealistic."
Results
Average three-attempt accuracy was 76.2% on the training set (SD 21.5%; pessimistic–optimistic range 73.3–77.2%) and 64.2% on the evaluation set (SD 22.8%; 55.9–68.9%). First-attempt accuracy was 59.9% and 47.8%, and two-attempt accuracy 72.6% and 60.2%, on the two sets. The proportion of tasks solved by at least one participant was 98.8% on each set, 790 of 800 tasks in all; 74 training tasks (18.5%) and 22 evaluation tasks (5.5%) were solved by every participant who tried them, while five tasks in each set (1.3%) were solved by no one, which the authors note means only that none of roughly ten people solved them. Participants averaged 1.3 attempts to solution on training tasks and 1.4 on evaluation tasks; 40.1% of training-set and 33.8% of evaluation-set participants solved every task they saw, and 8.6% and 16.7% solved none. The evaluation set was significantly harder (t(798) = 7.67, p < .001).
For comparison the paper cites two leaderboard entries: minimally prompted Claude 3.5 Sonnet with text-only grids ("Claude-3.5-N"), 19.3% at one attempt and 20.7% at two on a 150-task evaluation subset, and Greenblatt's GPT-4o program-synthesis pipeline ("GPT-4o-NS"), 38.5% and 42.0% on the evaluation set. Comparing two-attempt results, 225 evaluation tasks were solved only by humans, 2 only by GPT-4o-NS and 7 by neither. Humans gained far more from extra attempts (+21.2 and +25.9 points at the second attempt on the two sets, and +5.0 and +10.0 at the third) than the models (+7% and +9.1% proportional gains for Claude-3.5-N and GPT-4o-NS), which the authors read as better learning from minimal feedback.
Analyses of difficulty found evaluation-set output grids larger on average (235.1 versus 136.2 cells) but only weakly related to difficulty on the training set and unrelated on the evaluation set; participants spent more time per task (6m 39s versus 4m 19s) and more inter-action "thinking" time (1m 37s versus 1m 6s) on evaluation tasks. Among errors, 68.2% (training) and 73.5% (evaluation) of incorrect human submissions had the correct grid dimensions, against dimension errors in only 10.3% of Claude-3.5-N's and 8.3% of GPT-4o-NS's incorrect outputs; normalized edit distance from the true output was 0.19 for humans and both models, but human and machine errors differed pairwise by 0.25, "suggest[ing] that they are making substantially different errors." Copy errors accounted for 10.0% and 9.2% of human errors against 3.5% and 1.6% for the two models.
Interpretation
The authors caution that average performance "does not reflect some absolute measure of human performance" but "how a paid Internet worker from Amazon Mechanical Turk will typically do at the given pay rate," and distinguish it from competence: with over 90% of tasks solved by at least three randomly assigned participants and 98.8% by at least one, they conclude that "human intelligence is in principle capable of carrying out the required computations to solve any ARC task." They do not identify why the evaluation set is harder, suggesting that its primitive operations may be harder to infer or name, or its programs longer.
Key claims
- 1,729 participants; average three-attempt accuracy 76.2% on the 400 training tasks and 64.2% on the 400 public evaluation tasks, with bounds of 73.3–77.2% and 55.9–68.9% after imputing missing data.
- 790 of 800 tasks (98.8%) were solved by at least one participant within three attempts.
- The evaluation set is significantly harder than the training set for people; the cause is not established.
- Humans and the two compared models make errors of a different character despite similar edit distances.
- H-ARC releases 15,744 attempts with full action traces and natural-language rule descriptions.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2409.01374 (arxiv.org); the abstract page is arxiv.org. Data collection was supported in part by NSF BCS grant 2121102.
Relationships
- supports: ARC-AGI — the source of the independent human baseline cited there.
- depends-on: On the Measure of Intelligence (Chollet, 2019) — uses the original dataset and evaluation procedure.
- related: ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024) — cites this study for human solvability.
- related: Ryan Greenblatt — the GPT-4o-NS comparison method is his.
- instance-of: AI Benchmarks and Evaluation