This is a monograph by François Chollet, then at Google, posted to arXiv as 1911.01547 on 5 November 2019. It surveys a century of attempts to define and measure intelligence in psychology and AI, argues that measuring skill at specific tasks does not measure intelligence, proposes a formal definition of intelligence as skill-acquisition efficiency grounded in algorithmic information theory, and introduces the Abstraction and Reasoning Corpus (ARC), the dataset that became ARC-AGI. The paper acknowledges comments from José Hernández-Orallo, Julian Togelius, Christian Szegedy and Martin Wicke.
Summary
Two conceptions of intelligence
Part I traces two historical views. One, associated with Marvin Minsky's Society of Mind, treats intelligence as a collection of task-specific programs, which Chollet says led AI to benchmark intelligence by comparing human and machine skill at specific tasks such as board games and video games. The other, which Chollet traces to Turing's 1950 paper, locates intelligence in the general ability to acquire new skills through learning. The paper argues that skill at any given task "is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to 'buy' arbitrary levels of skills for a system, in a way that masks the system's own generalization power."
The paper distinguishes system-centric generalization (handling situations the system has not encountered) from developer-aware generalization (handling situations that neither the system nor its developer has encountered), and lays out a spectrum from local generalization ("robustness": new points from a known distribution), through broad generalization ("flexibility": a broad category of tasks without further human intervention), to extreme generalization ("adaptation to unknown unknowns across an unknown range of tasks and domains"). It characterizes contemporary deep learning models as local-generalization systems, "conceptually similar to a locality-sensitive hashtable."
The formal definition
Part II states the definition in words: "The intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience, and generalization difficulty." It then formalizes it using algorithmic information theory, equating intelligence with "the information-efficiency with which the system acquires its final task-specific skill … on average (probabilistic average over all applicable curricula), weighted by the developer-aware generalization difficulty of the task considered." From this it derives guidelines for a general AI benchmark: measure abilities rather than skills, feature only tasks unknown to the developer, quantify and control for priors and experience, and compare humans and machines only within a shared scope and with similar priors. The paper's closing section summarizes the position in four points: intelligence is the efficiency with which a learning system turns experience and priors into skill at previously unknown tasks; a measure must account for priors, experience and generalization difficulty; all intelligence is relative to a scope; and general AI should therefore be benchmarked against human intelligence on a similar set of priors, such as Core Knowledge.
The ARC dataset
Part III introduces ARC, described as "a general artificial intelligence benchmark, … a program synthesis benchmark, or … a psychometric intelligence test," similar in format to Raven's Progressive Matrices. Its goals are to stay close to psychometric tests so that humans can take it without practice, to measure developer-aware generalization through novel evaluation tasks, to measure broad generalization through highly abstract tasks specified by very few examples, to control experience by fixing the demonstration data per task, and to state its priors explicitly.
ARC comprises a training set of 400 tasks and an evaluation set of 600, split into a public evaluation set of 400 and a private evaluation set of 200; all tasks are unique and the sets are disjoint. Each task has a small number of demonstration examples (3.3 on average) and usually one test example (occasionally two or three). Each example is an input grid and an output grid of symbols drawn from 10 colours, of any size from 1×1 to 30×30 (median height 9, width 10). The test-taker must construct the output grid from scratch, including its dimensions; a task is solved only if every test output is exactly correct, with three trials allowed per test example and only binary feedback. The score is the fraction of evaluation tasks solved. The training set is intended as a development set for developers or a mock test for humans; Chollet states that "a typical human can solve most of the ARC evaluation set without any previous training."
The priors are listed as Core Knowledge categories: objectness (cohesion, persistence, influence via contact), goal-directedness, numbers and counting (quantities below about 10, addition and subtraction), and basic geometry and topology (lines and rectangles, symmetry, rotation, translation, scaling, containment, line drawing, copying). ARC differs from IQ tests, the paper says, by excluding crystallized knowledge, language and real-world imagery, by keeping evaluation tasks unknown to developers (enforceable through the private set), by offering hundreds of diverse tasks, and by hand-authoring tasks rather than generating them from a master program that could be reverse-engineered.
On human performance the paper reports only that "each task included in ARC has been successfully solved by at least one member of a group of three high-IQ humans," and it states that ARC "does not appear to be approachable by any existing machine learning technique (including Deep Learning)." It sketches a hypothetical solver as a program-synthesis engine over a domain-specific language encoding the Core Knowledge priors, and posits that a human-level ARC solver "would represent the ability to program an AI from demonstrations alone," while calling these claims "highly speculative."
Stated weaknesses
Section III.2 lists five: generalization difficulty is not quantified; test validity is not established; the dataset is limited to 1,000 tasks with possible conceptual overlap, leaving it "potentially vulnerable to shortcut strategies," to be probed through public competitions on the private set; the binary, close-ended format lacks granularity, and an interactive format in which the test-taker requests new inputs and receives feedback would measure intelligence more directly; and the Core Knowledge priors "may not be well understood and may not be well captured in ARC." Section III.3 proposes alternatives, including repurposing game benchmarks by evaluating on unseen variants and open-ended teacher–student task generation.
Key claims
- Intelligence is defined as skill-acquisition efficiency over a scope of tasks, controlling for priors, experience and generalization difficulty; task skill alone is not a measure of it.
- ARC has 400 training tasks and 600 evaluation tasks (400 public, 200 private), each a set of grid input-output pairs on up to 30×30 grids with 10 colours, scored by exact match with three trials.
- ARC assumes only Core Knowledge priors (objectness, goal-directedness, counting, basic geometry and topology) and excludes language and acquired knowledge.
- Every task was solved by at least one of three test-takers; no statistical human baseline is reported.
- The author lists unquantified generalization, unestablished validity, small size, binary scoring and uncertain priors as known weaknesses.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 1911.01547 (arxiv.org); the abstract page is arxiv.org. The task data were released at github.com/fchollet/ARC.
Relationships
- supports: ARC-AGI — the primary text for the benchmark's original design, format and rationale.
- depends-on: AI Benchmarks and Evaluation
- related: ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems, ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence — the successor datasets that retain the format and cite this definition.
- related: ARC Prize 2024: Technical Report (Chollet, Knoop, Kamradt and Landers, 2024), H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark (LeGris et al., 2024) — the competition record and the human baseline study for the original dataset.
- related: François Chollet, ARC Prize Foundation