Artificial Analysis Intelligence Index

The Artificial Analysis Intelligence Index is a composite score published by Artificial Analysis, a San Francisco benchmarking company, that averages a language model's results on a suite of evaluations the company runs itself. Version 1.0 dates from January 2024; the index has been rebuilt repeatedly since, and version 4.3, introduced in September 2026, combines ten evaluations in four weighted categories (Source: artificialanalysis.ai). The company calls it its "synthesis metric for assessing generalist model intelligence" (Source: x.com). Developers including SpaceXAI, Thinking Machines Lab and OpenAI cite the company's indices in launch materials, and the index appears beside cost, speed and token-use figures on the company's public leaderboard.

What it measures

Artificial Analysis describes the index as combining evaluations "to assess language model capabilities across reasoning, knowledge, maths and programming", and as "a helpful synthesis of overall language model intelligence" that, "like all evaluation metrics, … has limitations and may not apply directly to every use case." It is "primarily text-based" and English-language; image input, speech and multilingual performance are benchmarked separately (Source: artificialanalysis.ai). The company's guidance is that a higher score "indicates stronger aggregate performance across the benchmark suite, but individual benchmark results may be more relevant for specific use cases" (Source: artificialanalysis.ai).

Design

Components and weights in version 4.3

The methodology page states that the index "is calculated as a weighted average across four categories: Agents (30%), Coding (20%), Scientific Reasoning (20%) and General (30%)", with the weighting chosen to emphasize agentic tasks (Source: artificialanalysis.ai). The FAQ on the company's evaluation page still describes four categories contributing 25% each, the scheme introduced with version 4.0 (Source: artificialanalysis.ai).

(Source: artificialanalysis.ai; Source: artificialanalysis.ai)

As an example of an agentic item, a published GDPval-AA task casts the model as a department supervisor at a retail electronics store who must produce a daily task list as a PDF (Source: artificialanalysis.ai).

Scoring

Most components use pass@1: the model gets one attempt per item, and where an item is repeated the score is the share of all attempts that were correct. Two components are Elo ratings from pairwise comparisons, which have no fixed ceiling; to enter the average each is frozen when a model is added and mapped to a 0–1 range as (Elo − 500) / 2000, clamped at the ends. Several components are graded by other language models: in September 2026 GPT-5.6 Luna served as the equality checker for Humanity's Last Exam, AA-LCR and AA-Omniscience, and the pairwise judge panel consisted of Claude Opus 5, GPT-5.6 Sol and Gemini 3.8 Flash (Source: artificialanalysis.ai).

Test conditions

The company states four principles: standardized conditions for all models, answer extraction that does not penalize valid formatting, zero-shot instruction prompts with no worked examples, and published prompts and methods. Temperature is 0 for non-reasoning models and 0.6 for reasoning models unless the developer recommends otherwise; reasoning models receive their maximum output-token allowance; failed API calls are retried up to 30 times. Agentic components run in sandboxes from the provider e2b (Source: artificialanalysis.ai). Each reasoning-effort level of a model (low, high, max) is scored as a separate entry.

Uncertainty, cost and tokens

Artificial Analysis estimates "a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1%", based on more than ten repeat runs of certain models, while noting that individual components may vary more and that it looks "forward to disclosing further detail from our statistical analysis in the future." Alongside each score it reports the output tokens consumed and a cost per task: each component's cost at the provider's token prices, divided by its task count and weighted as in the index, with provider-reported token counts and measured cache-hit rates (Source: artificialanalysis.ai; Source: artificialanalysis.ai).

Held-out data

With version 4.2 the company said that 40% of the index's weight rested on "private, held-out test sets", double the share in version 4.1, comprising AA-Briefcase, AA-Omniscience and the solutions to CritPt, and that the share would rise again in version 5 (Source: artificialanalysis.ai).

Versions and variants

The methodology page keeps a version history (Source: artificialanalysis.ai), supplemented here by the launch announcements.

The company's evaluation page was headed "v4.3.2" on September 19, 2026, a sub-version the methodology page's history did not yet list. Announcing version 4.2 on September 4, 2026, the company called it an interim step toward a version 5, saying it had "deliberately held back updates to keep the Index stable through recent major model launches" (Source: artificialanalysis.ai).

Related measures from the same company include a Coding Index, a Multilingual Index based on Global-MMLU-Lite in 16 languages, a Healthcare & Medical Index and an Openness Index. Components are also reported singly: GDPval-AA, the AA-Omniscience knowledge and hallucination benchmark, and AA-LCR. Evaluations it runs outside the index include APEX-Agents-AA, Harvey LAB-AA, IFBench and MMMU-Pro (Source: artificialanalysis.ai).

Development and maintainers

Artificial Analysis was founded by Micah Hill-Smith, its chief executive and a former McKinsey consultant, and George Cameron, its chief product officer. Its listed backers are Nat Friedman, Daniel Gross, Andrew Ng, Adam D'Angelo, Clem Delangue and Guillermo Rauch. It reports having benchmarked more than 500 models (Source: artificialanalysis.ai). The company states that "all evaluations are conducted independently by Artificial Analysis"; there is no submission route for self-reported scores, and developer figures are not mixed in. Some models are evaluated through pre-release or limited-preview access, as with Muse Spark 1.3 (max) in September 2026 (Source: x.com). The component benchmarks are a mix of its own (AA-Briefcase, AA-Omniscience, AA-LCR), its implementations of others' datasets (GDPval-AA from OpenAI's GDPval, GDP.pdf from Surge AI, AutomationBench-AA), and external benchmarks run in collaboration with their authors (CritPt).

Results

Because composition changes, the leading score has fallen with each rebuild and not only risen with model progress: the company reported top scores of 73 under version 3.0 and 50 or below when version 4.0 launched in January 2026, when GPT-5.2 at its highest reasoning setting led, followed by Claude Opus 4.5 and Gemini 3 Pro (Source: x.com). Under the version 4.1 series the top score rose to 66, for Claude Fable 5.1, by September 2, 2026 (Source: x.com). Under version 4.3 the same model leads at 53, level with GPT-6 Astra (Source: artificialanalysis.ai).

Intelligence Index v4.3

Scores from the company's public leaderboard as retrieved on September 19, 2026, which carries no separate update date; the Date column is the retrieval date. One row per model at the highest-scoring configuration listed, with cost per index task. Entries the leaderboard marks with an asterisk are omitted because the retrieved page does not explain the mark.

Intelligence Index v4.1

Scores published while versions 4.1 and 4.1.1 were current (June to early September 2026). The August change of grader models means figures from before and after it may differ slightly for the same model. Rows reported by parties other than Artificial Analysis are marked.

Artificial Analysis Intelligence Index v4.0 — Intelligence Index v4.0

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Artificial Analysis Intelligence Index (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

A score is a weighted average of percentages, so 53 means a model earned roughly half the available weighted credit on a suite built to be hard, not that it is "53% intelligent". No human baseline exists for the index as a whole. The scale's meaning changes with the version number. The same model can move by more than ten points between versions with no change in the model: Claude Fable 5.1 (max) scored 66 under version 4.1.1 on September 2, 2026 and 53 under version 4.3 later that month (Source: x.com; Source: artificialanalysis.ai). Scores quoted without a version, which is common in launch posts and commentary, cannot safely be compared across months.

Within a version, the company's stated 95% interval of "less than ±1%" means that models one point apart are not clearly separated. Reasoning effort matters more: under version 4.3 GPT-6 Astra ranges from 46 at low effort to 53 at max, and Claude Opus 5 from 40 to 51, with cost per task rising from $1.10 to $5.86 for the latter (Source: artificialanalysis.ai). The company attributes the higher score of Muse Spark 1.3 (max) over its xhigh variant to "using more turns and total reasoning tokens" (Source: x.com). Because agentic components carry 30% of the weight and run in the company's own harnesses, the index reflects model-plus-harness performance in those harnesses, which can differ from a developer's own: Artificial Analysis measured DeepSeek V4 Flash 0731 at 79% on Terminal-Bench 2.1 where DeepSeek, using its own harness, reported 82.7 (DeepSeek V4 Pro / V4 Flash).

The index does not cover safety, multimodal input, speech or languages other than English, and two of its component scores depend on LLM judges comparing outputs pairwise.

Limitations and criticism

Artificial Analysis itself frames each rebuild as a response to two problems. One is saturation: it removed GPQA Diamond in version 4.2 as "an exceptional scientific reasoning evaluation that has now been saturated", and presented version 4.0 as "less saturated". The other is gaming: it described the rise in held-out weight as reducing "the ability for labs to game evaluations" (Source: artificialanalysis.ai; Source: x.com). The cost is continuity; there is no version-independent series, and the company does not publish a mapping between versions.

The methodology page flags a bias in one component: the authors of Humanity's Last Exam selected questions adversarially against GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet and OpenAI's o1 models, so Artificial Analysis discourages comparing those models directly with others on it (Source: artificialanalysis.ai). The statistical basis for the ±1% interval has not been published in detail. The pairwise judge panel is drawn from models of three of the developers being ranked.

Users of the index note its limits for their purposes. Exponential View used it as one of three capability weights for token volumes and found quality-adjusted output growth of 35x on this index against 33x on the Epoch Capabilities Index and 59x on METR's task horizon, concluding that "we can only draw directional conclusions" (The State of the AI Economy (Exponential View, June 2026)).

Use in policy and safety frameworks

No safety framework or regulator is recorded as using the index as a threshold. Its role is in market and competitive analysis. SpaceXAI's Grok 4.6 launch placed the model level with GPT-5.6 Sol (max) at 61 (Grok (xAI)). Thinking Machines Lab used the company's externally run scores for its own and competitors' models (Inkling-Small). OpenAI cited the Coding Index in claiming GPT-5.5 delivered leading results at half the cost of competing coding models (GPT-5.5 ('Spud')). Nathan Lambert cited Kimi K3's third place on the index in assessing how close open-weight models were to the frontier (Kimi K3: The open-weights escalation (Nathan Lambert, July 2026)). In remarks prepared for a September 2026 congressional briefing he used the index to compare Chinese and American open-weight models, giving scores "as of writing this on September 14, 2026" of 45 for GLM-5.3, 44 for Kimi K3 and 42 for GLM-5.3-Flash against 26 for Inkling and Inkling Small and 23 for Nemotron 3 Ultra, without naming an index version; the figures match the version 4.3 leaderboard as retrieved on September 19 (The current balance of power in open models (Nathan Lambert, September 2026)). In an August 2026 Fortune account, Hill-Smith said Google's strongest shipped model then sat behind models from Anthropic, OpenAI, one or two Chinese labs, xAI and Meta on the index (Google DeepMind). Open-weight comparisons in Open-Weight Frontier Models and Open-Source AI / Open-Weight Models draw on the same rankings.

Relationships

• instance-of: AI Benchmarks and Evaluation
• depends-on: GDPval, Humanity's Last Exam, Terminal-Bench — component evaluations; formerly GPQA, MMLU-Pro, LiveCodeBench, AIME (as a model benchmark), MATH and MATH-500, τ-bench.
• related: Epoch Capabilities Index (ECI), LMArena (Chatbot Arena), HELM (Holistic Evaluation of Language Models) — other aggregate instruments.
• related: The State of the AI Economy (Exponential View, June 2026), Kimi K3: The open-weights escalation (Nathan Lambert, July 2026), The current balance of power in open models (Nathan Lambert, September 2026)
• related: Kimi K3, DeepSeek V4 Pro / V4 Flash, Muse Spark (Meta Superintelligence Labs), Grok (xAI)

See also

• APEX-Agents — run by Artificial Analysis outside the index.
• Inference Economics and Token Pricing — cost-per-task reporting.