Finance Agent

Finance Agent is a benchmark maintained by the company Vals AI that tests AI agents on financial research over public-company filings. The agent is given a question of the kind a junior analyst at a bank or fund receives, a search interface to the U.S. Securities and Exchange Commission's EDGAR filing database, web search and tools for reading long documents, and must return an answer with sources. The paper, by authors at Vals AI and Stanford University, is dated May 2025 and was posted to arXiv as 2508.00828 (Source: arxiv.org; vals.ai). The best model in the paper, OpenAI's o3, answered 46.8% of questions correctly at an average cost of $3.79 per question. Version 1.1 topped out at 64.37% in mid-2026, and Vals AI released a harder version 2 in May 2026 on which the leading score in September 2026 was 61.44% (Source: vals.ai; vals.ai).

What it measures

Vals AI states that, after consulting experts at banks and hedge funds, it identified research on the SEC filings of public companies as the one task shared across nearly all financial-analyst workflows, underlying equity research, credit analysis and due diligence. The benchmark is presented as a test of whether agents can do the time-intensive work expected of entry-level analysts, and the paper reports cost and latency beside accuracy so that an agent can be compared with the person it would assist (Source: arxiv.org; vals.ai).

Design

Question writing

The paper describes a taxonomy of nine task types drawn up with seven experts from banks, hedge funds and private equity, each with at least two to three years at firms such as Goldman Sachs and J.P. Morgan. The experts wrote 537 questions, each with a reference answer, the source documents needed and the steps to reach the answer, concentrating on filings published in 2024 or later, after most models' training cutoffs. A second expert reviewed every question, and erroneous ones were corrected or removed. The types, with the paper's difficulty labels and counts, are quantitative retrieval (easy, 102), qualitative retrieval (easy, 97), numerical reasoning (easy, 83), complex retrieval (medium, 29), adjustments (medium, 43), beat or miss (medium, 69), trends (hard, 33), financial modeling (hard, 47) and market analysis (hard, 34). An easy example is "What was the quarterly revenue of Salesforce (NYSE:CRM) for the quarter ended December 31, 2024?"; a hard one asks how much acquisition "firepower" Amazon had at the end of fiscal 2024, counting cash and short-term investments plus borrowing of up to twice EBITDA (Source: arxiv.org).

Vals AI's page walks through one question, the percentage change in shares Netflix repurchased in the fourth quarter of 2024 against the fourth quarter of 2023. The expert opened EDGAR, found the two annual reports, located the table of issuer purchases, summed three months in each year and computed a fall of 79% (Source: vals.ai).

Harness

All models run in the same open-source harness, a simple loop in which the model alternates reasoning and tool calls. Version 1 gives four tools: EDGAR search through an SEC API, Google web search, a parser that loads and chunks an HTML page, and a retrieval tool for asking questions of the parsed text. Models were prompted identically, at temperature 0 where possible. The experts who wrote the answers used no additional tools (Source: arxiv.org; vals.ai).

Scoring

Answers are graded by a language model acting as judge. In version 1, GPT-4o extracted a set of checks from each expert answer, which the paper's authors reviewed by hand; the judge tests a response for each check and, separately, for any statement that contradicts the expert answer, and a response counts as correct only if it passes all of them. The paper's headline is class-balanced accuracy, the average of the nine per-category accuracies, which the authors prefer because the easy retrieval categories are overrepresented; they also report naive accuracy over all questions (Source: arxiv.org).

Splits

The 537 questions were divided into a public validation set of 50, a private validation set of 150 available to researchers and for license, and a private test set of 337. The paper's figures use all 537; Vals AI's leaderboards use only the private test set, which it says prevents training to the test (Source: arxiv.org; vals.ai).

Human comparison

The paper does not give a human accuracy score, since the experts' answers are the key. It reports that experts spent 16.8 minutes per question on average at an estimated cost of $25.66, against 3.1 minutes and $3.79 for o3, and notes that the human figures exclude the time spent writing and reviewing questions (Source: arxiv.org).

Versions and variants

• Version 1 (2025) is the 537-question benchmark in the paper.
• Version 1.1 kept 537 questions; the page for it was last updated on June 4, 2026, and results carry a margin of error of about ±2.8 points. OpenAI and Anthropic reported this version through April 2026 (Source: vals.ai; openai.com).
• Version 2, which Anthropic's Opus 4.8 system card dates to May 2026, has 927 expert-reviewed questions: 27 public, 450 in a private validation set and 450 in the test set. Questions target the depth expected of a second- or third-year investment banking analyst and are designed to have one unambiguous answer, to need several filings, to rely on sector conventions that the question does not spell out, and to turn on details buried in footnotes. The nine categories were reorganized around analyst workflows (general qualitative, general quantitative, earnings analysis, market analysis, disclosure analysis, adjustments, comparables, precedents and financial modeling). The harness adds a calculator and a price-history tool, and each task has a two-hour limit. Grading changed: each question is a set of weighted checks, some flagged as "dealbreakers" whose failure scores the whole answer zero; the primary metric, Partial Credit, is the weighted share of checks passed subject to that gate, and All-Pass counts only answers passing every check. A jury of three models (GPT-5.4, Gemini 3.1 Pro and Claude Sonnet 4.6) grades each response, and each model is run three times with the mean and its standard error reported (Source: vals.ai; System Card: Claude Opus 4.8 (Anthropic, May 2026)).

Development and maintainers

The paper's authors are Antoine Bigeard, Langston Nashold and Rayan Krishnan of Vals AI and Shirley Wu of Stanford University, and it states that the work was funded by Vals AI, "a startup dedicated to evaluating Large Language Models." Vals AI's benchmark page says the dataset was created in collaboration with Stanford researchers, a global systemically important bank that it does not name, and industry experts (Source: arxiv.org; vals.ai).

Vals AI describes itself as "the independent evaluator of artificial intelligence," building benchmarks of whether models can do the work of lawyers, bankers, engineers and doctors. It scores models on privately held test sets, licenses private validation sets to companies, and has open-sourced its evaluation infrastructure. Its other benchmarks include the Vals Index, a composite of agentic performance across finance, coding and legal tasks weighted by each sector's share of U.S. GDP; a hosted LegalBench leaderboard; Legal Research Bench; CaseLaw; CorpFin, a private benchmark on long credit agreements; TaxEval; Tax Agent Bench; MortgageTax; MedQA; and Public Benefits Bench, built with the Center for Civic Futures and Code for America (Source: vals.ai; vals.ai).

Vals AI runs all evaluations itself; developers do not submit scores. Anthropic's system cards state that Vals AI conducted the evaluations of its models on the benchmark (Claude Opus 4.6 System Card; Claude Opus 4.7 System Card). Where a model refuses a task, Vals AI marks scores that include a fallback retry: its version 2 page notes that one of Claude Opus 5's 1,350 run-task results used Claude Opus 4.8 as a refusal fallback, on a task that scored zero either way (Source: vals.ai).

Results

No model passed 50% in the paper: o3 led at 46.8% class-balanced accuracy, Claude 3.7 Sonnet with thinking scored 45.9%, and open-weight Llama models scored under 6% (Source: arxiv.org). By early 2026 Anthropic's system cards were reporting state-of-the-art results around 60%: 60.7% for Claude Opus 4.6, 63.3% for Claude Sonnet 4.6 and 64.4% for Claude Opus 4.7 (Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card; Claude Opus 4.7 System Card). The Stanford AI Index 2026 recorded Sonnet 4.6 leading version 1.1 at 63.33%, with the top score below two-thirds accuracy (Stanford HAI AI Index Report 2026). On version 2, Anthropic reported 53.92% for Claude Opus 4.8 in May 2026 and 56.31% for Claude Fable 5 in June, second to Gemini 3.5 Flash (System Card: Claude Opus 4.8 (Anthropic, May 2026); System Card: Claude Fable 5 & Claude Mythos 5 (Anthropic, June 2026)). Vals AI's version 2 table on September 19, 2026 placed Gemini 3.8 Flash first at 61.44%, while the page's summary text still named Muse Spark 1.2 as leader at 60.60%; under All-Pass scoring the best model reached 50.88% (Source: vals.ai).

Finance Agent v2

Partial Credit accuracy on the 450-question private test set; mean of three runs, three-model jury; measured by Vals AI. Cost is Vals AI's reported cost per test.

Finance Agent v1.1

Accuracy on the private test set of the 537-question version; measured by Vals AI unless stated.

Finance Agent, version not stated (Opus 4.6 system card)

Figures from Anthropic's Claude Opus 4.6 system card, which does not say which version of the benchmark was used; they differ from the version 1.1 leaderboard values for the same models.

Finance Agent v1 (paper)

Class-balanced accuracy on all 537 questions, from the introducing paper; temperature 0, four-tool harness. Cost is average API cost per question.

Interpreting scores

The versions are not comparable. The paper's figures are class-balanced accuracy over all 537 questions; the version 1.1 leaderboard is accuracy on the private test set; and version 2 is a harder question set scored with partial credit. Anthropic's Opus 4.8 system card notes that Opus 4.7, which scored in the 60s on version 1.1, scored 51.51% on version 2, and says scores should not be compared across versions (System Card: Claude Opus 4.8 (Anthropic, May 2026)). Under version 2's strict All-Pass scoring the best model reached 50.88% and most others were below 48%, so the Partial Credit figure should not be read as the share of questions answered fully correctly (Source: vals.ai).

On version 1.1 the margin of error is about ±2.8 points, so models within a few points of each other are not separated. Small differences also appear between a developer's quoted figure and the leaderboard for the same model: Anthropic's Opus 4.6 card gives 60.70% where its Sonnet 4.6 card and Vals AI's page give 60.05%, and the Opus 4.8 card refers to Opus 4.7's version 1.1 score as 63.4% where the Opus 4.7 card and Vals AI give 64.4% (Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card; System Card: Claude Opus 4.8 (Anthropic, May 2026)).

Accuracy varies by question type. Vals AI reports that models do best on simple quantitative and qualitative retrieval, which it calls easy but time-intensive for analysts; on version 2 the category leaders exceed 70% on retrieval and summarization categories and reach only 34.52% on financial modeling and 36.37% on precedents, and no model leads all nine categories. Vals AI also observes that the top performers make relatively many tool calls and that some hard queries cost more than $5 with o3 or Claude Opus 4.1 (Source: vals.ai; vals.ai). The paper found a roughly logarithmic relation between cost and accuracy, with sharply diminishing returns above $1 per question (Source: arxiv.org).

All results use Vals AI's own harness. The paper says the harness is simple and not representative of commercial retrieval systems or proprietary agent products, which might perform differently (Source: arxiv.org).

Limitations and criticism

The paper's limitations section says the dataset emphasizes questions answerable from relatively short passages or a final figure, whereas the experts consulted said much of their work involves spreadsheets and other structured tabular data, of which the benchmark has only a few examples. Models were asked for sources but the accuracy of those citations was not evaluated. Agents had access only to public filings and the open web, not to the private databases and internal documents that analysts use (Source: arxiv.org).

Grading rests on model judges checked by the authors rather than on a reported human-agreement study; the paper cites the tendency of model judges toward false negatives as its reason for rubric-based and contradiction checks (Source: arxiv.org). The test sets are private, which protects against contamination but means outside parties cannot audit the questions or reproduce leaderboard scores beyond the small public sets. Vals AI's About page presents this as a deliberate choice, arguing that openly released test sets are absorbed into training data and that benchmarks run by model developers are reported selectively (Source: vals.ai). No independent critique of the benchmark was located for this page.

Use in policy and safety frameworks

Finance Agent is a capability measure and is not used as a risk threshold. It appears in the capabilities sections of Anthropic's system cards from Claude Opus 4.6 onward, described as "a public benchmark published by Vals AI," alongside Anthropic's internal Real-World Finance evaluation (Claude Opus 4.6 System Card; Claude Sonnet 4.6 System Card; Claude Opus 4.7 System Card; System Card: Claude Opus 4.8 (Anthropic, May 2026); System Card: Claude Fable 5 & Claude Mythos 5 (Anthropic, June 2026)), in OpenAI's GPT-5.4 and GPT-5.5 launch tables, and in Google's Gemini 3.8 Flash announcement, which cited gains over Gemini 3.7 Flash on Vals AI's Finance Agent version 2 (Source: openai.com; openai.com; blog.google). The Stanford AI Index 2026 uses it in its chapter on domain-specific technical performance (Stanford HAI AI Index Report 2026).

Relationships

• instance-of: AI Benchmarks and Evaluation — a privately held, model-graded agent benchmark run by a third-party evaluator.
• related: APEX-Agents — includes investment-banking tasks, two of whose worlds also use EDGAR; GDPval — covers financial analysts among 44 occupations.
• related: GAIA — the paper cites GAIA for its final-answer accuracy metric; SWE-Lancer, PaperBench — cited as models for the harness.
• related: Agentic AI, AI Labor Disruption.

See also

• SWE-bench — Vals AI also publishes a third-party SWE-bench Verified leaderboard (see Claude Sonnet 4.6).
• Terminal-Bench — hosted among Vals AI's coding benchmarks.