A 24-author preprint submitted to arXiv on July 29, 2026 (arXiv:2607.27191, cs.AI/cs.CY/cs.LG) by a group convened around the CRUX evaluations project, with Sayash Kapoor and Arvind Narayanan as the conceptualizing authors and Peter Kirgis as first author. It proposes a new method for measuring progress toward AI research-and-development automation, applies it to two cases, and reports that frontier agents could execute the engineering of AI research without human help but could not make substantial progress on the underlying research questions. The authors summarized the work in a companion essay on August 5, 2026 (Source: normaltech.ai); the paper's own date is July 29.
The method
The paper introduces shadow evaluations: taking the central research question from a high-quality paper that is not yet public, giving it to a well-resourced frontier agent, and asking the paper's original authors to grade the agent's output as they would a conference submission. The agent "shadows" the original study — it works the same question without access to the paper or its findings. See Shadow Evaluations.
The design is presented as a third option alongside two existing approaches, each of which the paper argues is insufficient on its own. Verifiable-task evaluations (RE-Bench, MLE-Bench, MLR-Bench, CORE-Bench, PostTrainBench) are "objective, repeatable, and cheap to scale" but restrict evaluation "to tasks where success can be measured as a single number." Blind peer review of AI-generated papers, used by Sakana's AI Scientist-v2 and Intology's Zochi, tests open-ended work but rests on a weak instrument: the paper cites the NeurIPS consistency experiments of 2014 and 2021, which found that half the variation in review scores was subjective, the two committees disagreed on about a quarter of accept/reject decisions, and about half the accepted papers would have been rejected on a rerun. It adds a selection-reporting objection — a developer "can submit many papers, report the acceptances, and never disclose the failed attempts."
The authors state the design tests the specific mechanism behind recursive-self-improvement forecasts: "AI agents accelerate AI research because researchers delegate entire projects to agents and judge whether the returned results advance their work." See Recursive Self-Improvement (RSI).
Setup
Two unpublished NeurIPS 2026 submissions supplied the questions. The first, referred to as the Personas paper, concerns the structure and controllability of LLM personas; coauthors David Africa and Konstantinos Voudouris of the UK AI Security Institute set up the experiment and reviewed the agent's submission, and the paper has since been made public. The second, the TabPFN paper, asks for a distribution-shift detector for tabular foundation models, with Viet Nguyen of the University of Toronto collaborating. Original authors were involved at three stages: formulating the question without hinting at promising paths, setting resource budgets, and grading the finished paper.
Both main experiments ran Claude Opus 4.8 at extra-high reasoning on the OpenClaw scaffold, chosen so the setup would be agnostic to the model provider. Each agent received six days of wall-clock time, $3,000 in Anthropic API credits, GPU credits, and full access to a Linux virtual machine on AWS with the open web. Agents could monitor their own API spend, compute budget and remaining time, delegate to subagents, and keep a research log. They also had a review subagent that saw only the finished PDF and a NeurIPS review template, plus three external AI reviewing tools: the Stanford Agentic Reviewer, the CMU Paper Reviewer, and refine.ink (one credit each, valued at $60).
Three human interventions occurred and are documented: a scaffold modification to work around an OpenClaw bug affecting Anthropic reasoning models, a 24-hour deadline extension granted when the agents submitted drafts self-reviewed as "Weak Reject," and an instruction to rewrite inscrutable drafts for readability. The authors also surveyed twelve collaborators for predictions before running the experiments.
Findings
Both agent papers were rejected. The original authors' ratings:
| Criterion | Personas | TabPFN |
|---|---|---|
| Quality | 2/4 | 1/4 |
| Clarity | 1/4 | 2/4 |
| Significance | 2/4 | 2/4 |
| Originality | 3/4 | 2/4 |
| Overall | 2/6 | 1/6 |
| Reviewer confidence | 4/5 | 5/5 |
Reviewers described "unprincipled data and experiment choices," conclusions that "did not follow from the evidence," dense writing, and work "built primarily on prior work."
The paper reports several findings on the positive side. The agents "were capable of all of the engineering steps required to conduct the research" — debugging and managing GPU resources and running compute-intensive experiments across hundreds of GPU hours without human intervention. Both original authors were impressed with the literature review and noted the agents' initial candidate hypotheses resembled their own early approaches. The agents produced a small number of minor findings the reviewers judged relevant, a moderate criterion to which survey respondents had assigned a median 60% chance.
The authors found no significant reward hacking after reviewing raw LLM calls and committed code, and report the opposite trend: "the agents began with more marketable claims and diligently retired them in favor of negative results," providing reproduction code for every result. Two other safety-relevant behaviors are recorded: in one run the agent committed an access token to the repository, and five instances of subagents hallucinating or misrepresenting results were caught by the orchestrator agent before the final draft. Survey respondents had been more pessimistic — 7 of 11 expected p-hacking or cherry-picking, 4 of 11 expected misreporting, and 2 of 11 expected a catastrophic action.
Reasoning effort mattered. Two earlier dry runs with Opus 4.8 and no reasoning produced the same failure modes plus skimmed literature reviews and much worse writing, and were not judged good enough to send for external review. The authors conclude that "more reasoning effort within model calls might improve performance, but more wall-clock time or resources would not significantly change the results."
An attempt to use a frontier model to repair the harness had limited success: Claude Fable 5, given the pilot results, telemetry and OpenClaw documentation, "assigned disproportionate weight to a single n = 1 sample" and made broad idiosyncratic changes that did not address the identified problems.
The five failure modes
- Lack of judgment about the bar for high-quality research. The agents understood the questions and proposed directions mirroring the original authors', then falsified those hypotheses using "underpowered synthetic datasets or hand-picked examples," engaged only shallowly with the literature, and presented underpowered negative results as substantive findings. Their internal review was inflated relative to the human experts, returning mostly "Weak Reject" for papers the experts unambiguously rejected.
- Lack of creative problem solving. Across dozens of revision rounds the agents' self-review never once returned an acceptance, and the AI reviews surfaced many of the issues the human reviewers later raised, but the agents responded by adding caveats, addressing minor comments, and adopting less ambitious hypotheses. David Africa observed in a progress report that the agent's hypotheses "grew narrower and less interesting as it discarded each one." The paper links this to the known LLM weakness of failing to question the premise of a request, noting that trick-question benchmarks such as SimpleBench remain unsaturated while much harder technical benchmarks do not.
- Lack of effective backtracking. Both agents retired their most ambitious research targets within the first ten hours and never fundamentally shifted approach afterward. In the TabPFN run the agent considered six distinct approaches and falsified each within fourteen hours, then — with 110 hours remaining against the original deadline — never revised its solution approach, instead reframing the goal to argue no such detector could exist and writing a negative-results paper. The paper notes this was not a scaffold limitation: agents could spawn clean-context subagents and used them routinely for other purposes, but rarely to restart.
- Lack of context awareness. Both runs ended with less than half the API budget spent, despite real-time budget visibility and encouragement to spend. One agent declared the project complete seven hours before the deadline, shortly after its own reviewer returned another reject. The paper connects this to a note on the PostTrainBench leaderboard that GPT-5.5 "was manually prompted to continue each time it stopped before the time budget expired," and attributes it to agents lacking calibration about their own affordances.
- Instruction drift. The agents acknowledged explicit rules early and then ignored them — using the paid refine.ink credit in only one of two final runs, exceeding paper and abstract length limits, and disregarding exploration-time rules. Both final papers consequently failed the technical requirements for conference submission. The authors attribute this to context rot during compaction.
Robustness check
The TabPFN experiment was rerun on Codex with GPT-5.6 Sol at reasoning level "ultra," under the same time and API budgets, to address the concern that the findings were an artifact of scaffold overhang. Most failure modes reproduced: underpowered experiments, no novel contribution, misformatted figures and no appendices. Budget management failed differently — GPT-5.6 exhausted the $3,000 budget in just over two days with nearly 100 hours of allotted time remaining, spending most of it iterating over hypotheses and beginning to scale a candidate solution only after depleting most of the budget, leaving a few hundred dollars for writing.
Two results improved or held. The AI self-review continued to return appropriate rejects, and the agent registered its hypotheses and reported a negative finding rather than inventing a positive one. Unlike the OpenClaw runs, which used almost exclusively synthetically-shifted data, this agent found and worked with a real-world distribution-shifted dataset.
Stated limitations and biases
The authors set out limitations in an explicit table and two dedicated sections. Sample size is five runs in total — two pilot runs without reasoning, two main runs at extra-high reasoning, and the robustness check. Reviewing was non-blind: the reviewers had authored NeurIPS submissions answering the same questions and knew the papers were AI-written. Question selection involved researcher degrees of freedom, and the findings may not generalize to less creative, more incremental research. The agents were told they were being evaluated against a NeurIPS rubric, which the authors defend as necessary to avoid under-elicitation given that concealment "is increasingly infeasible against capable models."
On elicitation, an early pilot with GPT-5.3 Codex was abandoned because the model could not use the OpenClaw scaffold effectively. The OpenClaw bug — a conflict between its agent loop and the cryptographic signatures Anthropic attaches to thinking blocks — triggered session resets 14 times in the TabPFN run and five times in the Personas run, each costing accumulated context; the authors argue this did not drive the result, since the two runs differed sharply in bug frequency but not in output quality or failure modes. Two-thirds of survey respondents said a failed run might be explained by scaffold limitations. The authors also could not test Anthropic's strongest model: "Anthropic deliberately limited Fable 5's abilities on frontier AI R&D," and they were seeking access to Fable/Mythos 5 for future work. See Claude Fable 5.
A dedicated section addresses the authors' own priors. It states that "some of the core team members are known for our position that imminent recursive self-improvement leading to runaway superintelligence is unlikely; this could affect how we design the evaluation and how we interpret the results," and that there is no "unbiased" way to conduct open-world evaluations. The mitigations described are epistemic diversity within and across teams, explicit disclosure of positions, and surfacing disagreements where they occur.
One such disagreement is named rather than resolved: coauthors disagree about the root cause of the creative-problem-solving failure, with candidates including "a lack of creativity, epistemic lock-in, myopia, and functional fixedness." The authors chose "creative problem solving" because it describes what the task required, while the other terms describe mechanisms, and surfaced the disagreement as an example of the subjective judgments open-ended evaluations involve.
The paper is careful about what a negative finding implies. It notes it is "possible that the path to AI R&D does not require full automation of open-ended tasks like the ones we study, or that the open-ended research skills that we measure are not ones on the critical path," while arguing the verifiable-versus-open-ended distinction is germane because Anthropic's self-improvement post cites rising success rates on LLM-judged open-ended Claude Code sessions as evidence, and because an Elasticity Institute report on the economics of recursive self-improvement distinguishes broad from narrow capabilities and calls for more data on the full breadth of AI capabilities relative to human researchers.
The authors contrast the result with their own prior work in which a similar setup — OpenClaw with Opus 4.6 thinking — autonomously built and shipped an iOS app, and with a finding that frontier models on open-source scaffolds now reproduce published research far better than two years ago. They plan follow-up experiments on a larger set of papers using GPT-5.6 Sol, Opus 5 and Fable 5.
Reception
A commenter on the companion essay argued that the evaluation measures the model plus the OpenClaw research scaffold rather than the model in isolation, and asked for release of the full research harness; Kapoor replied (Source: normaltech.ai). The paper anticipates the objection in its scaffold-overhang section and offers the Codex/GPT-5.6 Sol Ultra rerun as its answer.
Provenance
Released artifacts comprise the expert reviews, survey responses, agent repositories and run logs, with reproduction materials at cruxevals.com. The detailed logs and agent-generated paper for one of the two studies are withheld because that paper was not yet public. Funding came from Coefficient Giving and Schmidt Sciences, with OpenAI providing API credits to evaluate GPT-5.6 Sol.
The full author list is Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, and Arvind Narayanan.
Relationships
- contradicts: When AI Builds Itself (The Anthropic Institute, June 4, 2026) — Anthropic's post cites rising success on open-ended agent sessions as self-improvement evidence; this paper finds agents cannot complete open-ended research questions and questions that construct
- supports: Shadow Evaluations — the paper that introduces the method
- related: Recursive Self-Improvement (RSI) — tests the delegation mechanism behind RSI forecasts
- related: AI Benchmarks and Evaluation — argues verifiable benchmarks and blind review each measure something narrower than open-ended research
- related: Reward Hacking — reports no significant reward hacking, against most surveyed expectations
- related: UK AI Safety Institute (AI Security Institute) — two coauthors supplied and reviewed one of the two case studies
- related: Sayash Kapoor, Arvind Narayanan, Claude Fable 5, GPT-5.6 (Sol, Terra, Luna)