A shadow evaluation takes the central research question from a high-quality research paper that is not yet public, gives it to a well-resourced frontier AI agent, and asks the paper's original authors to grade the agent's output as they would a conference submission. The agent "shadows" the original study: it works on the same question as the original authors, without access to their paper or findings. The term and the method were introduced by Kirgis, Kapoor, Schwartz, Rabanser and 20 coauthors in a July 29, 2026 preprint (Can AI agents conduct open-ended AI research? Early evidence from two case studies (Kirgis et al., July 2026)).
Design rationale
The method is proposed as a third approach alongside the two the paper identifies as dominant in evaluations of AI research-and-development automation, each with a limitation the shadow design is meant to address.
Verifiable-task evaluations — RE-Bench, MLE-Bench, MLR-Bench, CORE-Bench, PostTrainBench and similar — ask an agent to improve a fixed metric that an automatic verifier scores. The paper credits these as "objective, repeatable, and cheap to scale" but notes they restrict evaluation "to tasks where success can be measured as a single number," which excludes choosing candidate hypotheses, deciding what evidence would settle a question, or recognizing that an approach has failed.
Blind peer review of AI-generated papers, the approach behind Sakana's AI Scientist-v2 and Intology's Zochi, tests open-ended work but rests on an instrument the paper argues is weak independent of AI. It cites the NeurIPS consistency experiments of 2014 and 2021, in which two committees reviewed the same papers: half the variation in scores was subjective, the committees disagreed on roughly a quarter of accept/reject decisions, and about half of accepted papers would have been rejected on a rerun. A second objection is selective reporting — because generating a paper is cheap, a developer "can submit many papers, report the acceptances, and never disclose the failed attempts."
Against these, the shadow design claims four properties: the questions come from actual conference submissions, so output can be measured against top-conference quality; the findings are not on the web or in training data, so the evaluation is uncontaminated; the graders spent months on the same question and can judge in detail whether the agent made progress; and the method is repeatable, since new unpublished papers with willing authors serve as new test cases and the design carries over to stronger models and scaffolds.
The design also mirrors a specific mechanism in forecasts of recursive self-improvement — that AI accelerates AI research because researchers delegate whole projects to agents and judge whether the returned results advance their work.
Acknowledged weaknesses
The introducing paper sets out the trade-offs rather than claiming superiority, describing the three methods as complementary and noting they "might yield systematically different insights about the rate of progress."
- Small sample. In-depth grading needs both a topic expert and days of review time, which bounds the number of cases. The introducing study covered two papers across five runs.
- Non-blind review. Graders have already answered the question by their own method and know the output is AI-generated, either of which could change their behavior relative to an ordinary reviewer.
- Researcher degrees of freedom. Selecting papers, designing the scaffold and interpreting logs all require judgment, so the evaluators' priors can shape results. The introducing paper adds a disclosure section stating the core team's known position on recursive self-improvement and recommending that such evaluations be conducted by epistemically diverse groups with disagreements surfaced explicitly.
- Evaluation awareness. Agents are told they are being evaluated against a review rubric, on the reasoning that concealment "is increasingly infeasible against capable models" and that disclosure permits precise specification to avoid under-elicitation. See Unverbalized Evaluation Awareness.
- Scaffold attribution. A result measures the model together with its scaffold. The introducing paper addresses this by rerunning one case on a different model and its native scaffold; a commenter on the authors' summary essay nonetheless argued the evaluation measures model-plus-scaffold and asked for release of the full harness (Source: normaltech.ai).
Application to date
The only published application is the introducing study itself, run on two unpublished NeurIPS 2026 submissions with Claude Opus 4.8 on the OpenClaw scaffold at extra-high reasoning, six days of wall-clock time and $3,000 in API credits per run. Both agent papers were unambiguously rejected by the original authors, at overall scores of 2/6 and 1/6, while the agents completed all required engineering without human help. A robustness rerun with GPT-5.6 Sol Ultra on Codex reproduced most failure modes. The authors state an intention to run follow-up shadow evaluations on a larger set of papers using GPT-5.6 Sol, Opus 5 and Fable 5 (Can AI agents conduct open-ended AI research? Early evidence from two case studies (Kirgis et al., July 2026)).
Relationships
- instance-of: AI Benchmarks and Evaluation
- depends-on: Can AI agents conduct open-ended AI research? Early evidence from two case studies (Kirgis et al., July 2026) — the source that introduces the method
- related: Recursive Self-Improvement (RSI) — the forecasting question the method is designed to inform
- related: Unverbalized Evaluation Awareness, Reward Hacking, UK AI Safety Institute (AI Security Institute)