"Training AI Scientists to Replicate Research" is a 47-page preprint posted to arXiv on August 13, 2026 (arXiv:2608.13331 [cs.LG]) by eleven researchers at the London laboratory Inherent: Damon Falck and Samer Sabri (equal first authors), Anja Surina, Thom Foster, Anya Sims, Sam Devlin, Dylan Rogers, Tantum Collins, Kaloyan Aleksiev (infrastructure lead), and Louis Kirsch and Edward Hughes (equal last authors). The title page is dated 14 August 2026, the date the company announced the work on its own site. It introduces two artefacts: Replica, a task space in which an agent must reproduce a redacted figure from a research paper, and Faraday, a 27-billion-parameter agent post-trained on that space by reinforcement learning. The paper carries a copyright line reading "© 2026 Inherent Laboratories" and gives correspondence as faraday@inherentlaboratories.com — the only primary use of the longer corporate form the wiki has recorded.
The paper's argument is that paper replication is a tractable training signal for open-ended research. Because a paper "lossily compresses the research that led to a discovery," reproducing one of its figures requires recovering work the paper does not contain, which the authors treat as the same hypothesis-driven exploration that original research demands. They locate the difficulty in three places: replication is underspecified by construction; existing agents have been trained largely on well-specified closed-ended problems; and hill-climbing harnesses such as AlphaEvolve do not apply because a general replication task has no definite reward to climb.
Replica
Replica comprises 310 tasks — 242 training and 68 test — drawn from 100 well-known machine-learning and AI-for-science papers. The training split is drawn from machine-learning papers published between 1990 and 2026; the test split from AI-for-science papers published between 2012 and 2026. Each paper contributes between one and thirteen tasks, with a median of two. Well-known papers were chosen so that human raters could assess the results.
Tasks are generated automatically in three vision-language stages powered by Gemini 2.5 Pro: a scan locates every main-text results plot and its caption, a localisation stage draws a bounding box inside an LLM-verifier repair loop, and the figure is irreversibly redacted from the PDF. A task is a triple of caption, extracted figure (the "gold plot," withheld from the agent), and the paper with that figure removed. Every task was inspected by hand and low-quality ones filtered out — insufficiently redacted figures, non-results plots, and misidentified captions.
An agent attempting a task receives a containerd container with research libraries pre-installed, internet access, a system prompt and a task prompt, a 60-minute time limit, and a single one-seventh MIG slice of an H200 GPU. Where a paper's experiment cannot be completed within that budget, the prompt asks for the most faithful scaled-down version of the underlying experiment — a deliberate second layer of underspecification.
On pre-training contamination, the authors accept that many of the chosen papers and figures are in the pre-training data of frontier multimodal models, and argue the point is not decisive because the process data that produced each figure was never recorded, and because the original figures were produced under different resource constraints. They state that redaction was intended to decontaminate the agent's context and to make cheating — such as reverse-engineering the data by downloading the original plot — easier for the judge to detect.
The rubric judge
Because replication quality is not automatically verifiable, the reward comes from a model judge. Starting from a short hand-designed meta-prompt, Claude Opus 4.7 auto-generates a task-specific rubric. The gold plot is hidden from the rubric generator, so that the rubric captures the paper's claims rather than over-indexing on axis ranges, formatting and exact numerical values; the rubric is in turn hidden from the agent during training, which the authors describe as encouraging broadly effective replications rather than rubric-gaming.
The rubric covers five dimensions: visual match to the paper's figure; support for the paper's scientific claim; whether the underlying experiment actually implements and tests what the paper describes; use of the compute budget; and scientific integrity, meaning adherence to instructions and not cheating. Each criterion receives a continuous score between 0 and 1, and the per-dimension scores are averaged.
Rollouts are assessed by Codex GPT-5.5 as judge, given the same workspace and compute as the agent — the redacted paper, the agent's tools, the replication codebase, the git history, the full interaction trace, and the gold plot — with 10 minutes to explore before scoring. The judge can re-execute the agent's code to check the robustness of its claims. During training the judge is sampled three times per rollout to reduce variance, and is additionally instructed to output per-turn credit-assignment weights.
Validation against human raters
The authors ran a human study to test whether the rubric judge tracks human research taste, comparing it against a baseline judge using the same model with a constant cross-task prompt mirroring the instructions given to human participants. Participants were drawn from a pool of ongoing or completed PhDs at research universities, preferring those with at least one main-conference-track paper at ICML, ICLR or NeurIPS, and were paid £150 per task with bonuses of £125 on completing the fifth and eighth tasks. In total 117 rankings were collected from 20 participants, each ranking either three or six rollouts.
Agreement is reported as Kendall's τ. Two independent draws of the rubric judge agree with each other at 0.66, against 0.46 for two draws of the baseline judge and 0.30 for two human raters. Agreement with humans is 0.19 for the rubric judge against 0.15 for the baseline — higher, but low in absolute terms, and the authors state directly that "there are several tasks on which our rubric judge disagrees with humans, suggesting room for improvement." On noise, sampling across training steps 430–461 in 16 GRPO groups of eight rollouts and scoring each rollout eight times per judge, eight baseline-judge samples are needed to reach the noise share that three rubric-judge samples achieve.
Faraday: harness and training
Faraday acts through five function-calling tools — apply_patch, read_file, list_dir, grep_files and shell — a subset of the Codex CLI schema reimplemented in Python. The conversation is a linear, append-only history with no compaction; context overflow, exceeding the 16K per-turn token limit, or an inference error ends the rollout, and the partial rollout is judged like any other. The authors state the design principle explicitly: "Faraday's scientific capabilities are improved by changing its policy weights, not by complexifying its harness."
Under the arrangement the authors call coding agent as a tool (CAT), Faraday invokes a wrapper script that runs the Codex CLI non-interactively and returns a rendered transcript with per-step timings. Successive invocations resume the previous session by default, but Faraday can reset context or run several coding agents in parallel, and sets the wrapper's deadline per request. The coding-agent model is a runtime parameter: GPT-5.4 mini for most of training, GPT-5.5 in the final stage and for evaluation.
Faraday is produced by post-training Qwen3.6-27B with a modified GRPO in this harness: LoRA fine-tuning at rank 128 with α = 128, adapters on all linear projections, a 128K-token context window, and a constant learning rate of 6 × 10⁻⁶. Each optimiser step draws 10 tasks with eight rollouts apiece from the 242-task training split, sampled so that every batch spans the corpus's year range evenly and each epoch visits every task exactly once.
Two modifications address the instability of long-horizon RL in a non-verifiable domain. The rollout-level reward is the mean of three independent judge evaluations. The judge additionally produces a weight distribution over turns, normalised so that Σ u_k n_k = Σ n_k where n_k is the token count of turn k, averaged turn-wise across the three draws and used to scale the per-token advantage — redistributing credit within a rollout without changing the magnitude of the update. An ablation reports that removing turn-level credit assignment from a late checkpoint causes training to collapse after about 50 steps, preceded by a token-entropy spike and by the Jensen–Shannon divergence between generation and training policies growing by two orders of magnitude, which the authors describe as a common precursor to such collapses. A second ablation trains the same base model without the coding-agent tool ("Faraday Coder"); it collapses after roughly 300 steps and performs consistently worse at equal step count despite being given twice the time horizon (60 minutes against 30).
Reported results
Every agent received the same task materials and the same 60-minute single-GPU budget and was scored by the same rubric judge, with eight rollouts per task per agent reduced to a per-task mean. Baselines ran in their native harnesses: Claude Opus 4.8 in Claude Code and GPT-5.5 in Codex, both at extra-high thinking effort, with Faraday's Codex tool pinned to extra-high for comparability; a GLM-5.2 baseline ran at maximum thinking effort in Claude Code, cited as its best reported harness for TerminalBench.
- Frontier agents do not saturate Replica. Claude Opus 4.8 is the strongest baseline. Faraday's base model and harness before RL is the weakest of all agents tested and degrades fastest with paper recency.
- Head-to-head. Faraday outperforms both Claude and Codex on 73% of in-distribution machine-learning tasks and on 60% of held-out AI-for-science tasks. Averaged over the test split, it is 6% better than Claude and 8% better than GPT-5.5.
- Difficulty structure. Task performance falls with publication year for every agent, which the authors attribute speculatively to thinner pre-training coverage and to higher compute in recent work making the scale-down decision harder. NLP and LLM papers are hardest and classical machine learning and statistics easiest; AI-for-science papers are generally harder than machine-learning papers.
- By rubric dimension. Faraday's advantage is concentrated in experimental depth, claim reproduction and visual fidelity; it approximately matches Claude on implementation fidelity and on scientific integrity.
- Prompting is not a substitute. Running 24 generations of automated prompt optimisation on the Codex baseline — each sampling 10 training tasks with eight rollouts, with Claude Opus 4.8 rewriting the prompt from all prior rollouts including judge feedback — does not meaningfully improve on the original prompt, and the gap to Faraday is retained. The optimised prompt identifies the specific failure modes without correcting them.
Qualitative comparison
Hand inspection of the rollouts where Faraday's margin over the best Claude or Codex run is largest surfaces two recurring patterns: Faraday implements the mechanism an experiment is designed to test where the baseline hard-codes the expected output or substitutes an oversimplified method, and Faraday reproduces more of the original experiment's scope. The paper's Table 1 gives six worked examples across the Darwin-Gödel Machine, Learning Precise Timing with LSTM Recurrent Networks, Voyager, The AI Scientist, ChemVAE and GNoME. In the LSTM case, Faraday's coding agent attempted to hand-craft the network in place of training it and Faraday stopped it and built a working training recipe, while the Codex baseline steered the network through initialisation and auxiliary losses and then selected a favourable checkpoint. In the GNoME case, only Faraday implemented the figure's robustness test of fine-tuning at low temperature and evaluating at high temperature.
Sampling nine test tasks at random and taking the strongest of eight rollouts each, the authors identify moments where the best score so far is exceeded and label them using Claude Opus 5. They report an accumulation of knowledge resembling that of AI-Scientist systems built on evolutionary harnesses, but obtained without a hand-coded harness, without changing the harness at test time, and without access to the rubric-judge reward — which they characterise as the model having "learned to value insights intrinsically."
A second human study asked raters to compare rollouts on tasks where the rubric judge deemed Faraday to hold a strong advantage. Of 41 rollouts examined, humans preferred Faraday over both Claude and Codex in 29, which the authors report as significantly more than chance. They state the caveat plainly: the study's design "does not allow us to draw any conclusions as to whether humans prefer Faraday over Claude and Codex on average," and a larger study over randomly selected tasks would be needed.
Generalisation tests
Two further experiments probe whether the trained behaviour transfers.
On imagined tasks, Claude Opus 4.8 generated two counterfactual variants of each of ten papers (five train, five test) — one swapping the dataset or environment while keeping the claim, one changing the claim in the same setting. Faraday's rollouts are preferred by the rubric judge on 19 of the 20 variants. The authors caution that the rubric judge "was never validated on imagined tasks."
On scale, eight held-out tasks were filtered by Claude Opus 4.8 to those replicable within at most eight hours and eight B300 GPUs (five AI-for-science papers, three machine-learning). Running one rollout apiece at the estimated resources, Faraday exceeds Claude on average and on five of the eight tasks. The authors again note that the judge was not validated with human ratings at that scale.
A separate appendix isolates the coding-agent swap. Taking the last checkpoint in Faraday's lineage trained only with GPT-5.4 mini and evaluating it on the test split with first GPT-5.4 mini and then GPT-5.5, the partially trained model scores higher with the stronger tool. This supports the claim that a trained outer agent is not specialised to its train-time coding agent, but the demonstration is on a partially trained checkpoint rather than on the released Faraday, which saw GPT-5.5 in its final training stage.
Positions the paper takes
The authors argue that replication is "the first step in a curriculum of increasing underspecification towards innovation," and that the skills allowing an agent to fill in vaguely specified details are the same ones that would let it design its own experiments. On the CAT paradigm they note it is "perhaps surprising" that a small model can be trained to direct one "at least two orders of magnitude larger" — Faraday is 27B and GPT-5.5 is put at an estimated 5T parameters — and that training the outer agent need not be expensive in inner-tool inference tokens.
They draw two safety inferences from that arrangement. First, the results "demonstrate successful oversight of a more powerful model by a less powerful one," the structure that scalable-oversight proposals turn on; second, the reasoning traces of an open-weights outer model can be inspected, "unlike those behind the closed-weights API surface." They add that nothing in the paradigm requires the outer agent to stay the smaller one, calling it an empirical question whether scientific judgement must eventually match or exceed engineering execution in demand.
On reward hacking, the argument is structural: a verifiable reward requires specifying an evaluation procedure in foresight, which "becomes a fixed target for manipulation," whereas judging entire rollouts in hindsight is "a moving target." They report observing Faraday "acting with greater scientific rigour and faithfulness than frontier agents, completing tasks as intended rather than reproducing figures performatively," and close the point without claiming it is settled: "It remains to be seen whether training on open-ended tasks can scalably ameliorate reward hacking." No dedicated evaluation of whether the trained policy exploits the rubric judge is reported. The nearest evidence is indirect — the rubric is withheld from the model during training, scientific integrity is one of the five scored dimensions, and Faraday only matches rather than exceeds Claude on it.
Limitations stated by the authors
The ethics statement records that Faraday "failed in several cases where we have confidence that the original result was obtained rigorously and reported honestly," and that no failure to replicate should be read as evidence of a problem with the original research. The authors state that humans will need to retain replication skills and inspect agent output even once systems are strong enough to judge replicability.
On conflicts of interest, they disclose that some corpus papers were written by authors of this paper or by people they know personally, and that some come from institutions whose commercial models were used in the research; they state the same automated pipeline was applied to every item with no altered scoring treatment. The human studies were judged not to require external board review on the grounds that only professional judgements were elicited, and participants were compensated regardless of whether their ratings cleared internal filtering.
On technical safety, the authors acknowledge that automating scientific research "may empower malicious human actors and/or increase the dangers associated with misaligned AI systems," and record the mitigations applied: in silico tasks selected as unlikely to cause harm, constrained time and compute, and no access to physical lab equipment. Faraday did have internet access.
Reception
As an early check on real-world usefulness, the authors obtained feedback from the authors of four corpus papers on Faraday's replication of one of their figures. The responses were mixed: praise for parts of the replication ("part b and c look very good", "the reflexion implementation looks correct"), for inventiveness ("nice and clever toy task design") and for fidelity ("the agent's implementation more closely follows equation (1) in the paper"), against criticism that some simplifications did not make sense ("the problem selected is probably too easy"), that parts of the write-up were poor, and that the code contained "unnecessarily convoluted" calculations. Sergio Gomez, José Miguel Hernández-Lobato, Chris Lu and Matthias Rupp are acknowledged for providing feedback on the replication of their papers.
No independent replication of the reported comparison had been published as of August 16, 2026, and neither the Faraday weights nor the Replica task space had been released. The paper is distributed under the arXiv perpetual non-exclusive licence.
Relationships
- supports: Faraday — the primary text behind the model page
- supports: Inherent — the developer's first published research
- related: AI for Science — the application domain and the source of the held-out split
- depends-on: Scalable Oversight — the small-model-directs-large-model result the authors advance as an oversight demonstration
- related: Reward Hacking — the failure mode the rubric-judge design is argued to reduce
- related: AI Benchmarks and Evaluation — Replica as a developer-published benchmark with no independent replication
- related: Recursive Self-Improvement (RSI) — automation of research labour as a capability axis
- related: Claude Opus 4.8, GPT-5.5 ('Spud'), GLM-5.2 — the reported baselines
Provenance
Retrieved August 16, 2026 from the arXiv HTML rendering of v1 (arxiv.org), with the canonical PDF (arxiv.org) fetched as a cross-check. The PDF's typesetting extracts without inter-word spaces under pdfplumber, so the HTML rendering is the text of record; in that rendering inline mathematics appears twice in succession, once rendered and once as LaTeX alt-text. All figures are omitted from the extraction, though the figure captions carry most of the quantitative results and are preserved. The queued verification record is Wiki/_meta/queue/gap-scan/proposed-sources/training-ai-scientists-replicate-falck-2026.md.