Faraday

Faraday is a 27-billion-parameter agent published by the London laboratory Inherent, described by its developers as an "AI Scientist" post-trained to replicate results from research papers. The accompanying paper, "Training AI Scientists to Replicate Research," was submitted to arXiv on August 13, 2026 and announced on the company's site on August 14 (Training AI Scientists to Replicate Research; Source: inherentlabs.ai). Its authors report that it surpasses Claude Opus 4.8 and GPT-5.5 on held-out replication tasks despite being far smaller, by using a frontier coding agent as a tool rather than performing the coding itself. The model is named after Michael Faraday, whose 1821 manual replication of earlier electromagnetism experiments the company's research post recounts as its framing example.

Infobox

Replica

Replica is the task space the model is trained and evaluated on, introduced in the same paper. Each task requires an agent to replicate a figure from a research paper within a limited time and compute budget and without access to the original plot. The initial suite comprises 310 tasks drawn from 100 machine-learning and AI-for-science papers, spanning domains including natural-language processing, materials science and weather forecasting (Source: inherentlabs.ai). The paper splits these into 242 training tasks, drawn from machine-learning papers published between 1990 and 2026, and 68 test tasks, drawn from AI-for-science papers published between 2012 and 2026; each paper contributes between one and thirteen tasks, with a median of two (Training AI Scientists to Replicate Research). Tasks are generated automatically in three vision-language stages powered by Gemini 2.5 Pro — a scan for main-text results plots and captions, a bounding-box localisation stage inside an LLM-verifier repair loop, and irreversible redaction of the figure from the PDF — with every resulting task inspected by hand and low-quality ones discarded. The authors report that recent agents from the leading laboratories do not saturate the task space.

Each attempt runs in a containerd container with research libraries pre-installed and internet access, under a 60-minute limit on a single one-seventh MIG slice of an H200 GPU. Where a paper's experiment cannot fit that budget, the prompt asks for the most faithful scaled-down version of the underlying experiment, which the authors describe as a deliberate second layer of underspecification (Training AI Scientists to Replicate Research). Neither the weights nor the task space had been released as of August 16, 2026.

The design rationale the authors give is that reproducing a published figure requires recovering work the paper does not contain: papers describe what worked rather than the negative results that preceded it, so an agent must perform hypothesis-driven exploration of its own. On that reading replication is a tractable proxy for open-ended research and the basis of a curriculum that can be made progressively more underspecified — withholding more than a single plot, tightening or relaxing resource constraints, and ultimately presenting imagined papers (Source: inherentlabs.ai).

Capabilities and reported results

The authors report that Faraday produces more faithful replications than the Claude Opus 4.8 and GPT-5.5 baselines "for every category of paper in the task suite," with the advantage described as most pronounced in meta-learning, structural biology and materials science, and that it struggles less than the baselines with recent research unseen by the base model during pre-training (Source: inherentlabs.ai). Baselines were run in their native harnesses — Claude Opus 4.8 in Claude Code and GPT-5.5 in Codex — with thinking effort set to extra high, and Faraday's Codex tool pinned to extra high for comparability. A GLM-5.2 baseline was run at maximum thinking effort in the Claude Code harness. Every agent received the same task materials and the same 60-minute single-GPU budget and was scored by the same judge, with eight rollouts per task reduced to a per-task mean (Training AI Scientists to Replicate Research).

The paper reports the comparison numerically. Faraday outperforms both Claude Opus 4.8 and GPT-5.5 on 73% of in-distribution machine-learning tasks and on 60% of held-out AI-for-science tasks, and is on average 6% better than Claude and 8% better than GPT-5.5 on the test split. Claude Opus 4.8 is the strongest baseline; Faraday's base model in the same harness before reinforcement learning is the weakest agent tested. Decomposing the judge score into its five rubric dimensions places Faraday's advantage in experimental depth, claim reproduction and visual fidelity, with implementation fidelity and scientific integrity approximately matching Claude (Training AI Scientists to Replicate Research).

Difficulty is structured along two axes the authors report for every agent tested: performance falls with a paper's publication year, which they attribute speculatively to thinner pre-training coverage of recent work and to higher compute requirements making the scale-down judgement harder; and difficulty varies by research topic, with NLP and LLM papers hardest, classical machine learning and statistics easiest, and AI-for-science papers generally harder than machine-learning papers.

To test whether the advantage is reachable by prompting, the authors ran 24 generations of automated prompt optimisation on the Codex baseline, with Claude Opus 4.8 rewriting the prompt from all prior rollouts including judge feedback. The optimised prompt identifies the specific failure modes without correcting them and does not perform meaningfully better than the original, leaving the gap to Faraday intact (Training AI Scientists to Replicate Research).

Two generalisation tests are reported beyond the main comparison. On twenty counterfactual task variants generated from ten papers — one swapping the dataset or environment, one changing the claim — the judge preferred Faraday's rollouts on 19 of 20, though the authors note the judge "was never validated on imagined tasks." On eight held-out tasks run at full scale with up to eight hours and eight B300 GPUs, Faraday exceeded Claude on average and on five of the eight, with the same caveat that the judge was not human-validated at that scale.

The figures are the developer's own. No independent evaluator had reproduced the comparison as of August 16, 2026, which is the same posture recorded for other single-lab benchmark claims on AI Benchmarks and Evaluation.

Training and architecture

Faraday is produced by post-training Qwen3.6-27B with a modified GRPO in the paper's own harness, using LoRA at rank 128 with α = 128 on all linear projections, a 128K-token context window and a constant learning rate of 6 × 10⁻⁶. Each optimiser step draws ten tasks with eight rollouts apiece from the 242-task training split, sampled so every batch spans the corpus's year range evenly and each epoch visits every task once (Training AI Scientists to Replicate Research). The harness itself is deliberately minimal — five function-calling tools (apply_patch, read_file, list_dir, grep_files, shell) taken as a subset of the Codex CLI schema, a linear append-only conversation with no compaction, and a 16K per-turn token limit whose breach ends the rollout, which is then judged like any other. The authors state the design principle as improving scientific capability "by changing its policy weights, not by complexifying its harness."

Because replication quality is not automatically verifiable, the reward signal comes from a large-language-model judge. The authors report designing an auto-generated rubric-based judge and validating it against a human study, and state that per-task rubrics produce greater consistency and lower noise — measured by Kendall's tau agreement with human raters — than an LLM judge baseline without rubrics (Training AI Scientists to Replicate Research; Source: inherentlabs.ai). Rubrics are auto-generated by Claude Opus 4.7 from a hand-designed meta-prompt, with the original figure hidden from the generator and the rubric in turn hidden from the agent during training; scoring is performed by Codex GPT-5.5 given ten minutes and the same workspace as the agent, including the ability to re-execute its code. Two further training-time modifications are reported to address long-horizon instability: multi-sample aggregation of judge outputs (the mean of three independent evaluations) and turn-level credit assignment, in which the judge emits per-turn weights normalised to leave the overall update magnitude unchanged. An ablation reports that removing turn-level credit assignment collapses training after about 50 steps, and that a variant trained without the coding-agent tool collapses after roughly 300 steps while performing worse at equal step count despite twice the time horizon.

The authors frame the criterion the judge is meant to capture as broader than pixel-level reproduction, listing experimental design, scientific practice, faithfulness to the original paper's claims, and effective use of available resources — described collectively as "research taste." The rubric operationalises this in five dimensions scored continuously between 0 and 1 and averaged: visual match to the paper's figure, support for the paper's scientific claim, whether the experiment actually implements and tests what the paper describes, use of the compute budget, and scientific integrity (Training AI Scientists to Replicate Research).

The human validation drew 117 rankings from 20 participants, selected from ongoing or completed PhDs and preferring those with a main-track ICML, ICLR or NeurIPS paper. Two independent draws of the rubric judge agree with each other at Kendall's tau 0.66, against 0.46 for the baseline judge and 0.30 for two human raters; agreement with humans is 0.19 for the rubric judge against 0.15 for the baseline. The authors record that the rubric judge disagrees with humans on several tasks, "suggesting room for improvement in future work" (Training AI Scientists to Replicate Research).

Tool use and scale asymmetry

Faraday calls GPT-5.5 Codex as a tool rather than writing the replication code itself, a pattern the authors name "coding agent as a tool" and compare to a human scientist's use of a coding agent. The company's announcement describes the 27B model as directing a model "several orders of magnitude larger"; the paper puts the gap at "at least two orders of magnitude," giving GPT-5.5 an estimated 5 trillion parameters against Faraday's 27 billion (Source: inherentlabs.ai; Training AI Scientists to Replicate Research). Mechanically, Faraday invokes a wrapper script that runs the Codex CLI non-interactively and returns a transcript with per-step timings; it can resume the previous session, reset context, run several coding agents in parallel, and set the wrapper's deadline per request.

The company's announcement states that Faraday generalises at test time to directing GPT-5.5 Codex after having been trained against the smaller GPT-5.4 mini. The paper's own account is narrower: GPT-5.4 mini backed the tool for most of training but GPT-5.5 was used in the final stage and for evaluation, and the swap experiment is run on the last checkpoint in the lineage trained only with GPT-5.4 mini, which scores higher on held-out tasks when given the stronger tool. The demonstration therefore concerns a partially trained checkpoint rather than the released model (Training AI Scientists to Replicate Research). The authors draw the inference that the value of scientific judgement rises as frontier coding agents improve, while noting that nothing in the paradigm requires the outer agent to remain the smaller model.

The paper contrasts Faraday with prior AI-scientist systems on two design points: it requires no hand-coded evolutionary harness, and it has no test-time reward, which the authors characterise as the model having learned to value discoveries intrinsically. That claim rests on sampling nine test tasks at random, taking the strongest of eight rollouts each, and labelling with Claude Opus 5 the moments at which the best score so far is exceeded; the authors report an accumulation of knowledge resembling evolutionary-harness systems without a hand-coded harness, without test-time harness changes, and without the model having access to the judge's reward. The abstract frames the work as "a stepping stone towards AI agents capable of long-horizon scientific innovation without requiring complex harnesses" (Training AI Scientists to Replicate Research).

Hand inspection of the rollouts with the largest margins surfaces two recurring differences from the baselines: Faraday implements the mechanism an experiment is designed to test where a baseline hard-codes the expected output or substitutes an oversimplified method, and it reproduces more of the original experiment's scope. A second human study compared rollouts on tasks where the judge deemed Faraday to hold a strong advantage; of 41 rollouts examined, human raters preferred Faraday to both baselines in 29. The authors state that the study's design does not support any conclusion about average human preference across tasks (Training AI Scientists to Replicate Research).

Safety and evaluations

No system card, safety framework, or third-party evaluation accompanied the release, and Inherent has published no responsible-scaling or release policy. The company states that it is investigating how the methods behind Faraday "might advance scalable oversight and ameliorate reward hacking," and describes the arrangement in which a smaller model directs a larger one as a form of scientific oversight (Source: inherentlabs.ai). The paper puts the claim more directly, stating that the results "demonstrate successful oversight of a more powerful model by a less powerful one" and adding that an open-weights outer model's reasoning traces can be inspected "unlike those behind the closed-weights API surface" (Training AI Scientists to Replicate Research). A system in which a weaker model supervises a stronger one is the structural pattern scalable-oversight proposals turn on; the authors report it as a capability result rather than as an oversight evaluation, and publish no adversarial or misuse testing.

Reward hacking is directly implicated by the training design: the reward is a model-generated rubric score for a task whose ground truth — the original figure — is withheld from the agent. The authors argue structurally that hindsight judgement of whole rollouts is "a moving target" where a verifiable reward "becomes a fixed target for manipulation," and report observing Faraday complete tasks as intended "rather than reproducing figures performatively," while stating that "it remains to be seen whether training on open-ended tasks can scalably ameliorate reward hacking." The rubric-consistency and judge-agreement measurements the paper reports address reward noise; no dedicated evaluation of whether the trained policy exploits the judge is reported. The nearest evidence is indirect — the rubric is withheld from the model during training, and scientific integrity is one of the five scored dimensions, on which Faraday matches rather than exceeds Claude (Training AI Scientists to Replicate Research).

The paper's ethics statement records that Faraday "failed in several cases where we have confidence that the original result was obtained rigorously and reported honestly," and states that no such failure should be read as evidence against the original research. On technical safety, the authors acknowledge that automating research "may empower malicious human actors and/or increase the dangers associated with misaligned AI systems," and record the constraints applied: in silico tasks judged unlikely to cause harm, limited time and compute, and no access to physical laboratory equipment. Faraday did have internet access during rollouts.

Availability

The weights were not released and no API, pricing, or access programme was announced. The Replica task space was likewise not released. The paper, its PDF, HTML rendering and TeX source are available from arXiv under a perpetual non-exclusive distribution licence (Training AI Scientists to Replicate Research). The authors invite readers with ideas for using Faraday to write to faraday@inherentlaboratories.com.

Reception

As an early check on usefulness, the authors obtained feedback from the authors of four papers in the Replica corpus on Faraday's replication of one of their figures. The responses were mixed: praise for parts of the replication ("part b and c look very good", "the reflexion implementation looks correct"), for the inventiveness of a task design, and for fidelity to the source method, against criticism that some simplifications did not make sense ("the problem selected is probably too easy"), that parts of the write-up were poor, and that the generated code contained "unnecessarily convoluted" calculations. Sergio Gomez, José Miguel Hernández-Lobato, Chris Lu and Matthias Rupp are acknowledged for providing that feedback (Training AI Scientists to Replicate Research).

Relationships

• related: Inherent — developer
• related: Training AI Scientists to Replicate Research — the primary text describing the model and the Replica task space
• related: AI for Science — the application domain
• depends-on: Scalable Oversight — the small-model-directing-large-model arrangement the authors report
• related: Reward Hacking — the failure mode the rubric-judge design is aimed at
• related: AI Benchmarks and Evaluation — Replica as a developer-published benchmark without independent replication
• related: Recursive Self-Improvement (RSI) — automation of research work as a capability axis
• related: Claude Opus 4.8, GPT-5.5 ('Spud') — the reported baselines

Open questions

• The reported results are the developer's own; no independent evaluator had reproduced the comparison as of August 16, 2026, and neither the weights nor the Replica task space had been released for others to run.
• Whether Faraday is the same artefact as the "Faraday" platform Inherent announced at its May 2026 launch is not stated in any source.
• Whether the trained policy exploits the rubric judge is not evaluated in the reported work.
• The rubric judge's agreement with human raters (Kendall's tau 0.19) exceeds the baseline judge's but is low in absolute terms, and the authors do not report what score would count as sufficient agreement for a training reward.

Sources

• Training AI Scientists to Replicate Research — Falck, Sabri, Surina, Foster, Sims, Devlin, Rogers, Collins, Aleksiev, Kirsch and Hughes, "Training AI Scientists to Replicate Research," arXiv:2608.13331 [cs.LG] (submitted 2026-08-13)
• Inherent / "Training AI Scientists to Replicate Research" (2026-08-14) — inherentlabs.ai
• arXiv abstract record for 2608.13331, retrieved 2026-08-16