Primary text: anthropic.com Authors: Marina Favaro and Phillie Wright, The Anthropic Institute, with research direction from Jack Clark. Published September 17, 2026.
"Measurements for understanding the pace of AI development inside frontier labs" is a post published by Anthropic's Anthropic Institute on September 17, 2026. It proposes three measurements of how frontier AI models are built — the extent to which AI performs AI research and development, the company's ability to oversee and intervene in the actions of its internal AI agents, and the share of compute allocated to safety work — and publishes a snapshot of each from inside Anthropic, together with a methodological appendix. Its headline figures are that, as of August 2026, Claude "leads" 26% of Anthropic's AI R&D work and is not operating fully autonomously for any measured subset; that about 30,000 agents were working at any one time on the company's most-used internal platform; and that about 6% of AI R&D compute went to safety work over one week in July 2026. It is the Institute's second disclosure of quantified internal AI R&D data, after "When AI Builds Itself" in June 2026. The figures are Anthropic's own measurements of its own activity, largely produced by Claude, and the post reports no outside verification of them.
Purpose and framing
The post opens with the claim that AI systems "are becoming exponentially more powerful and have begun to automate more of the process of building themselves," and states that "as the world considers slowing the pace of frontier AI development," linking Dario Amodei's essay "We Must Pace the Frontier", "the public needs more information." It presents the measurements as tools for giving "the public, third parties, and governments better visibility into the pace of AI development inside frontier labs," and for each one describes what was measured, what the measurement showed, and what it would take to publish it regularly in a form others can verify.
The measurements concern how models are built rather than what they can do. The post distinguishes them from capability evaluations, which Anthropic publishes separately through its Responsible Scaling Policy risk reports, and argues that understanding the production process gives "a better chance of correlating model inputs, like compute, with model outputs, like capabilities." It connects them to Anthropic's Advanced AI Framework (AAIF), which proposes transparency obligations governments could require, and describes the measurements and the framework together as "a starting point for monitoring the pace of AI development from outside the labs."
The post states that Anthropic "would expect these numbers to shift if there were coordination on pacing the frontier, as called for by Anthropic CEO Dario Amodei." It restates the company's plan to embed independent third-party evaluators from multiple organizations at Anthropic, with access to internal processes, systems and data "comparable to what internal risk assessment teams have," to verify safety practices, report incidents, and monitor metrics such as the ones in the post.
Measurement 1: AI-led AI R&D
Rationale. The post argues that using AI to build future models lets labs in democratic countries develop more capable models more quickly and conduct more safety testing before release, but that "models accelerating their own development could make it more challenging for humans to understand or control these systems." It presents the metric as a way to understand "how close the world is to reaching recursive self improvement," which it defines as "a model fully autonomously building its successor" (Recursive Self-Improvement (RSI)).
The index. Anthropic built a prototype index of how much of its AI R&D is performed by Claude, called the Anthropic R&D Automation Index (Anthropic R&D Automation Index). It catalogues every kind of AI R&D work done at the company, rates how automated each task is, and aggregates the ratings. The ratings use an automation scale developed by Epoch AI that measures "Automation Level" (AL) from AL0 (no AI involvement) to AL5 (AI operates fully autonomously, with no human in the loop); the six levels are no AI involvement, minimal AI involvement, AI assists, AI collaborates, AI leads, and AI is autonomous. At AL3 AI "collaborates," doing large chunks of work under close human direction; at AL4 it "leads," completing most of a task end-to-end from a high-level prompt while a human supervises.
A footnote illustrates the levels with a broken nightly data pipeline. At AL3 an engineer brings Claude logs from the failed runs, perhaps with a hypothesis already formed; Claude may interview the engineer for context, then investigates and fixes, stopping when a further problem appears so the engineer can decide how to handle it; and the engineer may review the change line by line, rerun the pipeline and deploy it. At AL4 the engineer hands Claude the failure alert; Claude finds the failing stages and cause, writes and tests the fix, handles surprises itself, reruns the pipeline on a copy of the data, compares the output with the last good run and writes up what it changed, but does not deploy; the engineer reads the write-up and decides whether it ships. At AL5, "a level we have not yet reached," Claude would monitor for failures itself and design, test and deploy the fix, with a human involved only if they wanted to be.
Findings. As of August 2026:
- Claude is not operating fully autonomously for any measured subset of AI R&D work.
- Claude "leads" 26% of Anthropic's AI R&D work.
- The share of work at or above "AI collaborates" is above 90%.
The post's accompanying chart is captioned "Claude now leads 26% of Anthropic's model R&D tasks, up from under 1% in February 2026." The February figure appears only in the chart caption, not in the body text.
Reporting proposal. The post states that any frontier developer could publish these measures regularly using a public methodology, allowing comparison over time and potentially across labs. It names two obstacles to cross-lab comparison: the lack of a common methodology, and the use of a developer's own models to evaluate its systems, so that "the 'judge' model could make the same kinds of errors as the model it is checking." It proposes that a developer's measurements could be verified by a third party or by other developers' models, "with guardrails in place to limit the exchange of competitively sensitive data," and that such measures "could also become the trigger for stronger requirements, like a fixed testing window before a new model is used for further AI R&D."
Measurement 2: Oversight of AI agents
Rationale. Anthropic employees, "like other frontier developers," increasingly delegate tasks to agents that work semi-autonomously for long stretches and delegate work to one another. As work moves from "AI collaborates" toward "AI leads," the post argues, agents could make more consequential decisions, "such as which research direction to pursue next." It states that "in our monitoring data to date, individual agents rarely misbehave," but that "when there are millions or billions of agents operating in the economy, even rare events can happen regularly."
Metrics. The post defines three metrics: coverage, the share of an agent's actions that pass through a monitor before or after execution; review latency, the time between an action and its review, first by an automated monitor and then by a human; and escalation rate, the share of agent activities blocked or redirected (online monitors) or flagged for further review (offline monitors).
Findings. As of August 2026, approximately 30,000 agents were doing research and engineering work at Anthropic at any one time on its most-used internal platform; the measurements cover that platform only. Two kinds of monitor constrain these agents:
| Online monitor | Offline monitor | |
|---|---|---|
| Purpose | Checks agent actions in real time and blocks dangerous actions or steers misbehavior automatically; built for harms that would happen quickly and could not be undone, such as an agent copying model weights out of Anthropic's systems | Reads agent activity after the fact; built for dangerous behaviors that develop slowly and can be reversed, such as signs of misalignment. Flags are used to improve mitigations such as classifiers and deterministic controls; in a large imminent danger Anthropic states it would shut down the dangerous parts of the agent's deployment |
| Coverage | 100% of these agents' actions pass through it before execution, usually within seconds | 100% of these agents' actions are ingested after they are taken |
| Review latency | Automated review before the action; humans review any blocked action within one week | Roughly 100,000 transcripts flagged per week are reviewed by multiple classifiers for false positives and initial oversight review; most flags are resolved there, and the highest-priority flags, approximately 50 per week, are escalated to human review |
| Blocking or escalation rate | Of more than a billion decisions by research and engineering agents analyzed over August 2026, 0.002% (about 1 in 47,000) were blocked | Roughly one to two transcripts in every thousand are flagged for further review |
Reporting proposal. Any developer running agents on its own research and engineering workloads, the post states, could publish coverage, review latency and escalation rate, which "would allow society to see whether oversight is keeping pace with the growing role of AI in AI R&D." It states that Anthropic published all of these measurements in its August 2026 risk report (that report's coverage date is July 15, 2026, before the August 2026 period of the figures above). It adds that developers should share how often known agent misbehavior is caught by monitors and have third parties test this independently, and records that METR has previously "independently red-teamed our offline monitoring platform." See Post-Deployment AI System Monitoring.
Measurement 3: Compute allocation
Rationale. Developers use compute to build more powerful models, serve customers, and do safety work, for which the post gives three examples with links to Anthropic research: auditing a model's "thoughts" (Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations), training model organisms to study misalignment (Natural Emergent Misalignment from Reward Hacking in Production RL (MacDiarmid, Wright, Uesato et al., Anthropic)), and evaluating whether a model can be safely deployed. The post calls compute "among the most verifiable inputs to the AI R&D process," which "could be a critical lever in a future pacing effort," and states that a coordinated pacing effort "could encourage companies to increase the compute allocated to safety across the industry" (Compute Governance).
Findings. Anthropic examined how it used all of its compute from July 13 to July 20, 2026, sorting every workload into a small number of categories and asking how much of the compute going to AI R&D was safety work. Over that week about 6% of the compute that went to AI R&D, and about 12% of the compute that went to AI-driven AI R&D, was allocated to safety. The post calls these "deliberately conservative estimates": a token used to advance capabilities as much as safety was not counted as safety, and the figures exclude safeguards classifiers, which it describes as "a separate, comparable amount of compute." A footnote states that Anthropic manages compute as "a single, fungible pool" directed dynamically, so the figures are "a snapshot of how capacity happened to be directed in one week, not a fixed allocation," and that the engineering categories do not correspond to how expenses are classified.
The post states that safety research tends to use less compute than frontier training runs because it consists of individual researchers designing experiments that are time-consuming but not compute-intensive, so compute is "an imperfect proxy" for a company's focus on safety; the metric's value lies less in the absolute numbers than in comparing "like with like, across developers and over time."
Reporting proposal. Any frontier developer could publish the share of its AI R&D compute that goes to safety, with category definitions published alongside and the classification checked by an independent third party. Because safety research is hard to distinguish from capabilities research and "each developer will be tempted to draw the line generously," the post states that "the burden of proof should sit with the developer to show that work is safety-related," and that developers, governments and researchers would benefit from converging on a shared definition in advance. It suggests the measure could inform "a lab's commitments about the share of compute going to safety research, or limits on the share of compute going towards AI research agents."
Methodology
The R&D Automation Index
The index requires a complete map of AI R&D tasks, a way to rate automation, and a way to weight tasks. For each week of July 2026, Anthropic randomly sampled 20% of staff from each department in the model R&D loop; a Claude research agent reviewed each sampled person's week using Slack and internal documentation and listed the tasks they worked on, producing a flat list of about 15,000 granular model R&D tasks. Claude then organized these into a hierarchical tree, from all model R&D at the root through areas such as training and product, then pretraining and reinforcement learning, down to specific kinds of work. The tree has 542 nodes, of which 378 are leaves, such as "eval platform defect diagnosis and fixes," "RL sandbox egress and network policy," and "serving incident postmortems." The tree is frozen so that every measurement runs against the same basket of work.
For each node, a Claude agent researched how that kind of work is done across the company — who does it, with what tools, and how much AI performs — and an independent Claude judge read the evidence and assigned one of the six automation levels. When rating a given month, the rating agents may see only evidence from that month or earlier. Nodes are weighted by person-time: each sampled person carries one unit of weight per week, split evenly across the tasks they worked on that week (four tasks give 0.25 each; ten give 0.10 each), and a category's weight is the sum of the weights assigned to it. The post calls this "a crude approximation" that on average assigns more weight to categories many people work on.
Validation and limits. To check the judge, staff who own the relevant work areas rated the relative automation of their areas without seeing the evidence the models had gathered or how it had been judged. Model-versus-human exact agreement was 59%, against 35% for human-versus-human, and model and human ratings were within one level of each other 97% of the time; the post notes "real room for disagreement on borderline cases, such as where exactly 'AI collaborates' ends and 'AI leads' begins." Because the basket is frozen on a July 2026 baseline, a rising index shows that the work humans were doing then is being automated but not whether humans have shifted onto new kinds of work. To test this, Anthropic built an alternate tree from January 2026 data and compared new tasks arriving each month from February to July 2026 against it, finding "no rise in the number of 'novel' tasks" between the January and July baskets. It plans to rebuild the basket periodically and re-version published numbers.
Oversight of agents
Two design choices in Anthropic's most commonly used internal agent scaffold are described as aiding detection of "unknown unknowns" and letting agents monitor each other. The first is individual agent identity: each agent has an identity to which all the data it creates is tied, which lets it treat another agent's output "as a claim to check rather than a thought of its own," reduces the risk of correlated actions, and persists through model upgrades so that an agent's record is continuous; every action can be audited per agent over time. The second is open communication: agents communicate through a shared, open messaging system in which every message is tied to its author's identity and can link to original references, so agents can correct each other's errors, and all messages are cross-referenced with agent transcripts so that behavioral monitoring can traverse communication between agents.
The post states that "at this scale of agent activity, it is hard to be certain we have a complete picture of all possible kinds of behaviors," and that "we have been doing this kind of monitoring for months, not years."
Compute allocation
The measurement stitched together existing capacity-planning tools that periodically sample accelerator usage and tag workloads with best-effort labels (research and model development, internal usage, first-party inference and so on), with usage on third-party cloud compute reported by the providers. Claude then classified each workload as safety work or AI R&D using a prompted classifier. Safety work was defined as "work whose dominant purpose is making AI systems safer, more understandable, or more secure"; everything else, including capability research, production model training, product development and developer tooling, counted as AI R&D, as did work that helps capability as much as safety.
For research training and evaluation runs, the classifier read each run's metadata and code and returned a classification, justification and confidence level. Of the week's almost 10,000 runs, about 14% were sampled, weighted toward the runs that used the most compute. For inference by AI research agents, a variant of the classifier read the session transcript; where transcripts were inaccessible, usually because the work was compartmentalized, sessions were classified by the user's team or defaulted to AI R&D. The post excerpts the classifier prompt, which describes the task as an internal audit "to produce a public-facing breakdown of the usage of all of our AI accelerator chips." It reports that an extensive written definition with boundary cases brought the classifier within "one or two percentage points" of human reviewers, that some cases could not be determined "even after several hours of human review," and that scalable-oversight research is an example of work whose classification is unclear. It describes its definition as "one reasonable choice among many" that another developer or a regulator might draw differently.
Three further limitations are stated: the underlying labels are set by automated rules or occasionally by users and are "best-effort, not verified"; one week is enough to show the measurement can be made but "not enough to show a meaningful trend"; and compute share measures only spending, so that a more efficient safety classifier or a faster production inference stack lowers the safety share without reducing safety work. Anthropic plans to refine the pipeline so that an independent third party could re-run the classifier on a random subsample of jobs and transcripts and check both the sorting and the totals.
Scope of the figures
| Figure | Value | Denominator and period |
|---|---|---|
| Work Claude "leads" (AL4) | 26% | Anthropic's AI R&D work, as of August 2026; basket frozen from July 2026 |
| Work Claude "leads," earlier | under 1% | February 2026 (chart caption only) |
| Work at or above "collaborates" (AL3+) | above 90% | As of August 2026 |
| Work at AL5 | none | No measured subset |
| Concurrent agents | approximately 30,000 | Most-used internal platform only, August 2026 |
| Online-monitor block rate | 0.002% (about 1 in 47,000) | More than one billion agent decisions, August 2026, same platform |
| Offline-monitor flags | roughly 100,000 transcripts per week; one to two per thousand | Same platform; approximately 50 per week escalated to humans |
| Safety share of compute | about 6% | Compute going to AI R&D, July 13–20, 2026; excludes safeguards classifiers; dual-use work counted as AI R&D |
| Safety share of compute | about 12% | Compute going to AI-driven AI R&D, same week and definitions |
| Index validation | 59% exact, 97% within one level | Claude judge against staff ratings; human–human exact agreement 35% |
Relation to other disclosures and proposals
The post is the second Anthropic Institute disclosure of internal AI R&D data. "When AI Builds Itself" (June 4, 2026), co-written by Favaro and Clark, reported that more than 80% of code merged into Anthropic's codebase was Claude-authored as of May 2026 and that code merged per engineer per day was 8× the 2021–2024 level, and committed the Institute to building the verification systems a coordinated slowdown would require (When AI Builds Itself (The Anthropic Institute, June 4, 2026)). Where the June essay measured code authorship and volume, the September post rates automation task by task and adds oversight and compute measures.
OpenAI published internal measurements of research acceleration eleven days earlier, on September 6, 2026, covering researcher inference spend, the agent-to-human workday ratio, experiment velocity and agent task-success rates, and classified agent-token growth against an Epoch AI taxonomy of AI R&D work (Research acceleration: The view inside OpenAI (OpenAI, September 2026)). Both disclosures draw on Epoch's June 2026 proposal for tracking AI research automation, OpenAI on its task taxonomy and Anthropic on its automation scale (Epoch AI). They measure different quantities by different methods; the Anthropic post itself names the lack of a common methodology as the first obstacle to cross-lab comparison.
The post's references to pacing tie it to Amodei's essay of September 12, 2026, whose first step is the embedded external evaluators the post restates (We Must Pace the Frontier (Amodei, September 2026)). Its reporting proposals bear on several legislative and policy proposals that it does not cite: the Self-Improving AI Monitoring Act discussion draft, which would direct NIST to assess trends in autonomous AI research and development; the Institute for Progress report recommending that frontier companies publicly share information on trends in AI R&D automation (How Should the US Prepare for Increasingly Automated AI R&D? (IFP, August 2026)); and Americans for Responsible Innovation's federal blueprint, which proposes a mandatory quarterly disclosure programme on the automation of AI R&D. Its comparison of proposals for slowing frontier development is at Coordinated slowdown proposals compared.
Reception
Implicator.ai (Marcus Schuler) reported the post on September 18, 2026 under the headline "Anthropic Says Claude Leads 26% of Its AI Research, Up From Under 1% in February," linking it as the "Anthropic R&D Automation Index," and noted that the OpenAI and Anthropic disclosures measure different activities by different methods, so their figures are not directly comparable (Source: implicator.ai). No independent verification of the figures had been reported as of September 19, 2026.
Provenance
Captured on 2026-09-18 from the canonical anthropic.com page (HTTP 200); the main-content text, both footnotes, the appendix including the excerpted classifier prompt, and the credits were transcribed, with one chart represented by its caption. The page carries no date line. The September 17, 2026 date rests on three traces recorded in the raw file: the page was retrieved in full by a scheduled digest run at 22:05 PT on September 17 and listed by Techmeme that day; the anthropic.com site-settings record listing the route was last updated at 17:02 PT on September 17; and the earliest secondary coverage retrieved was published on September 18. A digest account dating the post to September 18 took Implicator's publication date. The Anthropic Institute index page lists the post as "Illuminating the Frontier."
The credits name Favaro and Wright as co-authors, with editorial support from Santi Ruiz, Adam Farina and Sarah Pollack; research direction from Jack Clark; feedback from Dan Altman, Kerry Persen, AJ Kourabi, James Bradbury, Holden Karnofsky, Kevin Troy and Avital Balwit; technical proofs of concept by Jun Shern Chan, Brian Calvert, Francesco Mosconi, Henry de Valence, Fabien Roger and Joe Benton; and a red-teaming workshop with external experts organized by Alex Cloud and Andrea Vallone. Nate Rush, Eli Lifland and Peter Wildeford are thanked separately for feedback. The Epoch AI post on which the automation scale is based (epochai.substack.com) is not held in the source record.
Relationships
- supports: Recursive Self-Improvement (RSI) — task-level measurement of how far AI performs Anthropic's AI R&D, and of distance from full autonomy
- instance-of: Anthropic R&D Automation Index — first publication of the index
- supports: Post-Deployment AI System Monitoring — coverage, review latency and escalation rate for internal agents
- supports: Compute Governance — the safety share of AI R&D compute as a disclosure metric
- depends-on: Epoch AI — source of the AL0–AL5 automation scale
- depends-on: We Must Pace the Frontier (Amodei, September 2026) — the pacing proposal the post situates its measurements against
- related: When AI Builds Itself (The Anthropic Institute, June 4, 2026) — the Institute's earlier disclosure of internal AI R&D data
- related: Research acceleration: The view inside OpenAI (OpenAI, September 2026) — OpenAI's September 2026 internal research-acceleration measurements
- related: Anthropic Risk Report: August 2026 (Redacted) — the risk report the post says carries its oversight measures
- related: Anthropic's Advanced AI Framework (June 2026) — the transparency obligations the post pairs with its measurements
- related: AI Transparency, Self-Improving AI Monitoring Act, Coordinated slowdown proposals compared
- related: Anthropic, The Anthropic Institute, Jack Clark, METR
Sources
- Raw source:
Raw Sources/Anthropic - Measurements for understanding the pace of AI development inside frontier labs (2026-09-17).md - Primary text: anthropic.com
- Coverage: implicator.ai