OpenRCA

OpenRCA is a benchmark for root cause analysis (RCA): given a natural-language question about a software failure and the raw operational telemetry surrounding it, a model must identify what went wrong. It consists of 335 failure cases drawn from three real enterprise systems, accompanied by more than 68 GB of logs, metric time series and dependency traces (Source: openreview.net). The paper was published at the Thirteenth International Conference on Learning Representations (ICLR 2025) in the datasets-and-benchmarks area, and the dataset and harness are released by Microsoft under an MIT licence (Source: github.com).

The authors describe the task as one that current models "can only handle the simplest cases": with their own purpose-built RCA-agent scaffold, the best model in the paper, Claude 3.5, solved 11.34% of cases (Source: openreview.net). Anthropic began reporting OpenRCA in its Claude 4.6-generation system cards, where scores rose to 34.9% for Claude Opus 4.6 (Claude Opus 4.6 System Card).

What it measures

OpenRCA measures whether a model can move from a symptom to a cause across heterogeneous, long-context operational data. The authors situate it deliberately against the code-generation benchmarks that dominate software-engineering evaluation: their stated motivation is that research "predominantly focuses on the early stages of development, such as code generation, while overlooking the post-development phases that are crucial to user experience" (Source: openreview.net).

Each case gives the model a failure window and asks it to name some combination of three fields — the datetime at which the root cause occurred, the component responsible, and the reason for the failure. The repository states that answering requires the model to "understand complex system dependencies and perform comprehensive reasoning across various types of telemetry data, including KPI time series, dependency trace graphs, and semi-structured log text" (Source: github.com).

Design

Dataset construction

The 335 cases come from three enterprise systems, referred to as Telecom, Bank and Market; the Market telemetry is collected in two separate service groups, cloudbed-1 and cloudbed-2, so evaluation runs address four datasets. Telemetry is organised per system and per date into log, metric and trace directories, alongside a record.csv of ground-truth faults and a query.csv of generated questions. All faults are timestamped in UTC+8, a detail the maintainers flag in the repository FAQ as a recurring source of mismatched results for users in other timezones (Source: github.com).

Deployment structure differs by system: Bank components are all at pod level with no vertical deployment hierarchy, while Market records the node-to-pod-to-service relationship in metric_container.csv and Telecom publishes its structure separately (Source: github.com).

Task format

Questions are generated from the fault records against a task specification, and fall into seven shapes according to which of the three fields they ask for — one field, two, or all three. The repository exposes the generator, so new queries can be produced against the same telemetry or against a user's own private telemetry (Source: github.com).

Predictions are submitted as a JSON-like string per query listing, for each failure in chronological order, the root-cause occurrence datetime, component and reason.

Scoring

Scoring is by exact match on each requested field, with a partial score available per field and a strict score that requires every requested field to be correct. Anthropic's system cards report the strict figure and describe the range as 0% to 100%, "where 100% indicates full identification of the root cause" (Claude Opus 4.6 System Card). No human baseline is reported.

Submissions

The maintainers accept results from third-party agents. Where an agent is not open-source, the FAQ requires the submitter to supply descriptions of its tools, MCPs or skills for review, "to ensure that no ground-truth information from records.csv is being read from local directories or websites" (Source: github.com).

Development and maintainers

The paper is authored by Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang and Qi Zhang, with affiliations spanning Microsoft Research, the Chinese University of Hong Kong–Shenzhen and Tsinghua University; Pinjia He described it publicly as joint work with Microsoft (Source: openreview.net; Source: x.com). The repository is hosted in the microsoft GitHub organisation and was last updated in April 2026 (Source: github.com).

Alongside the dataset the authors released RCA-agent, an execution-based multi-agent baseline that writes Python to retrieve and analyse telemetry rather than reading it into context directly, which the repository presents as the way to keep the task tractable over 68 GB of data (Source: github.com).

Results

Reported scores moved from 11.34% in the original paper to 34.9% in Anthropic's February 2026 reporting. Anthropic is the only frontier developer known to report OpenRCA in a system card as of September 2026; its cards state that all figures are three-run averages and that the benchmark "was run on the author's agent harness," so developer-reported rows are not directly comparable to the paper's own baseline, which used a different model generation under the same scaffold. Competitor scores in the Sonnet 4.6 card are attributed to the benchmark authors rather than run by Anthropic (Claude Sonnet 4.6 System Card).

Overall

Breakdown by enterprise system

Anthropic's Opus 4.6 card decomposes the overall figure by system, and states that the telecom system "remains the most challenging across all models" (Claude Opus 4.6 System Card). These are components of the Overall rows above rather than separate results, and are not comparable across settings.

Anthropic additionally reports Opus 4.6 fully identifying the root cause in 117 of 335 cases (35%), against 90 (27%) for Opus 4.5, and reducing the number of zero-scoring cases from 163 to 136 (Claude Opus 4.6 System Card).

Interpreting scores

Strict accuracy multiplies across the fields a question asks for, so a score depends heavily on the mix of one-, two- and three-field questions in the set, and a model can carry real signal on one field while scoring zero overall. The metric has no chance-level baseline published in the paper.

The scale is also not comparable across harnesses. Both the paper's baseline and Anthropic's reported figures run through RCA-agent, but across different model generations and run counts, and Anthropic's card distinguishes its own three-run averages from competitor figures supplied by the benchmark authors (Claude Sonnet 4.6 System Card).

Limitations and criticism

Anthropic's own assessment, repeated in both 4.6-generation cards, is that OpenRCA "was described in a peer-reviewed paper and is grounded in real enterprise telemetry, but it is a simplified proxy: the dataset does not heavily test reasoning across complex service dependency chains" (Claude Sonnet 4.6 System Card; Claude Opus 4.6 System Card).

A July 2026 issue filed on the repository reports that a predictor reading no telemetry at all — always answering the most frequent component-and-reason pair for that system, estimated leave-one-out, and guessing the midpoint of the question's time window — scores 7.46% strict accuracy, against the paper's headline 11.34% for RCA-agent. The submitter first reproduced the published 11.34% exactly from the committed prediction files before reporting the baseline, notes that the 3.88-point difference does not reach significance under a paired exact McNemar test (p = 0.0596, 41 discordant pairs) while cautioning that this "is not evidence that the two are equal," and reports that the sign of the difference reverses on two of the four datasets. Broken down by field, the agent roughly doubles the prior on the reason field (21.1% against 10.6%) but is close to it on component (18.9% against 17.8%); on the one task shape asking for all three fields, both the agent and the baseline score zero. The issue frames itself as "a suggestion about reporting, not a claim that the benchmark is invalid." It remained open and without a maintainer response as of September 2026 (Source: github.com).

Relationships

• related: SWE-bench — the code-generation benchmark OpenRCA's authors position their post-development task against
• related: Terminal-Bench — companion agentic software-operations benchmark reported in the same system cards
• instance-of: AI Benchmarks and Evaluation — benchmark methodology
• related: Claude Opus 4.6 System Card — highest reported score and the per-system breakdown
• related: Claude Sonnet 4.6 System Card — cross-developer comparison table
• related: Microsoft — repository host and co-author affiliation
• related: Agentic AI — the scaffolded, tool-using setting the benchmark is run in

See also

• MCP Atlas
• OSWorld
• Claude Opus 4.6