This is a paper by Shivalika Singh, Marzieh Fadaee and Sara Hooker of Cohere Labs (the corresponding authors) with Yiyang Nan, Alex Wang, Daniel D'souza, Ahmet Üstün and Beyza Ermis of Cohere and Cohere Labs, Sayash Kapoor (Princeton), Sanmi Koyejo (Stanford), Yuntian Deng (Waterloo), Shayne Longpre (MIT) and Noah A. Smith (Allen Institute for AI and University of Washington), posted to arXiv as 2504.20879 in April 2025. It audits the Chatbot Arena leaderboard and argues that private testing with selective disclosure, unequal sampling and data access, and silent model deprecation "have resulted in a distorted playing field." The authors disclose that Cohere submitted several open-weight models to the arena, that the study began after they observed Aya Expanse being undersampled, and that their inquiries to the organizers in November 2024 may have prompted the arena's December 2024 policy statement.
Summary
Data and scope
The analysis combines sources covering about 2 million battles, 243 models and 42 providers between January 2024 and April 2025: the arena's released historical battle datasets (100,000 and 1.9 million battles), 43,000 battles from LMArena and Cohere, 5,800 battles scraped between January and March 2025 to identify anonymous models, 197,000 prompts received through Cohere's API, and 14,300 leaderboard snapshots from the leaderboard's commit history. Anonymous models were attributed to providers by prompting them to identify themselves, a method the authors call "inherently approximate." Models are grouped by licence as proprietary, open-weight or open-source.
Private testing and selective retraction
Scraped battles show that providers may test many private variants simultaneously without releasing or de-anonymizing them. Between January and March 2025 Meta had 27 private variants on the main leaderboard, in the lead-up to the Llama 4 release, and 16 more on the vision leaderboard (43 in all); Google had 10, mostly Gemini; Amazon was also prominent; Reka had one; and no academic lab tested privately, which leads the authors to conclude that "only certain providers were made aware they could submit multiple private variants." Cohere submitted four private variants as part of the study's own experiments and none before.
The paper argues that choosing the best of N privately tested variants violates the Bradley–Terry model's unbiased-sampling assumption, since the expected score of the maximum exceeds that of any single submission. Simulations show the expected maximum score rising with the number of variants, and that a weaker model family allowed to test privately can outrank a stronger family limited to one submission. A real-world experiment submitted two identical Aya-Vision-8B checkpoints in March 2025, which received Arena scores of 1052 (±21/22) and 1069 (±19/23) with four models ranked between them, and two different Aya-Vision-32B candidate checkpoints, which scored 1097 (±29/25) and 1059 (±18/23) with nine models between them. The authors also note that GPT-4.5 and Grok-3 reached the top of the leaderboard on the same day (4 March 2025) and that Gemini and ChatGPT-4o variants exchanged the top position three times within a week in November 2024, which they read as consistent with parallel testing of variants.
Data access asymmetries and overfitting
Proprietary models received between 54.3% and 70.1% of battles each month from January 2024 to March 2025, and 61.4% of all data over the period. Google and OpenAI are estimated to have received 19.2% and 20.4% of all arena data; 83 open-weight models together received 29.7% and 41 fully open-source models 8.9%; OpenAI, Google, Meta and Anthropic together received 62.8%, "68 times more than the combined share" of Allen AI, Stanford, Princeton and UC Berkeley. Maximum daily sampling rates observed between January and March 2025 were 34% for Google and OpenAI, 22.0% for xAI, 17.9% for Meta and 3.3% for Reka, which the authors say is inconsistent with the active sampling rule proposed in the arena's own paper (Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)) and with the stated policy. Providers serving models through their own APIs receive every prompt, whereas providers whose models are hosted by a third party may request only 20% of their battle data.
To estimate the value of this access, the authors fine-tuned the 7B base model of the Cohere Command family with 0%, 30% or 70% of the training mix drawn from arena battles, for 1,300 steps at batch size 128 without tuning. Win rates against Llama-3.1-8B-Instruct on the 500 ArenaHard prompts, judged by gpt-4o-2024-11-20, rose from 23.5% to 42.7% and 49.9%, relative gains of 81.7% and 112.3%, while MMLU moved from 66.5% to 64.4% and 65.9%. Two further findings support the claim that arena data is predictive of future arena data: the prompt distribution shifts over time (English fell from over 80% to about 50% of prompts; Russian rose from 1% in April 2023 to 15.7% by December 2024; multilingual prompts rose from 23.9% to 43.5%), and prompts repeat, with de-duplication removing an average of 20.14% of API prompts between November 2024 and April 2025 (peaking at 26.5% in March 2025), 7.3% of December 2024 prompts reappearing verbatim in January 2025 and 9% under a semantic-similarity criterion.
Model deprecation
The arena's backend lists 47 models as deprecated, but the authors identify 205 models whose sampling rate was reduced to near zero without notification; 64% of the silently deprecated models are open-weight or open-source, and 87.8% of open-weight and 89% of open-source models had been deprecated against 80% of proprietary models. Two simulations with the arena's own scoring code show that deprecating a model while the task distribution shifts changes the final ranking of the remaining models, and that a sparse or disconnected comparison graph produces rankings that diverge from true skill, both of which the authors present as violations of Bradley–Terry's transitivity and connectivity assumptions.
Recommendations and limitations
The paper recommends prohibiting score retraction after submission (with disclosure of the number of private variants tested), a transparent per-provider cap on concurrent private variants (for example three), deprecation applied proportionally across licence categories (for example the bottom 30th percentile of each), adoption of the active sampling rule from the arena's own paper, and public reporting of all tested models, deprecations and sampling rates. The stated limitations are the lack of access to the arena's raw data, a scraping window (January–March 2025) that coincided with Meta's Llama 4 launch and may understate other providers, training experiments that likely underestimate the potential to overfit, and reliance on self-identification to attribute anonymous models.
Discrepancies noted
The paper states the simulated effect of private testing two ways. The findings box opening Section 3 says "testing just 10 variants yields a notable increase of approximately 100 points in the maximum score identified," while the body of Section 3.2 says "testing just 20 variants yields a notable increase of approximately 50 points in the maximum score identified, compared to the baseline scenario with no private testing." Both refer to Figure 7; the text does not reconcile them.
Key claims
- 27 private Meta variants on the main leaderboard (43 with vision) between January and March 2025; 10 for Google; none for academic labs.
- Google and OpenAI received an estimated 19.2% and 20.4% of all arena data; 83 open-weight models 29.7%; proprietary providers 61.4%.
- Maximum daily sampling rates of 34% (Google, OpenAI) against 3.3% (Reka).
- Identical checkpoints scored 1052 and 1069; different candidate checkpoints 1097 and 1059.
- Training with 70% arena data raised ArenaHard win rate from 23.5% to 49.9% (relative +112.3%) with no MMLU gain.
- 205 models silently deprecated against 47 officially listed.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2504.20879 (arxiv.org); the abstract page is arxiv.org.
Relationships
- contradicts: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024) — disputes that the platform's sampling and disclosure practices match the design described there.
- supports: LMArena (Chatbot Arena) — the source for the private-testing, data-access and deprecation findings and the recommendations described there.
- supports: AI Benchmarks and Evaluation — evidence on leaderboard gaming and overfitting.
- related: Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing — a later treatment of the same evaluation-integrity problems.
- related: Cohere