This is the paper describing the Chatbot Arena platform, now LMArena, written by Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez and Ion Stoica of UC Berkeley, Stanford and UC San Diego, posted to arXiv as 2403.04132 in March 2024 with ICML as its stated venue keyword. It sets out the platform's design, the statistics of the data collected in its first nine months, the ranking method, and the analyses the authors offer to establish the credibility of crowdsourced preference votes.
Summary
Rationale
The authors classify benchmarks along two axes: whether questions are static or live, and whether scoring uses ground truth or human preference. Static, ground-truth benchmarks such as MMLU, HellaSwag and GSM-8K dominate because they are cheap and reproducible, but the paper argues they are not open-ended, can be contaminated over time, and cannot supply a ground truth for many tasks. Chatbot Arena is presented as "the first open, large-scale, and crowdsourced benchmark platform that utilizes live human interaction," in the spirit of DynaBench but at larger scale.
Platform and data
On the free website a user enters any prompt, receives responses from two anonymous randomly sampled models, and votes for the better one, or "tie" or "both are bad," with model identities revealed only after the vote; the user may continue the conversation before voting. Users accept terms permitting public release of the data. The platform had run since April 2023 and, as of January 2024, had collected around 240,000 votes from over 90,000 users in more than 100 languages, covering more than 50 models; 77% of conversations were in English and 5% in Chinese. Table 1 records 243,329 conversations, 50 models, 90,051 users and 149 languages as of 21 January 2024, averaging 1.3 turns, 94.9 prompt tokens and 269.0 response tokens. About 8,000 votes were collected per model, at 1,000–2,000 votes per day, with visits estimated at over one million by March 2024. Conversations mentioning model or company names were filtered to preserve anonymity, and the OpenAI moderation API flagged 3% of requests as unsafe. Pairs were sampled non-uniformly to concentrate votes on closely matched models.
Ranking method
The paper replaces the Elo score used in earlier versions of the leaderboard with Bradley–Terry coefficients estimated by reweighted maximum likelihood, "because the BT coefficients are better for the purpose of statistical estimation." Confidence intervals use "sandwich" robust standard errors, chosen over a pivot bootstrap for their smaller size in large samples, and an approximate ranking is derived so that with high probability no model's performance is understated. An active sampling rule chooses model pairs in proportion to the expected reduction in confidence-interval width; in a replay of 213,576 historical votes, random sampling needed 54% more data than adaptive sampling to estimate the win matrix to a precision of 0.2 and 5% more to estimate scores to a precision of 0.3. A procedure for detecting anomalous users computes a p-value for each vote against the historical distribution for that model pair and combines them with Fisher's method at five random checkpoints; on 25 hand-identified anomalous and 25 normal users it reached about 90% true positives and 60–70% true negatives.
Data analyses
A topic-modelling pipeline (OpenAI text-embedding-3-small, UMAP to five dimensions, HDBSCAN, GPT-4-Turbo labelling) found 600 clusters, the largest covering only 1% of prompts and most under 0.5%, which the authors describe as a long-tailed, diverse distribution. Sampling 30 prompts from each of several clusters and judging with GPT-4-Turbo, GPT-4-0613's win rate against Llama-2-70b-chat was 96.7% on Python game programming and 86.7% on C/C++ multithreading but 53.3% on movie recommendations and 58.3% on travel planning, showing that some clusters discriminate between models while others do not. The authors also construct an "Arena Bench" of selected hard prompts that widens the measured gap between proprietary and open models relative to MT-Bench.
To validate vote quality, 160 battles each between GPT-4-Turbo and Llama-2-13b-chat and between GPT-4-Turbo and GPT-3.5-Turbo-0613 were relabelled by two UC Berkeley graduate students who fact-checked responses, taking three to five minutes per item. Crowd–expert agreement ranged from 72.8% to 83.1%, against expert–expert agreement of 89.8% and 79.4% and crowd–GPT-4-judge agreement of 75.6%. GPT-4-Turbo's win rate against Llama-2-13b was 81.2% by crowd vote, 89.4% and 86.9% by the two experts and 78.8% by GPT-4; against GPT-3.5-Turbo it was 76.3%, 82.5%, 89.4% and 79.4%. The authors attribute the remaining crowd–expert gap to users overlooking factual errors, and expert disagreement mainly to prompts without a ground truth.
Limitations and plans
The authors expect the user base to consist "primarily of LLM hobbyists and researchers," which may bias the distribution; note that prompts from a chat interface "might not accurately reflect the real-world usage of LLMs in production environments or specialized domains"; and state that the platform assesses helpfulness but "overlooks their safety aspects." Planned work includes topic leaderboards and sections for multimodal and agent-based models. The paper commits to releasing a dataset of over 100,000 pairwise votes. Sponsors listed include Kaggle, MBZUAI, a16z, Together AI, Anyscale and Hugging Face, with partial support from Accenture, AMD, Google, IBM, Intel, Microsoft, Samsung SDS, SAP, Uber and VMware; the authors also describe collaborating with OpenAI, Google, Anthropic, Mistral and Hugging Face to host their latest models.
Key claims
- About 240,000 votes from 90,000 users in more than 100 languages and 50 models between April 2023 and January 2024; 77% of conversations in English.
- Rankings use Bradley–Terry coefficients with sandwich confidence intervals, having replaced Elo; adaptive pair sampling reduces the data needed by up to 54%.
- Crowd votes agree with expert labels 72–83% of the time, close to the 79–90% agreement between experts.
- Prompt topics form a long tail of 600 clusters, none above 1% of the data.
- Stated limitations: a hobbyist and researcher user base, chat-interface prompts, and no assessment of safety.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2403.04132 (arxiv.org); the abstract page is arxiv.org. The platform described was hosted at chat.lmsys.org.
Relationships
- supports: LMArena (Chatbot Arena) — the primary text for the platform's design, early statistics and ranking method.
- contradicts: The Leaderboard Illusion (Singh et al., 2025) — the later critique disputes the platform's fairness in model sampling and private testing.
- related: Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing
- instance-of: AI Benchmarks and Evaluation