LMArena (Chatbot Arena)

LMArena, launched as Chatbot Arena and since January 2026 branded simply Arena, is a public website on which a visitor types a prompt, receives answers from two unnamed AI models, and votes for the better one. The votes are fitted with a statistical model into a single rating per model, displayed on leaderboards for text, code, vision, search, image and video generation. Researchers at UC Berkeley began collecting votes in April 2023 and published the first leaderboard on May 3, 2023 (Source: lmsys.org; Source: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)). In 2025 the project was incorporated as Arena Intelligence Inc., which has raised $250 million in venture funding (Source: arena.ai). Unlike a fixed test, it has no answer key: a score records how often people preferred a model's answers, not whether the answers were correct.

What it measures

The introducing paper describes the platform as a way of "evaluating LLMs based on human preferences" in open-ended, real-world use, and positions it against static, ground-truth benchmarks such as MMLU and GSM8K, which the authors say cannot capture open-ended interaction, can leak into training data over time, and lack a definitive correct answer for many tasks Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024). The construct claimed is helpfulness as judged by the people using the model. The paper states that the study "concentrates on assessing the helpfulness of LLMs but overlooks their safety aspects."

Design

Battles and votes

In battle mode a user enters any prompt; none is preset. Two models sampled by the platform answer side by side under anonymous labels. The user votes for the left answer, the right answer, a tie, or "both are bad", and may continue the conversation before deciding. Model names are revealed only after the vote, and only votes cast while the names were hidden count toward the leaderboard (Source: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024); Source: arena.ai). Votes cast in the side-by-side mode, where the user picks the two models, are collected for research but do not affect rankings (Source: arena.ai). Since May 12, 2026, one in ten direct-chat sessions has been converted into a battle between two randomly sampled models, and those votes also count (Source: arena.ai).

By January 2024 the platform had logged about 240,000 votes from roughly 90,000 users across more than 50 models and more than 100 languages, 77% of conversations being in English. A topic-modelling pass found 600 prompt clusters, the largest holding 1% of prompts, spanning poetry, coding, mathematics and medical questions Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024).

From votes to a score

The first leaderboards used the Elo rating system from chess, in which each game nudges the two players' ratings and the rating gap predicts the chance of winning (Source: lmsys.org). In December 2023 the team switched to the Bradley–Terry model, a statistical model for paired comparisons that estimates every model's strength at once from the full vote history, assumes a model's strength does not change over time, and gives tighter confidence intervals; the team described it as the maximum-likelihood counterpart of Elo (Source: lmsys.org). In practice the fit is a logistic regression of vote outcomes on which models were shown, and the resulting coefficients, "which we formerly called the Elo score", are published as the Arena Score (Source: arena.ai). An independent audit gives the conversion as Arena Score = 1000 + (400 / ln 10) × the fitted coefficient, so that one point equals about 0.00576 on the log-odds scale The Leaderboard Illusion (Singh et al., 2025).

Several adjustments sit on top of the basic fit. Style control, introduced in August 2024, adds answer length and counts of markdown headers, bold text and lists as extra regression terms, so that the part of a model's win rate explained by formatting is assigned to the style terms and not to the model; the team found length to be the dominant factor, and reported that with the control applied GPT-4o-mini and Grok-2-mini fell while Claude 3.5 Sonnet, Claude 3 Opus and Llama-3.1-405B rose (Source: arena.ai). LMArena announced on May 16, 2025 that style control would become the default leaderboard view, citing a community survey, and the notes to its leaderboard dataset record the change under that date (Source: x.com; Source: huggingface.co). On July 23, 2025 confidence intervals moved from bootstrapping to a closed-form method, and battles were reweighted in inverse proportion to how often each pairing appears, which the maintainers said widens intervals and produces more ties (Source: arena.ai). Terms added in May 2026 absorb a position bias toward the left-hand answer and an advantage for models from the same developer as earlier turns of a conversation (Source: arena.ai).

Data filtering

Votes are deduplicated by prompt, with prompts in the top 0.5% by frequency capped, which removes around 10% of votes; conversations that try to make a model reveal its identity are filtered by a classifier, affecting under 4% of votes; and since September 2025 votes from users with statistically anomalous voting patterns are removed (Source: arena.ai). The paper validated vote quality by having Berkeley graduate students re-label samples with fact-checking: crowd votes agreed with the experts 72% to 83% of the time, against 79.4% and 89.8% agreement between the two experts in the two test sets Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024).

Listing rules and pre-release testing

The published policy lists only models that are publicly available through open weights, a public API or a public service, after at least 1,000 votes. Developers may test unreleased models under anonymous labels, may "test multiple variants of their models before making them public", receive the results privately, and receive conversation data from those tests. A model first tested anonymously carries a "preliminary" tag until enough votes accumulate after public release. At least one model in every battle is publicly available, and the policy reserves the right to retire models (Source: arena.ai).

Versions and variants

The Text Arena is divided into 29 categories, including coding, mathematics, creative writing, instruction following, multi-turn, hard prompts, an "Expert" category of prompts judged to come from people at the forefront of a field, and occupational categories such as legal and government or medicine and healthcare (Source: arena.ai; Source: arena.ai). Separate arenas cover web development and code, vision, documents, search, text-to-image, image editing, video, and, since June 4, 2026, agents, which are scored on behavioural signals such as retries and file downloads and not on preference votes alone (Source: arena.ai; Source: huggingface.co). Two optional views were added in July 2026: a factuality adjustment that blends preference votes with automated checking of claims at a default weight of 25%, and "AutoEval" scores, in which a reward model trained on past votes casts proxy votes so that a new model has a labelled provisional rating within an hour (Source: arena.ai; Source: arena.ai).

Development and maintainers

Chatbot Arena was built by the LMSYS group; the 2024 paper lists Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez and Ion Stoica, from UC Berkeley, Stanford and UC San Diego Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024). It was first funded through grants and donations. The company raised a $100 million seed round in May 2025 at a $600 million valuation and a $150 million Series A on January 6, 2026 at $1.7 billion, led by Felicis and UC Investments, and sells an "AI Evaluations" service to labs and enterprises that it launched in September 2025 (Source: techcrunch.com). It renamed itself from LMArena to Arena on January 28, 2026 (Source: arena.ai). In June 2026 chief executive Anastasios Angelopoulos reported a $100 million annualized revenue run rate, more than 10 million monthly visitors, and 82 million votes across all arenas (Source: arena.ai). The ranking code is open source, a portion of the vote data is released, and methodology changes have been logged publicly since July 2025 (Source: arena.ai).

Results

The first leaderboard, from 4,700 votes, ranked nine open models with Vicuna-13B first at 1169 (Source: lmsys.org). GPT-4 entered a week later at 1274, with the best open model, Vicuna-13B, at 1083 (Source: lmsys.org). By the end of 2024 Simon Willison counted 18 organizations with models ranked above the original GPT-4, which had fallen to roughly 70th place (Simon Willison's Year-in-Review LLM Posts (2023–2025)). The Stanford AI Index reported the leading developers clustered within 25 points in March 2026, with Anthropic at 1,503 (Stanford HAI AI Index Report 2026). On the September 13, 2026 snapshot the top score was 1506 and the twentieth was 1482 (Source: arena.ai).

Text Arena score

Overall category, default view (style control on), snapshot dated September 13, 2026. The site labels the figure "Arena Score"; it is written here with the unit Elo because it sits on an Elo-like scale. All rows come from one global fit and are comparable with each other; model names are the leaderboard's own.

Earlier published snapshots

Scores as published at the time. They predate style control as the default, the 2025 reweighting and, for 2023 rows, the move from online Elo to Bradley–Terry, and are not comparable with the current fit or with each other.

WebDev Arena

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

A score is relative. It carries no meaning on its own, only through its gap to other models in the same fit, and the gap translates into an expected share of head-to-head wins through the logistic curve that Elo uses (Source: lmsys.org). As an illustration of the scale from a different Elo-rated evaluation, Anthropic's Opus 4.7 system card equates a lead of about 79 points with a pairwise win rate of about 61.2% (Claude Opus 4.7 System Card). Because every score is refitted whenever data or method changes, a model's number drifts after release, and numbers from different dates or views (style control on or off, a category, the factuality blend) should not be mixed.

Many adjacent ranks are statistical ties. The leaderboard prints a confidence interval and a "rank spread" for each model. On the September 13, 2026 snapshot the first-ranked model's spread ran from 1st to 7th; GPT-6 Astra (max), with 2,693 votes and an interval of ±12, was ranked 24th with a spread from 5th to 51st (Source: arena.ai). The whole top twenty sat within 24 points.

The score reflects what voters like, which is not the same as accuracy. The paper attributes the 5 to 10 point gap between crowd–expert and expert–expert agreement mostly to crowd users "overlooking factual errors", and expects the user base to skew toward "LLM hobbyists and researchers" Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024). The maintainers' factuality analysis found that the scores of most models at the top of the preference ranking fall as weight shifts toward checked factual accuracy (Source: arena.ai). The style-control work found that longer, more heavily formatted answers win more votes (Source: arena.ai).

Limitations and criticism

Llama 4

Meta's April 5, 2025 Llama 4 announcement cited "an experimental chat version scoring ELO of 1417 on LMArena" (Source: ai.meta.com). TechCrunch reported that this version differed from the downloadable model and gave long answers with many emojis (Source: techcrunch.com). On April 8 LMArena released more than 2,000 of the model's battles, wrote that "Meta's interpretation of our policy did not match what we expect from model providers" and that Meta "should have made it clearer" the model was "customized … to optimize for human preference", and said it was updating its policies (Source: x.com). The released Maverick was then ranked below GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro; a Meta spokesperson said the company experiments with "all types of custom variants" (Source: techcrunch.com). See Llama 4 (Meta).

The Leaderboard Illusion

A paper of April 2025 by Shivalika Singh, Sara Hooker and co-authors from Cohere Labs, Princeton, Stanford, Waterloo, MIT, the Allen Institute for AI and the University of Washington argued that undisclosed private testing let a few developers submit many variants and publish only the best. It counted 27 private Meta variants on the main leaderboard between January and March 2025, and 43 including vision. It estimated that Google and OpenAI had received 19.2% and 20.4% of all arena data while 83 open-weight models together received 29.7%; found maximum daily sampling rates of 34% for Google and OpenAI against 3.3% for Reka; identified 205 models "silently deprecated" against 47 officially listed; and reported relative gains of up to 112% on the ArenaHard test set from training on arena data. Two identical checkpoints the authors submitted scored 1052 and 1069. The authors, several of whom had submitted Cohere models, recommended banning score retraction, capping private variants, clearer retirement rules and fairer sampling The Leaderboard Illusion (Singh et al., 2025). A Google DeepMind report later cited the paper as evidence of benchmark hacking (Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing).

LMArena's reply of May 9, 2025 accepted some recommendations: it would state explicitly that all providers may test multiple variants, mark retired models, and label scores provisional where more than ten variants had been tested in parallel. It disputed the rest. It said open models made up 40.9% of the leaderboard, not 8.8%; that the simulated 100-point boost rested on an arbitrary variance while its own estimate was about +11 points after 50 tests, shrinking to zero as fresh votes arrive; that the identical-checkpoint scores had overlapping confidence intervals; that the pre-release policy had been public since March 2024; and that the 112% gain was measured on a separate LLM-judged benchmark (Source: arena.ai). Simon Willison wrote that the replies did not address the ability to test many variants and publish one, and asked for disclosure of how many were tested; he also relayed Andrej Karpathy's suggestion of usage-based OpenRouter rankings as a complementary signal (Source: simonwillison.net). The Stanford AI Index summarized the research as suggesting that leaderboard standing "may partly reflect adaptation to the platform rather than general capability" (Stanford HAI AI Index Report 2026).

Use in policy and safety frameworks

The 2026 Stanford AI Index uses the Arena leaderboard as its headline measure of convergence among developers, of the gap between open- and closed-weight models (3.3% in March 2026 in the chapter highlights; 3.4%, or 49 points between Claude Opus 4.6 at 1,503 and GLM-5 at 1,454, in the chapter body), and of the gap between US and Chinese models (Stanford HAI AI Index Report 2026). NIST's AI 800-4 report cites LMArena as one method of measuring model competency through user preferences, while noting that such measurement is rarely published for post-deployment settings (NIST AI 800-4: Challenges to the Monitoring of Deployed AI Systems). Developers cite Arena ranks at launch: Google stated that Gemma 4 31B would debut third among open models (Gemma (Google open-weight models)), and Meta reported second-place rankings for Muse Image (Muse Image and Muse Video (Meta Superintelligence Labs)).

Relationships

• instance-of: AI Benchmarks and Evaluation
• related: Double-Blind Evaluation — cites the Leaderboard Illusion findings as motivation.
• related: Llama 4 (Meta) — the experimental-variant episode.
• related: Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing, Stanford HAI AI Index Report 2026, NIST AI 800-4: Challenges to the Monitoring of Deployed AI Systems, Simon Willison's Year-in-Review LLM Posts (2023–2025)
• related: Cohere — home of the Leaderboard Illusion's lead authors; Meta AI; Hugging Face — hosts the released vote data.
• related: Artificial Analysis Intelligence Index, Epoch Capabilities Index (ECI), HELM (Holistic Evaluation of Language Models) — other aggregate instruments.

See also

• GDPval — an evaluation whose Artificial Analysis implementation also reports Elo from pairwise comparisons.
• MMLU, GSM8K — static benchmarks the platform was designed to complement.
• Open-Weight Frontier Models