This is the paper introducing HELM, written by 50 authors at the Stanford Center for Research on Foundation Models and the Institute for Human-Centered Artificial Intelligence, with Percy Liang, Rishi Bommasani and Tony Lee as lead authors, and posted to arXiv as 2211.09110 on 16 November 2022; the arXiv text records that it was reviewed on OpenReview. It sets out a framework for evaluating language models across many use cases and many desiderata at once, applies it to 30 models under identical conditions, and releases the prompts, completions and toolkit.
Summary
What "holistic" means
The authors define holistic evaluation by three elements. First, broad coverage with "recognition of incompleteness": because no benchmark can cover every scenario or desideratum, HELM begins with a top-down taxonomy of scenarios (use cases) and metrics (desiderata) and then selects a subset to implement, making explicit what is missing, such as languages other than English or metrics for trustworthiness. Second, multi-metric measurement: "Societally beneficial systems reflect many values, not just accuracy," so every desideratum is measured on each scenario rather than deferred to bespoke datasets. Third, standardization: the object of evaluation is the language model itself, so the adaptation procedure is held fixed and every model is run on the same scenarios.
Structure
HELM operates at two levels, an abstract taxonomy and a concrete implemented set chosen for coverage, value and feasibility. The core set contains 16 scenarios, each a triple of task, domain and language, spanning six user-facing tasks (question answering, information retrieval, summarization, sentiment analysis, toxicity detection and miscellaneous text classification), several domains, and English only, including varieties such as African-American English. Seven metric categories are measured on each: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Of the 112 possible scenario–metric pairs, 98 (87.5%) are measured. Seven targeted evaluations, covering linguistic understanding, knowledge, reasoning, memorization and copyright, disinformation, bias and toxicity, add 26 scenarios, of which 21 were new or not previously used in mainstream language-model evaluation, for 42 scenarios in all.
Thirty models from 16 families and 12 organizations were evaluated, including AI21 Labs' J1-Jumbo, Anthropic-LM v4-s3 (52B), BigScience's BLOOM (176B), Cohere xlarge, EleutherAI's GPT-NeoX (20B), Google's UL2, Meta's OPT (175B), Microsoft/NVIDIA's TNLG v2 (530B), OpenAI's davinci and text-davinci-002, Tsinghua's GLM (130B) and Yandex's YaLM (100B), spanning open, limited-access and closed models. Before HELM, the authors found, models had on average been evaluated on 17.9% of the core scenarios, with T5 and Anthropic-LM sharing no dataset in their original papers and several models reporting no public results at all; HELM raised coverage to 96.0%, the remainder lost to technical issues. All models were adapted by few-shot prompting with deliberately simple, generic prompts, chosen "to orient the development of language models towards generic language interfaces," with five in-context examples, three random seeds for example selection, and 1,000 instances per scenario. The evaluation comprised 4,939 runs, 12,169,227,491 tokens and 17,431,479 queries, costing $38,001 in commercial API fees and about 19,500 GPU hours for open models.
Findings
The paper lists 25 top-level findings. Among them: text-davinci-002 performed best on accuracy, robustness and fairness across the core scenarios, with Anthropic-LM v4-s3 (52B) in the top three on all three despite being more than ten times smaller than TNLG v2 (530B), which the authors read as evidence that instruction-tuning "provides a broad set of advantages"; a consistent accuracy gap separated the open models from the closed and limited-access ones at the time; the relationship between accuracy and calibration depended on the scenario (improving accuracy worsened calibration on HellaSwag and improved it on OpenBookQA); robustness and fairness perturbations correlated strongly with accuracy but could produce large drops, TNLG v2 falling from 72.6% to 38.9% on NarrativeQA under robustness perturbations; performance disparities appeared wherever demographic metadata existed, OPT (175B) degrading from 1.506 to 2.114 bits per byte between White and African American English on TwitterAAE; bias and toxicity in generations were low and similar across models on the core scenarios; no strong trade-off appeared between accuracy and efficiency; automated summarization metrics on CNN/DailyMail and XSUM "largely fail to discriminate" model quality; most models were near chance on CivilComments toxicity detection (OPT at 50.1%), with sharp robustness drops on the Black split; code models, especially code-davinci-002, outperformed text models on reasoning, reaching 52.1% on GSM8K against 35.0% for text-davinci-002 and under 16% for all others; regurgitation of copyrighted text was uncommon but correlated with accuracy; on BBQ the three most accurate models were the only ones whose biases in ambiguous contexts aligned with broader social biases; all models were highly sensitive to prompt formatting and in-context examples, and OPT (175B) scored 79.1% on HellaSwag when answer choices were presented separately but 30.2% when presented jointly; bits-per-byte on The Pile was a poor predictor of downstream accuracy; and scale predicted accuracy within a family but not across models, every model winning head-to-head comparisons well above chance having at least 50B parameters.
Limitations stated by the authors
Section 11 groups limitations three ways. On results: few-shot prompting results may not transfer to fine-tuned or prompt-optimized use and "model behavior is very sensitive to prompt design"; train–test contamination "largely remains unclear" for models trained on undocumented web data; and three seeds with 1,000 instances may leave comparisons statistically insignificant. On implementation: coverage of what is missing, and the absence of a unified standard for dataset validity and reliability, which the authors connect to Strathern's law that "when a measure becomes a target, it ceases to be a good measure." On design, they note the framework should extend to other modalities. The paper also devotes a section to what is missing, including scenarios, metrics, targeted evaluations, models (PaLM and Gopher were not evaluated) and adaptation methods, and describes HELM as "a living benchmark" to be updated continuously, with raw prompts and completions released and a modular open-source toolkit.
Key claims
- 16 core scenarios × 7 metric categories (98 of 112 pairs measured), plus 7 targeted evaluations over 26 scenarios; 42 scenarios in total, 21 new to mainstream evaluation.
- 30 models from 12 organizations evaluated under one standardized 5-shot prompting procedure; prior coverage of the core scenarios averaged 17.9%, raised to 96.0%.
- 4,939 runs, about 12.2 billion tokens, 17.4 million queries, $38,001 in API costs and roughly 19,500 GPU hours.
- text-davinci-002 led on accuracy, robustness and fairness; Anthropic-LM v4-s3 (52B) reached the top three at a tenth of TNLG v2's size.
- Results are highly sensitive to prompt format and multiple-choice adaptation (OPT: 79.1% versus 30.2% on HellaSwag); upstream bits-per-byte does not predict downstream accuracy across families.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2211.09110 (arxiv.org); the abstract page is arxiv.org. Results were published at crfm.stanford.edu and the toolkit at github.com. The arXiv text does not itself name the journal of publication.
Relationships
- supports: HELM (Holistic Evaluation of Language Models) — the primary text for the framework's design, model set and release-time findings.
- supports: AI Benchmarks and Evaluation — evidence on prompt sensitivity, contamination uncertainty and multi-metric trade-offs.
- related: Stanford Center for Research on Foundation Models (CRFM), Stanford HAI, Rishi Bommasani
- related: MMLU — one of the knowledge scenarios in the targeted evaluations.