This is a research paper by Shivalika Singh, Beyza Ermis and Sara Hooker of Cohere For AI (the corresponding authors) with 21 co-authors from EPFL, Hugging Face, Mila and McGill University, AI Singapore, the National University of Singapore, Cohere, MIT, KAIST, the Instituto de Telecomunicações and Instituto Superior Técnico in Lisbon, the MIT-IBM Watson AI Lab, Carnegie Mellon University, CONICET and the Universidad de Buenos Aires, and Meta AI Research. It was posted to arXiv as 2412.03304 in December 2024. The paper does two things: it measures how much of the original MMLU test set depends on culturally specific knowledge, and it releases Global-MMLU, a 42-language version of MMLU with human-verified translations for a subset of languages and per-question cultural-sensitivity labels.
Summary
Motivation
The authors observe that multilingual evaluation of language models is often conducted on machine translations of English benchmarks, which they collectively call "transMMLU" when the benchmark is MMLU. They identify two problems with this practice. First, translation does not remove cultural bias: MMLU contains subsets that are specific to the United States (US history, US accounting, US law), and a translated version carries that content into every language, so that optimising models for translated MMLU risks "overfitting to the datasets' cultural biases." Second, machine translation introduces translation artifacts ("translationese") that degrade evaluation quality.
Annotating MMLU for cultural sensitivity
To quantify the first problem, the authors worked with 200 professional and community annotators to review a uniform random sample of the English MMLU: 50 questions from each of the 57 subjects, 2,850 questions in all, which the paper calls MMLU Annotated (MA). Annotators labelled each question for whether answering it required cultural knowledge (with the culture chosen from a list: Western, East Asian, Middle Eastern, South Asian, African, Latin American, Other), geographic or regional knowledge (North America, South America, Europe, Asia, Africa, Australia and Oceania, Antarctica), or dialect knowledge (regional variation within English). Every question was reviewed by at least three annotators and up to ten; 96.4% were reviewed by more than three. Labels were assigned by majority vote. A question carrying any of the three labels is classed Culturally-Sensitive (CS); one carrying none is Culturally-Agnostic (CA). Annotators also checked for temporal dependence and found that only 2.4% of the sample had answers that change over time.
The annotation produced 792 CS questions and 2,058 CA questions, so 28% of the sample is culturally sensitive. Within the CS set, geographic knowledge is the most common tag (54.7% of CS questions), followed by cultural knowledge (32.7%) and dialect knowledge (0.5%); 10.6% of CS questions require both cultural and geographic knowledge and 1.5% require all three.
The authors describe the labels as concentrated on the West. Of the questions tagged for cultural knowledge, 86.5% concern Western culture; South Asian culture is next at 4%, with Latin American, African and Indigenous cultures at 1.3%, 1.1% and 0.7%. Among the Western-culture questions, 73.9% concern the United States and 8% the United Kingdom. Of the geography-tagged questions, 64.5% concern North America and 20.4% Europe, which together give the paper's headline figure that 84.9% of questions requiring geographic knowledge focus on those two regions.
Sensitivity varies by subject. In the Humanities category 68% of sampled questions are CS, and more than 80% of the questions in Philosophy, Moral Scenarios, High School US History and High School Government and Politics were labelled CS (the Moral Scenarios prompt specifies "moral standards in the US"). In STEM only 30 of 950 sampled questions (3.15%) were CS, and every sampled question in Clinical Knowledge, Computer Security and Econometrics was CA. Table 1 of the paper gives the CS counts by category: STEM 23, Humanities 442, Social Sciences 208, Medical 19, Business 36, Other 64. All questions in World Religions and Moral Scenarios carried at least one cultural, regional or dialect reference, while twelve subjects contained no CS questions at all. Inter-annotator agreement measured by Krippendorff's alpha was high for most subjects, unanimous for Anatomy, and lowest for Moral Scenarios.
Building Global-MMLU
Global-MMLU contains all roughly 14,000 MMLU questions in 42 languages (including English), 589,764 items in total. The English set was first machine-translated into 41 languages with the Google Translate API, chosen over GPT-3.5-turbo after a ChrF++ comparison and to avoid the bias of models favouring their own generations. Human involvement then came in three forms. Compensated professional annotators post-edited the translations for four "Gold Set" languages: Arabic, French, Hindi and Spanish. Community annotators, working on the Argilla platform, post-edited translations for languages that reached a threshold of at least 50 human-translated samples; eleven languages met it (Amharic, Czech, Malay, Persian, Romanian, Russian, Sinhala, Telugu, Turkish, Ukrainian, Vietnamese). For a further ten languages (Bengali, Chinese, German, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba) the authors incorporated OpenAI's professionally translated MMMLU set. The introduction summarises the result as professional translations with post-edits for 14 languages, crowdsourced translations for 11 and machine translations for 16. In all, 7,565 edits were made, 36.9% of the samples reviewed; professional annotators edited an average of 789 samples per language (38.5%) and community contributors 362 (17.7%), a difference the authors attribute to time and resources rather than translation quality.
The CS and CA labels from the English annotation are propagated to every language, giving a 119,700-item MA subset, a 33,264-item CS subset and an 86,436-item CA subset across the 42 languages. Global-MMLU Lite is a smaller set of 200 CS and 200 CA questions per language for 15 fully human-translated or post-edited languages, 6,000 items in all. Both datasets are released under a permissive licence on Hugging Face (CohereForAI/Global-MMLU and CohereForAI/Global-MMLU-Lite).
Model evaluations
The authors evaluated 14 models from nine families in a 5-shot setting using lm-evaluation-harness for open models and API generation for GPT-4o and Claude 3.5 Sonnet, with prompt instructions in the language of the question. The models were grouped as small (Aya Expanse 8B, Gemma2 9B, SEA-LION v3 9B, Llama 3.1 8B, Mistral Nemo 12B, Qwen 2.5 7B), mid-size (Aya Expanse 32B, Command R 34B, Gemma2 27B, Qwen 2.5 32B), large (Llama 3.1 70B, Command R+) and closed (GPT-4o, Claude 3.5 Sonnet). The paper states that its aim is to compare behaviour on CS versus CA questions rather than to rank models, since sizes and evaluation methods differ.
The central finding concerns rankings. Measured against each model's rank on the MA subset, rankings on the CA subset changed by an average of 3.4 rank changes and 3.7 position shifts per language, whereas rankings on the CS subset changed by 5.7 rank changes and 7.3 position shifts. For high-resource languages the CS figures were 6.8 rank changes and 9.1 position shifts against 3.3 and a maximum shift of 3 on CA. Low-resource languages showed the largest individual movements, up to 5 positions for Malagasy and 13 ranking changes on Ukrainian. Large models were the most stable (average rank change 0.21 on CA, 0.67 on CS), mid-size models the least (0.33 and 1.97); small models showed little difference (0.35 and 0.45), which the authors attribute to weak performance on both subsets. Average accuracy was 51.3% (CA) and 54.8% (CS) for small models, 59.1% and 61.7% for mid-size, and 61.6% and 66.8% for large.
Across the full set, average accuracy is higher on CS than CA questions, which the paper attributes to CS questions being drawn mostly from Humanities and Social Sciences where models score better, whereas CA contains more Medical and STEM questions; on the balanced Global-MMLU Lite the pattern reverses and CS accuracy is lower with greater variance. The standard deviation of accuracy across languages is consistently higher on CS than CA and rises with lower resource levels: 3.21 (CA) and 3.86 (CS) for high-resource languages, 3.42 and 4.6 for mid-resource, and 6.37 and 6.78 for low-resource, increases of 98% and 75% over the high-resource figures.
Comparing human- and machine-translated CS data, models scored higher on human translations for French, the gap narrowed for Korean, and for Yoruba Claude 3.5 Sonnet and GPT-4o scored "significantly better" on machine-translated questions, which the authors attribute to reliance on machine-translated training data for low-resource languages; Aya Expanse 32B alone was consistent across both.
Recommendations and limitations
The paper recommends reporting on Global-MMLU instead of translated MMLU, and reporting CS and CA results separately. Its stated limitations are uneven community contributions across languages, coverage of only 42 of roughly 7,000 languages with no treatment of dialects, no flagging of toxic content, a six-region geographic taxonomy the authors would replace with the World Bank's, and that identifying cultural sensitivity "does not guarantee cultural inclusion."
Key claims
- 28% of a 2,850-question uniform sample of MMLU requires culturally sensitive knowledge; 792 questions are CS and 2,058 CA.
- 84.9% of geography-dependent questions concern North America (64.5%) or Europe (20.4%); 86.5% of culture-dependent questions concern Western culture, and 73.9% of those concern the United States.
- Global-MMLU covers 42 languages and 589,764 items; 14 languages have professional translations or post-edits, 11 community translations, 16 machine translations only.
- Model rankings shift more on the CS subset (5.7 rank changes, 7.3 position shifts on average) than on the CA subset (3.4 and 3.7).
- Cross-language variance is higher on CS than CA and roughly doubles for low-resource languages.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 of 2412.03304 (arxiv.org); the abstract page is arxiv.org. The datasets are at hf.co and huggingface.co.
Relationships
- supports: MMLU — the source for the cultural-bias figures and the Global-MMLU variant described there.
- depends-on: MMLU — the dataset is a translated and annotated derivative of MMLU.
- related: Cohere — Cohere For AI led the work.
- instance-of: AI Benchmarks and Evaluation