"Measuring Perceived Slant in LLMs" is a 106-page working paper by Sean J. Westwood (Dartmouth), Justin Grimmer (Stanford / Hoover), and Andrew B. Hall (Stanford GSB / Hoover), published May 8, 2025 (full PDF at https://modelslant.com/paper.pdf; the raw source carries the first 10 pages verbatim). It measures the political slant of large language models as perceived by users rather than as an objective property of the models, drawing on 180,126 pairwise assessments collected from 10,007 US respondents. The study reports that nearly all leading models are perceived as left-leaning, including by many Democratic respondents.
Methodology
The authors treat perceived slant as a user-evaluator question rather than an algorithm-measurement question. They argue that existing methods often treat political slant as an objective property of models, whereas in their framing it varies with prompt, reader, timing, and context. Their design places real users in the role of arbiter: respondents read ecologically valid prompts on 30 political topics, see paired outputs drawn from 24 LLMs, and rate which output leans more left or right. The study evaluated 24 LLMs across 30 political topics, producing 180,126 pairwise assessments from 10,007 US respondents.
The authors describe the instrument as generalizing across users, topics, and model types, such that the same approach could be applied to other politically relevant outcomes.
Findings
The paper reports that nearly all leading models are perceived as significantly left-leaning, including by many Democratic respondents. One widely used model is perceived as leaning left on 24 of the 30 topics. OpenAI models drew the most intensely perceived left-leaning slant, which the authors describe as roughly four times greater than perceptions of Google models; Google models were perceived as the least slanted overall.
When models were prompted toward neutrality, their output became more ambivalent and users perceived more neutrality. In that condition, Republican users reported modestly increased interest in using the models.
The authors limit the claim to perceived slant and do not assert objective bias.
Reception and related work
The study is described as the first large-N user-evaluator study of LLM political slant. It serves as a companion to Manhattan Institute — Measuring Political Preferences in AI Systems (Rozado) (Rozado, January 2025), which measures bias from model outputs using four methods; the Stanford paper instead measures user perception via pairwise comparisons, and the two approach the question from different angles. The paper has been cited alongside the Trump White House executive order targeting "woke AI," though the authors limit their claim to perceived rather than objective bias (Source: washingtonpost.com). Its framing also intersects with the First Amendment theory in xAI LLC v. Weiser (challenging the Colorado AI Act), where xAI cites left-leaning AI as motivation.
Provenance
Working paper, 106 pages, published May 8, 2025. Authors: Sean J. Westwood (Dartmouth), Justin Grimmer (Stanford / Hoover), Andrew B. Hall (Stanford GSB / Hoover). Full PDF: https://modelslant.com/paper.pdf.
Relationships
- supports: AI Political Bias, AI and the First Amendment, America's AI Action Plan (WH executive order on "free-of-ideological-bias" AI).
- pairs-with: Manhattan Institute — Measuring Political Preferences in AI Systems (Rozado).
- contradicts: Anthropic's / OpenAI's framing of their models as politically neutral.
- related: xAI LLC v. Weiser (challenging the Colorado AI Act).
Sources
- Primary: Stanford GSB Measuring Perceived Slant in LLMs
- Full PDF: https://modelslant.com/paper.pdf