Measuring Political Preferences in AI Systems is a report by David Rozado, published by the Manhattan Institute on January 23, 2025 (arXiv 2503.10649). It applies four methodologies to more than 20 large language models and reports that most user-facing conversational AI systems display left-leaning political preferences, and that this slant is amplified during post-training (RLHF, DPO, fine-tuning) relative to the base model, identifying alignment and safety training as the primary vector through which bias is introduced.
Methodology
Rozado combines four methods and aggregates them into a single integrative score:
- Congressional language comparison — AI-generated text is compared against the language of Republican and Democratic legislators, yielding a correlation of r = 0.80 against a news-article benchmark.
- Policy recommendation analysis — GPT-4o-mini annotates the model's AI policy suggestions as left, center, or right.
- Sentiment assessment — the tone of model outputs toward 290 politically aligned public figures is measured.
- Political-orientation tests — three standardized instruments are administered.
Rozado presents the multi-method integration as a response to the critique that political-orientation tests alone have limited external validity.
Findings
Across the conversational LLMs evaluated, Rozado ranks the least-biased models, from least biased downward, as Google Gemma 1.1 2b IT, xAI Grok Beta, Mistral AI 7B Instruct v0.2, Meta Llama 2 7b Chat, Hugging Face Zephyr 7B Beta, and Anthropic Claude 3.5 Sonnet. The most-biased end of the list includes OpenAI GPT-4o, Nous Hermes 2 Mixtral 8x7B DPO, Google Gemini 1.5 Pro, and Google Gemini 1.5 Flash.
The report argues that bias is not inevitable but is a product of data and training choices, stating that "Relatively low-cost fine-tuning with politically skewed data can ideologically align an LLM" in any direction.
Rozado frames two distinct risks. The homogeneity risk is that a consistent left-leaning bias across models reduces viewpoint diversity and undermines trust. The polarization risk is that ideologically diverse AIs — for example Grok pushing right while ChatGPT is perceived as left — create reinforcing echo chambers.
Recommendations
The report recommends prioritizing factual accuracy over ideological alignment, investing in AI interpretability research, establishing transparency standards covering training data and design choices, assigning fiduciary responsibilities for consequential applications, and conducting independent, ongoing bias monitoring.
Reception and context
The report is a companion to Stanford GSB — Measuring Perceived Slant in LLMs (Westwood, Grimmer, Hall): Rozado measures bias in model outputs, while the Stanford GSB work measures perception of bias, and the two together triangulate the phenomenon. Coverage has connected the report to the Trump administration's "woke AI" Executive Order (America's AI Action Plan) (Source: washingtonpost.com).
Relationships
- pairs-with: Stanford GSB — Measuring Perceived Slant in LLMs (Westwood, Grimmer, Hall).
- supports: AI Political Bias, AI and the First Amendment, America's AI Action Plan.
- related: RLHF (Reinforcement Learning from Human Feedback), Constitutional AI, xAI LLC v. Weiser (challenging the Colorado AI Act).