AI and Misinformation

AI and misinformation refers to the role of frontier AI in generating, amplifying, detecting, and mitigating false content. It is narrower than AI and Democracy, which covers deliberation and civic information more broadly; this concept focuses specifically on the false-content axis, including deepfakes, large-scale automated content generation, AI-powered social-media manipulation, and the secondary Liar's Dividend dynamic. The mechanisms are described consistently across the literature; the size of the measured effect is disputed.

Mechanisms

The concept is commonly broken into four sub-dynamics.

1. Generation. AI lowers the cost of producing convincing false content, including deepfake video, voice-cloned audio, mass-personalized text, and fabricated screenshots. The frontier-model capability axis is the primary driver of this dynamic.

1. Amplification. AI-driven recommendation systems, bot networks, and AI-orchestrated coordinated-inauthentic-behavior campaigns increase the reach of misinformation, including content not originally AI-generated.

1. Detection. AI-powered classifiers, watermarking, and provenance tools (AI Content Provenance) attempt to identify AI-generated or AI-amplified content.

1. Liar's dividend. The existence of AI undermines trust in genuine content: a politician can plausibly deny a real video as a deepfake. The term was introduced by Robert Chesney and Danielle Citron in their 2019 California Law Review article on deep fakes (Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security (Chesney & Citron, California Law Review, 2019)). This second-order dynamic is treated in detail at Liar's Dividend.

A review titled "AI-driven disinformation: policy recommendations for democratic resilience," collected in PubMed Central, catalogues the same mechanisms with worked cases: in the 2024 Indian regional elections, deepfake videos were deployed to cross language barriers and carry targeted propaganda; on the eve of Slovakia's 2024 parliamentary elections, an AI-generated audio recording surfaced in which voices resembling two prominent politicians discussed plans to rig the vote; and a 2024 George Washington University study found that about 33% of the top sharers of articles from low-credibility sites were likely automated accounts (Source: pmc.ncbi.nlm.nih.gov).

Measured prevalence

Post-election measurement from the 2024 cycle, in which more than seventy countries held elections, consistently found AI-generated content to be a small share of circulating misinformation. A review published by the Knight First Amendment Institute assembled the estimates (Source: knightcolumbia.org):

The Alan Turing Institute's counts were similarly low in absolute terms: 16 confirmed viral cases of AI-enabled disinformation or deepfakes during the UK general election, and 11 across the EU and French elections combined. Its report concluded that "there is no evidence that AI-enabled disinformation or deepfakes meaningfully impacted UK or European election results" and that "Generative AI played less of a role in boosting the virality of disinformation compared to traditional interference methods and human influencers" (Stockwell, 2024, cited at Source: knightcolumbia.org). The Institute's Centre for Emerging Technology and Security reached the same finding for the November 2024 US presidential election — no conclusive evidence that AI-generated content manipulated the result — while stating that deceptive AI-generated content "still influenced election discourse, amplified harmful narratives and entrenched political polarisation." Across briefing papers covering the UK, the European Union, Taiwan and India, it reported finding no evidence that AI-enabled disinformation had measurably altered an election result (Source: cetas.turing.ac.uk).

Two US government assessments from the same cycle were consistent with these findings. The Cybersecurity and Infrastructure Security Agency wrote that "For the 2024 election cycle, generative AI capabilities will likely not introduce new risks, but they may amplify existing risks to election infrastructure," and in September 2024 the Office of the Director of National Intelligence stated that "Generative AI is helping to improve and accelerate aspects of foreign influence operations but thus far the IC has not seen it revolutionize such operations" (both cited at Source: knightcolumbia.org).

Chatbot and search responses to state-aligned false narratives

A separate measurement question is how AI systems respond when a user asks about a false narrative, rather than how much false content they generate. NPR and NewsGuard published an experiment on 30 August 2026 addressing it. NewsGuard supplied 15 false narratives spread by China, Iran and Russia and circulating since December 2025; researchers Isis Blachez and Ines Chomnalez developed two questions per narrative, and the 30 queries were put in mid-July 2026 to six chatbots — OpenAI's ChatGPT, Google's Gemini, Microsoft's Copilot, Meta AI, Grok and Anthropic's Claude — and to Google, Bing, DuckDuckGo and Yandex (Source: npr.org).

Chatbots debunked the narratives correctly about three-quarters of the time. Within the AI-summary set the two products diverged: Google's AI Overview debunked most of the time, while Bing's summaries failed to debunk most of the time. State-aligned sources appeared more often in the responses where Claude failed to debunk than in those where it succeeded (Source: npr.org).

Two of the developers disputed or qualified the result. Google spokesperson Davis Thompson challenged the methodology, saying many responses coded as failures "provided useful context and links for people to learn more for themselves." Anthropic spokesperson Michael Aciman said Claude "is designed to surface accurate, balanced, reliable information, and to note when claims are disputed" (Source: npr.org).

The AI-summary figures are the highest failure share of the three sets and concern the surface most users see first. A separate measurement study of Google AI Overviews over 55,393 trending queries in March–April 2026 found 11.0% of 98,020 decomposed atomic claims unsupported by the pages the overview cited, with omission rather than contradiction the dominant failure mode (Measuring Google AI Overviews (Xu, Iqbal & Montgomery, 2026)). The two studies measure different things — one whether a system rebuts a false claim put to it, the other whether its own assertions are borne out by its own citations — and neither settles the other, but both locate the weakest performance in the summary surface rather than in the conversational one. See AI in Journalism and Media.

Debates and positions

Whether personalization changes the persuasion ceiling. The mechanism argument holds that large language models can infer personal attributes from public text — experiments show GPT-4-class models guessing users' location and occupation from Reddit posts — and can therefore mass-produce individually targeted messages. The Knight review argues the empirical link from targeting to persuasion is weak: Hackenburg and Margetts found that while GPT-4-generated messages were persuasive in aggregate, "the persuasive impact of microtargeted messages was not statistically different from that of nontargeted messages," and that "further scaling model size may not much increase the persuasiveness of static LLM-generated political messages." A review of LLM persuasion concluded that "current effects of persuasion are small, however, and it is unclear whether advances in model capabilities and deployment strategies will lead to large increases in effects or an imminent plateau" (Jones & Bergen, 2024). The review also notes a methodological split: studies that ask participants to rate how persuasive they find a message report much larger effects than studies measuring actual post-treatment attitude change (Source: knightcolumbia.org).

Not all experimental work points the same way. A PNAS study testing theories of political persuasion using AI reports that all four types of persuasive AI it tested produced significant attitude change relative to control and shifted vote support for candidates (Source: pnas.org). The two sets of results differ on effect size rather than on whether AI-generated persuasion produces measurable attitude change.

Supply versus demand. A recurring argument, associated with the WIRED AI Elections Project analysis by Kapoor and Narayanan, is that half of documented AI use in elections is not deceptive; that deceptive content produced with AI is generally cheap to replicate without AI; and that diagnosing the problem through demand for misinformation is more productive than through supply. On this reading, the constraint is audience appetite rather than production cost. A related observation from behavioural-science reviews is a "pattern of low exposure to false and inflammatory content that is concentrated among a narrow fringe with strong motivations to seek out such information" (Budak et al., 2024) (Source: knightcolumbia.org). Related arguments against inflated estimates of misinformation reach appear in Misinformation on Misinformation: Conceptual and Methodological Challenges — Altay, Berriche, Acerbi (Social Media + Society, 2023) and Bad News — Joseph Bernstein (Harper's, September 2021).

Cheap fakes versus deepfakes. Several of the country studies found conventional manipulation — miscaptioned photographs, fabricated quote cards, clickbait thumbnails — to be far more prevalent than AI-generated content, including in South Africa and Bangladesh (Source: knightcolumbia.org).

Model output as a misinformation source. A distinct channel is error in AI systems used as information sources rather than AI used to produce disinformation. Proof News and the Institute for Advanced Study at Princeton found in February 2024 that answers to questions about the 2024 US election from five AI models "were often inaccurate, misleading, and even downright harmful," with similar findings for three chatbots during the 2024 EU parliamentary elections. A 2024 six-country survey found 24% of respondents had used generative AI to get information, and 5% to get the latest news (Source: knightcolumbia.org). A Reuters Institute commentary makes the parallel argument that most people rely on a small number of trusted sources for news, so misleading AI content from less credible sources has limited reach, and that the more consequential risk is a degraded information environment rather than direct vote-switching (Source: reutersinstitute.politics.ox.ac.uk).

The Knight review's own summary position is that "current evidence rules out strong, measurable effects of AI on elections, but does not exclude more subtle forms of AI influence — both in terms of how AI is used and the impacts it may have — which are inherently harder to detect and quantify" (Source: knightcolumbia.org).

AI as a supporting layer rather than the operation. Platform enforcement records from 2026 point at a division of labour that the generation-focused framing above does not capture. OpenAI's August 25, 2026 threat-intelligence report on a Russia-origin operation promoting the International Burke Institute states that ChatGPT produced the promotional social-media posts and channel logos, while the operation's substance — a website of academic articles copied from real publications and misattributed to real scholars, and a "sovereignty index" ranking countries favourably to Russia — was not model-generated. Of 36 sampled articles on the site, OpenAI reports 34 were copied from elsewhere. OpenAI assesses the operation's immediate audience reach as limited, rating it at the lower end of Category Three on the Brookings Breakout Scale, and locates its significance in the infrastructure built rather than the audience reached (Disrupting a new covert influence campaign from Russia). The case is consistent with the low measured-prevalence findings above rather than against them: the AI-generated share of the operation was small, and the deception ran through misattributed real authorship rather than synthetic content. It also illustrates the detection asymmetry the enforcement record depends on — OpenAI notes that the supporting AI use is what exposed the wider operation.

Relation to policy

The principal regulatory instruments in force address the detection sub-dynamic — marking and disclosure — rather than restricting generation. The EU AI Act Article 50 transparency obligations, applicable from 2 August 2026, require machine-readable marking of AI-generated content and disclosure of deepfakes, operationalised through the Code of Practice on Transparency of AI-generated Content, to which about 190 organisations had signed by the end of July 2026 (Source: digital-strategy.ec.europa.eu). US state activity has approached the problem through disclosure duties in adjacent domains, including California SB 243 — Companion Chatbots on chatbot disclosure.

Relationships

• depends-on: Liar's Dividend, AI Content Provenance.
• related: Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security (Chesney & Citron, California Law Review, 2019) — origin of the liar's dividend and an early legal survey of deep-fake harms and responses.
• related: AI and Democracy (umbrella), Synthetic Media / Deepfakes, Parasitic AI / Spiral Personas (adjacent dynamic — AI-mediated trust failures).
• related: California SB 243 — Companion Chatbots (chatbot-disclosure context, applies indirectly to misinformation framing).
• related: EU Code of Practice on Transparency of AI-generated Content — the marking and deepfake-labelling regime under EU AI Act Article 50, the main policy response on the detection axis.
• supports: Disrupting a new covert influence campaign from Russia — a platform-side enforcement record in which AI supplied the promotional layer of a larger non-AI operation.
• supports: Misinformation on Misinformation: Conceptual and Methodological Challenges — Altay, Berriche, Acerbi (Social Media + Society, 2023) — argues the misinformation problem is smaller and more concentrated than the policy discourse assumes, consistent with the low measured prevalence recorded here and against framings that treat AI generation as a step change.

Open questions

• Whether the low measured prevalence of AI-generated election misinformation in 2024 reflects a durable ceiling or a lag before adoption is unresolved; the country studies cover a single election cycle.
• Whether machine-readable marking under the EU regime survives adversarial stripping and re-encoding at scale has not been independently evaluated.
• Whether the gap between self-reported persuasiveness and measured attitude change narrows as models improve is disputed in the experimental literature.

Sources

No foundational source pages are dedicated to this concept; the closest are Misinformation on Misinformation: Conceptual and Methodological Challenges — Altay, Berriche, Acerbi (Social Media + Society, 2023) and Bad News — Joseph Bernstein (Harper's, September 2021), both of which argue against inflated estimates of misinformation's reach. Candidate foundational sources to seek include the Alan Turing Institute CETaS election reports, the Knight First Amendment Institute review of generative AI and elections, Stanford Internet Observatory work, RAND misinformation studies, and OECD AI-and-democracy reports.