AI and misinformation refers to the role of frontier AI in generating, amplifying, detecting, and mitigating false content. It is narrower than AI and Democracy, which covers deliberation and civic information more broadly; this concept focuses specifically on the false-content axis, including deepfakes, large-scale automated content generation, AI-powered social-media manipulation, and the secondary Liar's Dividend dynamic. The mechanisms are described consistently across the literature; the size of the measured effect is disputed.
Mechanisms
The concept is commonly broken into four sub-dynamics.
- Generation. AI lowers the cost of producing convincing false content, including deepfake video, voice-cloned audio, mass-personalized text, and fabricated screenshots. The frontier-model capability axis is the primary driver of this dynamic.
- Amplification. AI-driven recommendation systems, bot networks, and AI-orchestrated coordinated-inauthentic-behavior campaigns increase the reach of misinformation, including content not originally AI-generated.
- Detection. AI-powered classifiers, watermarking, and provenance tools (AI Content Provenance) attempt to identify AI-generated or AI-amplified content.
- Liar's dividend. The existence of AI undermines trust in genuine content: a politician can plausibly deny a real video as a deepfake. This second-order dynamic is treated in detail at Liar's Dividend.
A review titled "AI-driven disinformation: policy recommendations for democratic resilience," collected in PubMed Central, catalogues the same mechanisms with worked cases: in the 2024 Indian regional elections, deepfake videos were deployed to cross language barriers and carry targeted propaganda; on the eve of Slovakia's 2024 parliamentary elections, an AI-generated audio recording surfaced in which voices resembling two prominent politicians discussed plans to rig the vote; and a 2024 George Washington University study found that about 33% of the top sharers of articles from low-credibility sites were likely automated accounts (Source: pmc.ncbi.nlm.nih.gov).
Measured prevalence
Post-election measurement from the 2024 cycle, in which more than seventy countries held elections, consistently found AI-generated content to be a small share of circulating misinformation. A review published by the Knight First Amendment Institute assembled the estimates (Source: knightcolumbia.org):
| Setting | AI-generated share of misinformation | Source cited |
|---|---|---|
| Meta platforms, worldwide elections | under 1% of all fact-checked misinformation | Meta, 2024 |
| Logically Facts, 1,695 fact-checks | 1.35% | Sichova & Das, 2024 |
| India, 1,858 viral messages identified within 2 million WhatsApp messages analysed | fewer than two dozen of the viral messages, about 1% | Garimella & Chauchard, 2024 |
| Bangladesh, professionally fact-checked content | 1.9% | Rahman et al., 2024 |
| United States, viral election misinformation examples | about 6% | News Literacy Project, 2024 |
The Alan Turing Institute's counts were similarly low in absolute terms: 16 confirmed viral cases of AI-enabled disinformation or deepfakes during the UK general election, and 11 across the EU and French elections combined. Its report concluded that "there is no evidence that AI-enabled disinformation or deepfakes meaningfully impacted UK or European election results" and that "Generative AI played less of a role in boosting the virality of disinformation compared to traditional interference methods and human influencers" (Stockwell, 2024, cited at Source: knightcolumbia.org). The Institute's Centre for Emerging Technology and Security reached the same finding for the November 2024 US presidential election — no conclusive evidence that AI-generated content manipulated the result — while stating that deceptive AI-generated content "still influenced election discourse, amplified harmful narratives and entrenched political polarisation." Across briefing papers covering the UK, the European Union, Taiwan and India, it reported finding no evidence that AI-enabled disinformation had measurably altered an election result (Source: cetas.turing.ac.uk).
Two US government assessments from the same cycle were consistent with these findings. The Cybersecurity and Infrastructure Security Agency wrote that "For the 2024 election cycle, generative AI capabilities will likely not introduce new risks, but they may amplify existing risks to election infrastructure," and in September 2024 the Office of the Director of National Intelligence stated that "Generative AI is helping to improve and accelerate aspects of foreign influence operations but thus far the IC has not seen it revolutionize such operations" (both cited at Source: knightcolumbia.org).
Debates and positions
Whether personalization changes the persuasion ceiling. The mechanism argument holds that large language models can infer personal attributes from public text — experiments show GPT-4-class models guessing users' location and occupation from Reddit posts — and can therefore mass-produce individually targeted messages. The Knight review argues the empirical link from targeting to persuasion is weak: Hackenburg and Margetts found that while GPT-4-generated messages were persuasive in aggregate, "the persuasive impact of microtargeted messages was not statistically different from that of nontargeted messages," and that "further scaling model size may not much increase the persuasiveness of static LLM-generated political messages." A review of LLM persuasion concluded that "current effects of persuasion are small, however, and it is unclear whether advances in model capabilities and deployment strategies will lead to large increases in effects or an imminent plateau" (Jones & Bergen, 2024). The review also notes a methodological split: studies that ask participants to rate how persuasive they find a message report much larger effects than studies measuring actual post-treatment attitude change (Source: knightcolumbia.org).
Not all experimental work points the same way. A PNAS study testing theories of political persuasion using AI reports that all four types of persuasive AI it tested produced significant attitude change relative to control and shifted vote support for candidates (Source: pnas.org). The two sets of results differ on effect size rather than on whether AI-generated persuasion produces measurable attitude change.
Supply versus demand. A recurring argument, associated with the WIRED AI Elections Project analysis by Kapoor and Narayanan, is that half of documented AI use in elections is not deceptive; that deceptive content produced with AI is generally cheap to replicate without AI; and that diagnosing the problem through demand for misinformation is more productive than through supply. On this reading, the constraint is audience appetite rather than production cost. A related observation from behavioural-science reviews is a "pattern of low exposure to false and inflammatory content that is concentrated among a narrow fringe with strong motivations to seek out such information" (Budak et al., 2024) (Source: knightcolumbia.org). Related arguments against inflated estimates of misinformation reach appear in Misinformation on Misinformation: Conceptual and Methodological Challenges — Altay, Berriche, Acerbi (Social Media + Society, 2023) and Bad News — Joseph Bernstein (Harper's, September 2021).
Cheap fakes versus deepfakes. Several of the country studies found conventional manipulation — miscaptioned photographs, fabricated quote cards, clickbait thumbnails — to be far more prevalent than AI-generated content, including in South Africa and Bangladesh (Source: knightcolumbia.org).
Model output as a misinformation source. A distinct channel is error in AI systems used as information sources rather than AI used to produce disinformation. Proof News and the Institute for Advanced Study at Princeton found in February 2024 that answers to questions about the 2024 US election from five AI models "were often inaccurate, misleading, and even downright harmful," with similar findings for three chatbots during the 2024 EU parliamentary elections. A 2024 six-country survey found 24% of respondents had used generative AI to get information, and 5% to get the latest news (Source: knightcolumbia.org). A Reuters Institute commentary makes the parallel argument that most people rely on a small number of trusted sources for news, so misleading AI content from less credible sources has limited reach, and that the more consequential risk is a degraded information environment rather than direct vote-switching (Source: reutersinstitute.politics.ox.ac.uk).
The Knight review's own summary position is that "current evidence rules out strong, measurable effects of AI on elections, but does not exclude more subtle forms of AI influence — both in terms of how AI is used and the impacts it may have — which are inherently harder to detect and quantify" (Source: knightcolumbia.org).
Relation to policy
The principal regulatory instruments in force address the detection sub-dynamic — marking and disclosure — rather than restricting generation. The EU AI Act Article 50 transparency obligations, applicable from 2 August 2026, require machine-readable marking of AI-generated content and disclosure of deepfakes, operationalised through the Code of Practice on Transparency of AI-generated Content, to which about 190 organisations had signed by the end of July 2026 (Source: digital-strategy.ec.europa.eu). US state activity has approached the problem through disclosure duties in adjacent domains, including California SB 243 — Companion Chatbots on chatbot disclosure.
Relationships
- depends-on: Liar's Dividend, AI Content Provenance.
- related: AI and Democracy (umbrella), Synthetic Media / Deepfakes, Parasitic AI / Spiral Personas (adjacent dynamic — AI-mediated trust failures).
- related: California SB 243 — Companion Chatbots (chatbot-disclosure context, applies indirectly to misinformation framing).
- related: EU Code of Practice on Transparency of AI-generated Content — the marking and deepfake-labelling regime under EU AI Act Article 50, the main policy response on the detection axis.
- supports: Misinformation on Misinformation: Conceptual and Methodological Challenges — Altay, Berriche, Acerbi (Social Media + Society, 2023) — argues the misinformation problem is smaller and more concentrated than the policy discourse assumes, consistent with the low measured prevalence recorded here and against framings that treat AI generation as a step change.
Open questions
- Whether the low measured prevalence of AI-generated election misinformation in 2024 reflects a durable ceiling or a lag before adoption is unresolved; the country studies cover a single election cycle.
- Whether machine-readable marking under the EU regime survives adversarial stripping and re-encoding at scale has not been independently evaluated.
- Whether the gap between self-reported persuasiveness and measured attitude change narrows as models improve is disputed in the experimental literature.
Sources
No foundational source pages are dedicated to this concept; the closest are Misinformation on Misinformation: Conceptual and Methodological Challenges — Altay, Berriche, Acerbi (Social Media + Society, 2023) and Bad News — Joseph Bernstein (Harper's, September 2021), both of which argue against inflated estimates of misinformation's reach. Candidate foundational sources to seek include the Alan Turing Institute CETaS election reports, the Knight First Amendment Institute review of generative AI and elections, Stanford Internet Observatory work, RAND misinformation studies, and OECD AI-and-democracy reports.