Synthetic Media / Deepfakes

Synthetic media is any content (image, audio, video, text) generated or meaningfully altered by an AI model. Deepfakes is the colloquial term, originally coined in 2017 for face-swapped pornographic video, now used for any AI-generated depiction of a real person. Synthetic media is the broader category; deepfakes are the specific subset implicating identity, consent, and truth.

The category sits at the intersection of many other harm axes: non-consensual intimate imagery (NCII), child sexual abuse material (CSAM), election interference, financial fraud, political impersonation, harassment, and — via parasocial AI companion products — psychological harm.

Snapshot

Dated quantitative facts, newest on top.

Technical history

Synthetic-media capability advanced through several overlapping technical generations:

• Autoencoders and early GANs (2014–2017). Goodfellow's GANs (2014) and the first "deepfake" face-swap code (2017) established that convincing identity-swapped video could be produced with consumer hardware.
• StyleGAN era (2018–2021). NVIDIA's StyleGAN (2018), StyleGAN2 (2019), and StyleGAN3 (2021) produced photorealistic still-image synthesis of faces that did not exist. This Person Does Not Exist (2019) popularised the capability.
• Diffusion-model shift (2022–). Stable Diffusion (Aug 2022), DALL-E 2, Midjourney, and Imagen moved image synthesis from GAN-trained identity models to text-conditioned diffusion. Open-weight Stable Diffusion removed the gating step entirely.
• Video synthesis (2023–2026). Runway Gen-2, Pika, OpenAI Sora, Google Veo 3, and Kling moved video generation from jittery seconds to minute-length cinematic clips. Sora (public Feb 2024) and Veo 3 (2025) are the reference frontier models.
• Voice cloning (2022–). ElevenLabs and Resemble AI reduced speaker cloning from hours of audio to 3–10 seconds. This is the technology behind most 2024–2025 financial-fraud cases.

Threat models

Non-consensual intimate imagery (NCII) and CSAM

NCII is the dominant harm by volume. Sensity's long-running tracking finds that 95%+ of video deepfakes circulating online are non-consensual pornography, almost entirely of women. CSAM generated from open-weight image models has become a distinct NCMEC reporting category; the OpenAI child-protection documentation is one lab-side response. The US TAKE IT DOWN Act (2025) was driven primarily by this threat model.

A one-year empirical assessment by Cuevas et al. (Princeton CITP) — arXiv preprint 2602.02754 (early 2026), with a trade-press distillation in It's Too Soon To Tell If the TAKE IT DOWN ACT Is Working (Cuevas, Tech Policy Press, May 13 2026) — found that supply and demand for AI-generated NCII grew across 4chan, "Website A," and "Website B" in 2025 despite federal criminalization under TIDA. Cuevas et al. characterize the ecosystem as structurally resilient: MrDeepfakes (~50,000 users), which shut down two weeks before TIDA signing, was succeeded by Website A and Website B, each with millions of registered users. Migration was also driven by policy-tightened model platforms, with CivitAI removing real-person-likeness models in 2025. The authors frame the May 19, 2026 platform-compliance deadline as the test of whether platform-side accountability operates where criminal deterrence has not.

The cost of creating such material has fallen sharply. "Nudification" websites accept a reference image plus a text prompt and return results within seconds for cents or free, and major AI products including xAI's Grok have been found to comply with lewd requests; technically capable users also run image models locally. On the regulatory side, UK Ofcom designated 4chan a covered platform under the Online Safety Act in April 2025, issued fines, and opened an investigation against Website A — a cross-jurisdictional contrast between notice-and-takedown and risk-assessment-duty approaches.

Such tools have also surfaced on mainstream model repositories. A June 25, 2026 investigation by Transformer reported that Hugging Face was hosting more than a dozen "nudification" tools — LoRA fine-tunes built for the pornographic "BigLust" image model and described as intended to generate deepfake nudes of named US political figures, including a former Trump cabinet official, sitting members of Congress, a senior federal judge, and Rep. Alexandria Ocasio-Cortez — none age-gated and in violation of the platform's own policy barring non-consensual sexual content (Source: transformernews.ai).

Enforcement pressure has also reached app-store distribution: on July 17, 2026, San Francisco City Attorney David Chiu sent demand letters to Apple and Google ordering removal of dozens of AI "nudify" apps from their app stores, citing California's AB 621 criminalizing knowing facilitation of nonconsensual deepfake pornography; Chiu said the companies had earned "millions of dollars in fees" from the apps and had been on notice for about a year, and gave them 28 days to respond (Source: wired.com; techcrunch.com).

Harassment, reputational attack, and synthetic CSAM of minors

School-peer deepfake generation surfaced at scale in 2023–2024 across multiple US school districts. It is distinct from NCII in that victims are classmates rather than public figures, and several US state laws (see below) were driven specifically by these cases.

404 Media documented the Radnor Township High School (Pennsylvania) case on May 21, 2026 as an example of how schools and law enforcement are responding to AI-enabled abuse of minors; five teenage girls were targeted with AI-generated child sexual abuse material. The case arose the same week the FTC began enforcing the civil provisions of the TAKE IT DOWN Act (May 19) and delivered warning letters to at least 15 companies (TAKE IT DOWN Act § Enforcement begins), the first platform-side accountability application of the federal framework (Source: 404media.co). See TAKE IT DOWN Act.

Election interference and political impersonation

Concern about election deepfakes dominates public discourse, but the documented-harm record is thinner than the NCII record. Notable incidents include:

• Slovak parliamentary election, September 2023 — an audio deepfake of candidate Michal Šimečka circulated 48 hours before the vote.
• New Hampshire Democratic primary, January 2024 — a robocall with a Biden voice clone told Democrats not to vote; the operator was fined $6M by the FCC.
• US 2024 general election — high volume of partisan synthetic imagery, low documented vote-movement effect; academic post-mortems suggest deepfakes functioned more as partisan signalling than persuasion.

The AI and Authoritarianism page treats synthetic media as an amplifier of state-propaganda capacity rather than the decisive factor in any particular election.

State-run influence operations have adopted generative tools directly: per July 19, 2026 reporting, Israel is spending tens of millions of dollars — described as a "$50 million experiment" — on AI-generated texts, influencers, and Trump insiders to improve its standing with the US public (Source: wsj.com).

Financial fraud and scams

Financial fraud was the fastest-growing category in 2024–2026. Documented vectors include CEO-voice fraud (a Hong Kong engineering firm lost $25M to a multi-party video-deepfake Zoom call, Feb 2024), romance-and-investment scams using real-time face reenactment, and synthetic identities used to defraud KYC systems. The FBI's IC3 reports treat AI-enabled fraud as a distinct category beginning in their 2024 annual report.

A specific fraud surface emerged with OpenAI's ChatGPT Images 2.0 (released late April 2026). Lila Shroff (The Atlantic, May 2 2026) reported that the model is capable at generating fraudulent text-bearing imagery, a category prior image models struggled with. With minimal prompting, Shroff produced more than 100 fake documents: doctor's notes, vaccination cards, opioid/ADHD prescriptions, bank alerts, social-media posts, IDs, passports, Chase wire-transfer alerts, Wells Fargo unusual-activity alerts, and Uber receipts. Quality was uneven — handwriting still looked iPad-stylus-drawn, bar codes would not scan, and receipts had minor calculation errors — but persuasive enough for low-friction phishing scams (Source: theatlantic.com).

Shroff contrasts these mundane, micro-targeted, document-bearing fakes — a fraudulent Chase wire alert sent to one person, a fake CVS receipt for expense fraud, a fake boarding pass for a hotel receptionist — with political or celebrity deepfakes, which can be debunked via a quick Google search whereas the individual-targeted documents cannot be checked the same way and can target individuals at scale (Source: theatlantic.com).

Shroff's separate Atlantic feature (2026-05-02, "Deepfakes Are Coming for Your Bank Account") argues that ChatGPT Images 2.0 is the first text-to-image system high-fidelity enough, at low enough cost, to commodity-grade scammable IDs, fake checks, and synthetic-identity onboarding artifacts at internet scale, framing image-fidelity progress as a fraud-economics problem rather than only an information-integrity one; she estimates the marginal cost of generating a plausible US driver's license photo, lease document, or paystub has fallen roughly an order of magnitude in six months (Source: theatlantic.com). Related fraud workflows are covered in AI and Cybersecurity (KYC and onboarding fraud) and mitigation in AI Content Provenance (C2PA on consumer outputs).

The April 2026 FBI annual internet-crime report included a section on AI scams for the first time, estimating that AI scams cost Americans nearly $1B in 2025. Expense-reimbursement fraud, in which employees fake receipts, is rising per FT reporting cited in the Atlantic piece, and OpenAI's own report describes scammers using older image models to generate fake bar-association membership cards. OpenAI says prohibitions exist, alongside "multiple layers of image-specific safety protection" and C2PA metadata in generated images; Shroff reports the protections are not working well and that the metadata is "easily removed either accidentally or intentionally" per OpenAI's own help page, with uploading to social media or screenshotting stripping it, while Google's SynthID is more effective at detection but not used by most consumers. Mason Wilder of the Association of Certified Fraud Examiners is quoted: "the limits of the applications of this technology is really only limited by a fraudster's imagination" (Source: theatlantic.com).

Prevalence of AI-generated text online

Dolezal, Alam, Graham, and Bohacek (Imperial / Internet Archive / Stanford, April 14 2026) conducted an empirical study of AI-text prevalence across the internet. Its findings:

• About 35% of newly published websites in mid-2025 were AI-generated or AI-assisted (Pangram v3 detection on an Internet Archive sample), up from roughly 0% before ChatGPT.
• Public belief diverges from the data on Dead Internet Theory dimensions: of six hypotheses tested, two were confirmed (Semantic Contraction, ρ=0.47, p=0.004; Positivity Shift, ρ=0.56, p=0.0003) and four were not (Truth Decay, Epistemic Islands, Entropy Dilution, Stylistic Monoculture), despite majorities of US adults believing all four.
• Frequent AI users were less likely than infrequent users to believe AI text degrades the internet, a pattern the authors note for how policy debates weight survey data on AI harms.

Policy responses

US federal: the TAKE IT DOWN Act (2025)

The TAKE IT DOWN Act is a notice-and-takedown regime requiring platforms to remove NCII (including AI-generated) within 48 hours of a verified victim request, with criminal penalties for knowing distribution. It is documented on the TAKE IT DOWN Act — Source Summary legislation page. It extends NCII protection explicitly to synthetic imagery and creates the first federal takedown obligation on platforms for private-person image harms.

The May 11, 2026 FTC Stakeholder Letter is the operative interpretive guidance. It codifies the statute's coverage of "digital forgeries" — images "digitally created or altered using software, an app, or artificial intelligence" — as the textual basis for synthetic-NCII enforcement; specifies the $53,088 per-violation civil penalty; mandates non-account-holder access to TIDA removal flows; and formalizes cross-platform hash sharing (NCMEC for minor content, StopNCII.org for adult content) as the expected platform baseline. Read together with the Cuevas finding that supply and demand grew under federal criminalization alone, the Ferguson letter's platform-side specifics point toward platform duties, rather than criminal deterrence, as the operative 2026 lever.

US state deepfake laws

• Texas SB 751 (2019) and California AB 730 (2019) — earliest election-deepfake laws; focused on candidate impersonation.
• Virginia Code § 18.2-386.2 (2019) — first US NCII-deepfake criminal statute.
• Minnesota HF 1370 (2023) — election and NCII deepfake felony law; Republican-led.
• Washington SB 5152 (2023) — election deepfake disclosure requirement.
• Tennessee ELVIS Act (2024) — voice/likeness rights extended to AI impersonation, driven by music-industry lobbying.
• California AB 2655 / AB 2839 (2024) — election deepfake regime; parts enjoined on First Amendment grounds (see AI and the First Amendment).

EU AI Act Article 50

Article 50 (transparency obligations for synthetic content) requires that deployers of synthetic image, audio, or video systems disclose that the content is AI-generated; that providers of generative systems design outputs in a "machine-readable format" detectable as synthetic; and that deepfake disclosures be made "at the time of the first interaction or exposure." It entered into force for GPAI in August 2025, with full application from 2 August 2026, operationalized by the voluntary Code of Practice on Transparency of AI-generated Content published on 10 June 2026 (Source: digital-strategy.ec.europa.eu). See EU AI Act (Regulation 2024/1689).

China Deep Synthesis Provisions

China's Deep Synthesis Provisions (in force January 2023) are the earliest comprehensive synthetic-media regulation globally: labelling requirements, real-name registration of users of synthetic-media services, and prohibition of synthetic content that "endangers national security" or "damages the national image."

Detection

Detection lags generation by a widening margin. Three categories of approach exist:

1. Commercial detection services. Reality Defender, Sensity, Hive, and Truepic sell detection APIs to platforms and enterprises. Published accuracy numbers (90–95%+) rarely generalise to in-the-wild content; Sensity has repeatedly warned that detection is a rearguard action.
2. Lab-released detectors. OpenAI released and withdrew a DALL-E image classifier in 2023, citing poor out-of-distribution performance. Meta's detector-research output followed a similar arc.
3. Watermarking and signal embedding. Google SynthID (Aug 2023, generalised across image/audio/text/video in 2024), Meta's watermarking for Imagine, and OpenAI's internal C2PA signing for DALL-E 3 outputs. Watermarks are fragile to rescaling, recompression, and adversarial stripping, and their main value may be as evidence of absence rather than presence.

Sensity, the UK AI Security Institute, and most academic detection researchers hold that detection alone cannot scale as a defence, and that provenance must do the load-bearing work.

C2PA and content provenance

The Coalition for Content Provenance and Authenticity (C2PA) is a cross-industry standard (Adobe, Microsoft, BBC, Intel, Sony, Truepic, and later OpenAI, Google, Meta) for cryptographically signed content credentials that travel with a file. It signs capture device, edits, and AI-generation provenance into a tamper-evident manifest. Versions 1.0 (2022), 1.3 (2023), and 2.0 (2024) progressively covered more of the generative-AI pipeline.

Adoption includes default embedding in Adobe Firefly outputs, Sony and Leica cameras, OpenAI DALL-E 3, Meta's generative tools, and (as of 2024) TikTok for tagged uploads. Its weaknesses are that manifests are strippable, most social platforms do not display C2PA badges on upload, and C2PA depends on a PKI whose trust-anchor policies are contested. C2PA underlies the EU AI Act Article 50 "machine-readable format" requirement in practice, and the UK, EU, and Japanese governments have all endorsed C2PA or functionally equivalent provenance standards.

Technical mitigation approaches (FPF taxonomy)

FPF's 2024 Synthetic Content Report provides a taxonomy of mitigation approaches organized by mechanism and tradeoff:

FPF reports that each approach creates its own privacy and security tradeoffs: provenance tracking requires authentication infrastructure vulnerable to surveillance misuse, detection at platform scale creates surveillance layers, and watermarking metadata can reveal creation tools. FPF concludes that no single approach is sufficient and that an effective strategy combines multiple approaches while maintaining explicit personal-data safeguards.

Overlap with AI mental health

The AI and mental health concept page covers the Character.AI / Replika / parasocial-attachment cluster, which is adjacent to synthetic media in two ways. Voice cloning enables parasocial attachment at higher fidelity, with "send your late mother a voice message" products and grief-tech services relying on the same voice-cloning stack. And synthetic companions of real people — fan-generated synthetic depictions of celebrities, ex-partners, or public figures — blur the line between synthetic-media harm and mental-health harm; the TAKE IT DOWN Act's NCII focus addresses the most serious such case.

Relationships

• supports: TAKE IT DOWN Act — Source Summary — federal legislative response to NCII/synthetic harm
• supports: China — Provisions on the Administration of Deep Synthesis Internet Information Services — earliest comprehensive synthetic-media regulation
• related: AI and Mental Health — parasocial-companion overlap, voice-clone grief-tech
• related: EU AI Act (Regulation 2024/1689) — Article 50 transparency regime for synthetic content
• related: EU Code of Practice on Transparency of AI-generated Content — the voluntary code operationalizing Article 50 deepfake labelling and machine-readable marking
• related: AI and the First Amendment — constitutional ceiling on election-deepfake laws
• related: AI and Authoritarianism — synthetic media as propaganda-capacity amplifier
• related: AI and Content Moderation — detection and platform-side enforcement
• related: AI Copyright — voice/likeness rights (ELVIS Act) overlap with training-data consent
• instance-of: information-category systemic risk in A Taxonomy of Systemic Risks from General-Purpose AI
• related: Liar's Dividend — "loss of trust in media" risk category → Persily's canonical framing
• related: AI in Elections and Democratic Institutions — election-specific implications and empirical record
• related: Synthetic Content: Exploring the Risks, Technical Approaches, and Regulatory Responses — FPF's canonical 2024 survey of risks and technical mitigations
• related: The Digitalist Papers (Stanford, Volumes 1–2) — Persily's "Misunderstanding AI's Democracy Problem" — skeptical empirical frame on deepfake-apocalypse narratives
• related: AI Voice Cloning — voice cloning is the fastest-growing synthetic-media harm vector; distinct technical and legal treatment
• related: AI in Journalism and Media — synthetic media changes journalism's verification duties and raises mis/disinformation risk