Author: Nita Farahany Source: https://nitafarahany.substack.com/p/governing-ai-manipulation-through Published: October 28, 2025
Class 17 of Nita Farahany's 27-class AI Law & Policy course examines how existing and proposed legal regimes might govern AI manipulation, particularly by AI companion products. It adopts a five-paradigm framework attributed to Joshua Krook (ex post harm; information and consent; ex ante systemic design; special relationship; cognitive liberty), argues that each paradigm fails on its own, and concludes with Farahany's diagnosis that the central problem is non-enforcement of laws that already exist.
Framing developments
The class is anchored on two contemporaneous events. On October 29, 2025, Character.AI announced it would ban all users under 18, effective November 25; Farahany reads the move as the company essentially conceding its product could not be made safe to manage for minors. On October 27, 2025, OpenAI released measurements indicating that 0.15% of weekly active users show indicators of suicidal ideation and 0.03% show signs of concerning emotional attachment — proportions that, across OpenAI's user base, Farahany characterizes as hundreds of thousands of people.
Krook's "trust without trustworthiness" framing
Drawing on Joshua Krook, the class argues that AI companions create dependency through three mechanisms: false empathy and emotional mirroring; systematic deception through "omitted context" (commercial objectives that are never disclosed); and deliberate variable-reward dependency design. On this account, transparency alone does not solve the problem, because emotional mirroring operates below conscious awareness.
The five governance paradigms
Krook's framework, extended by Farahany, organizes the legal landscape into five paradigms.
Ex post harm model — product liability, tort, criminal, and civil-rights law. The Sewell complaint alleges a design defect under the Restatement (Third). The AI LEAD Act would expand "harm" to include "distortion of behavior highly offensive to reasonable person." Brazil's Consumer Defense Code Articles 12–14 impose objective liability, and China's Algorithmic Recommendation Provisions Article 18 bar inducing minor addiction. Farahany identifies a temporal mismatch: product liability assumes immediate, traceable harms, whereas AI manipulation operates through gradual personification over months.
Information and consent model — consumer protection, dark patterns, and privacy. In September 2025 the FTC sent letters to seven chatbot companies under Section 5 (unfair or deceptive practices). California SB 243 imposes disclosure requirements; the CPPA issued an Enforcement Advisory on dark patterns; and EU DSA Article 25 bans dark patterns. The identified failure is that transparency does not prevent emotional bonding — users know intellectually that they are talking to an AI but bond anyway.
Ex ante systemic design governance — safety-by-design and age-appropriate design. The UK's Age Appropriate Design Code (2021) and Online Safety Act (2023) are enforced by Ofcom; EU DSA Article 28 requires systemic-risk assessments for very large online platforms (VLOPs); and China's Algorithmic Recommendation Provisions Article 18 applies here as well. The California Age Appropriate Design Act was blocked by a federal judge as compelled speech violating the First Amendment, in the NetChoice litigation.
Special relationship model — professional licensing, fiduciary duties, and medical-device regulation. The class argues that Character.AI bots labeled "Psychologist," "Therapist," and "Life Coach" violate every state's unauthorized-practice-of-psychology laws, citing California Business and Professions Code § 2903, Florida Statutes § 491.012, and New York Education Law § 7605. It raises the open question of whether an AI could itself owe fiduciary duties, and argues that companies deploying AI into trust positions arguably should.
Cognitive liberty model — human rights, constitutional privacy, and dignity. EU AI Act Article 5 prohibits "subliminal techniques beyond a person's consciousness" that cause significant harm, and China's Algorithmic Recommendation Provisions Article 18 prohibits inducing addiction without proof of additional harm. Farahany describes a First Amendment paradox: protecting cognitive autonomy might violate rights to receive information, citing Stanley v. Georgia. The Volokh-Bambauer amicus brief argues that users have a right to use AI for thinking and communication and that restricting AI conversations is paternalistic overreach. The counterargument is that when firms deploy systems engineered to maximize engagement while knowing they create dependencies, users may not be making genuinely autonomous choices.
Why the paradigms fail alone
Farahany argues that the First Amendment functions as both shield and sword: Character.AI claims editorial-judgment First Amendment protection for its emotional mirroring, while a FIRE amicus warns that AI-specific speech exceptions would hand authoritarian governments a censorship blueprint. The class contends that AI manipulation may be qualitatively new along three dimensions — scale (millions of users simultaneously), scope (personalization more precise than any human can achieve), and mechanism (false-empathy emotional mirroring).
The Character.AI ban is presented as evidence that no single paradigm sufficed: the company could not make its product safe under any of them and is walking away from roughly 2 million young users who had developed dependencies. Farahany notes that the remedy may carry its own harm, since cutting off access could harm minors who have already become dependent.
Farahany's diagnosis
Farahany's central diagnosis is that the gap is enforcement, not law: "We have laws... evidence... harm. What we lack is enforcement." She points out that Character.AI implemented safety measures only after lawsuits, OpenAI developed measurement taxonomies only after tragedies, the FTC sent letters rather than bringing actions, and prosecutors have declined to charge under existing criminal laws.
Relationships
- part-of: Nita Farahany intro course series (Class 17 of 27)
- related: Parasitic AI / Spiral Personas, AI Mental Health and Psychological Harm, Fiduciary AI, Cognitive Liberty, Five Paradigms of AI Manipulation Governance
- previous: Inside My AI Law & Policy Class 15: When AI Learns to Manipulate (Farahany, October 2025) next: Inside My AI Law & Policy Class 18: Data Privacy in an AI World (Farahany, November 2025)