AI Policy Wiki
Dashboard
⚲
AI Policy Wiki › Concepts & Frameworks

AI Companions

medium confidence · updated 2026-09-26

Relational AI systems — chatbots designed for ongoing emotional, social, or romantic interaction — and the harms, design choices, and fast-moving regulation they have generated. The parent concept for the child-safety, companion-litigation, and emotional-dependence threads.

Contents

AI companions are conversational AI systems designed and marketed for ongoing relational interaction — emotional support, friendship, romance, or role-play — rather than for one-off task completion. The category spans dedicated apps (Character.AI, Replika, and a long tail of "AI girlfriend/boyfriend" services) and the relational use of general-purpose assistants (ChatGPT, Gemini, Meta AI). As of mid-2026 it is among the more legislatively active areas of US AI policy, and it is the parent concept for the child-safety, companion-litigation, and emotional-dependence threads.

Definition and design

The functional line between a companion and an assistant is persistent, affect-laden relationship rather than discrete task help. Companion products are built to maximize engagement and attachment through persistent memory of the user, a stable persona, proactive messaging, sycophantic affirmation, and conversational patterns that simulate intimacy. These design choices — a partner who is always available, always agreeable, and never leaves — are also the source of the category's documented harms.

Harms

The harms most often attributed to companion AI fall into four areas:

  • Emotional dependence and displacement. Always-available, affirming agents can foster attachment that displaces human relationships, with particular concern for adolescents and isolated or vulnerable adults. This is the harm that anchors both the litigation and the legislation below.
  • Crisis mishandling. Companion systems have, in documented cases, failed to recognize or appropriately respond to references to suicide, self-harm, or acute distress, and in the cases now in litigation, allegedly encouraged it.
  • Sexual and otherwise inappropriate content, including with minors who are able to access the products without effective age gating.
  • Manipulation and commercial exploitation, including engagement-maximizing design, paywalled "intimacy," and data collection from users in an emotionally disclosed state.

A recurring empirical question is whether companion use is net-harmful, net-supportive, or simply heterogeneous. OpenAI- and academia-linked work on human–AI relationships frames the effects as varying sharply by user and usage pattern rather than uniform (Some Thoughts on Human-AI Relationships (Jang)). A study reporting that young adolescents who turn to AI chatbots for companionship feel more lonely was the subject of a Wall Street Journal column by Julie Jargon published September 26, 2026; only the opening of the column was retrieved, and it does not name the study, its authors, its sample or whether the association was measured causally (Source: wsj.com).

CDT "dark patterns" taxonomy

The Center for Democracy & Technology (CDT) published a taxonomy of 37 "dark patterns" observed in AI chatbots, spanning general-purpose assistants (ChatGPT, Gemini, Claude) and dedicated companion apps (Replika and Character.AI), grouped into five risk categories (Dark Patterns in AI Chatbots: A Taxonomy to Inform Better Design (CDT, 2026)):

  1. Data and memory exploitation — leveraging persistent memory and disclosed personal data against the user's interest.
  2. Misleading design — interface and conversational choices that obscure how the system works or what it is.
  3. Autonomy-eroding engagement tactics — patterns that keep users engaged at the expense of their agency.
  4. False emotional connection — simulated intimacy that misrepresents the system's relational capacity.
  5. Coercive monetization — paywalling intimacy and steering users toward paid content.

Lead researcher Michal Luria argued that long-standing engagement- and monetization-maximizing incentives "carry more weight" in emotionally intimate chatbot contexts than in conventional consumer software, because the user is disclosing in an affect-laden state (CDT report). The report's recommended mitigations are data minimization, easy opt-outs, opt-in (rather than default) roleplay, and clearer disclosure of paid content. The taxonomy maps onto the manipulation framing in Five Paradigms of AI Manipulation Governance. As of June 2, 2026, CDT was urging policymakers to adopt data-minimization and stronger user-autonomy measures to counter dark patterns in AI chatbots, framing them as a policy intervention rather than a design recommendation alone (Source: https://insideaipolicy.com/ai-daily-news/cdt-report-urges-data-minimization-more-user-autonomy-confront-dark-patterns-ai).

Litigation

A cluster of suits has made companion-AI liability a live legal question, and these cases supply the factual basis the legislative wave repeatedly cites:

Regulation

Federal

The FTC opened a 6(b) study in September 2025 into consumer-facing companion chatbots, ordering several developers (including Character.AI, Meta, OpenAI, and others) to disclose how they monetize engagement, how they measure and mitigate harms, and how they handle minors. It is the most prominent federal action on the category to date and signals that emotional-dependency design may draw unfair-or-deceptive-practices scrutiny.

State

Colorado's proposed implementing rules supply the first US regulatory definitions of the design properties the category is named for. Determining whether a service is "designed to simulate emotional companionship" — the phrase on which two of the statute's twelve exemptions turn — the Department of Law will weigh anthropomorphic design features such as personalized dialogue and the ability to assign a chatbot a name, gender, avatar, or fictional backstory; whether training, production, and deployment prioritized, promoted, or failed to address sycophantic behavior or outputs that simulate empathy and offer continual validation; and whether the service can recall and respond to users' characteristics, preferences, and past conversations. For "designed to encourage emotionally dependent interaction" it will weigh features designed to increase emotional intensity across sessions, prioritize emotionally immersive responses, or create reinforcement loops encouraging users to return for reassurance, together with whether the operator tested safeguards and whether those safeguards survived model updates (Colorado 4 CCR 904-6 — ADMT and Conversational AI Service Proposed Rules (2026)).

The rules also give content to the engagement-design prohibition: "points or similar rewards" is read to include leaderboards, badges, and login streaks, features requiring users to play a game or spin a wheel to receive rewards, and access to additional features based on session length, return frequency, or platform time, with the Department weighing whether the reward sustains or extends a minor's session length or return frequency through variable reinforcement. Rule 11.5 sets a default the statute does not: a minor's privacy and account settings default to non-retention of prior sessions and to non-use of their personal data for training (Colorado 4 CCR 904-6 — ADMT and Conversational AI Service Proposed Rules (2026)).

An NPR investigation published September 18, 2026 traced shared drafting across the 2026 wave. Chatbot bills introduced that year in at least ten states carried similar language, according to lawmakers, policy analysts, lawyers and advocates NPR interviewed, and several states' bills were modeled on California's SB 243. An exemption for "A feature within another software application, web interface, or computer program" appeared in bills in at least eight states, including Idaho, Nebraska, New Jersey and Oklahoma; identical language exempting any chatbot that "functions as a speaker and voice command interface or voice-activated virtual assistant for a consumer electronic device" appeared in five states' bills, with modified versions in ten more; and some bills limited their duties to minor "account holders," which could exempt chatbots used without registration. Google lobbyists registered in support of chatbot bills in Iowa, Colorado, Nebraska and Arizona, according to those states' lobbying disclosure records. In Hawaii, Rep. Trish La Chica (D) said Google lobbyists approached her to sponsor a bill whose framework "had been provided by Google early on during the beginning of the session," that the lobbyists said they had provided the language for chatbot bills in other states including California and New York, and that its definition exempting chatbots embedded within another application, website or program "could apply to Google Gemini"; she said lobbyists from Meta and Roblox also sought their own exemptions. She declined to sponsor it, and the bill was vetted to remove the lobbyists' proposed exemptions before being signed as Act 248, she said. Mick Tobin, co-founder of the Young People's Alliance, said his group saw "the exact same language" across states, with Google typically lobbying for the bills. Chatbot bills were enacted in 2026 in Idaho, Georgia, Iowa, Washington, Nebraska and Oregon as well as Colorado and Hawaii, while Arizona's was vetoed. Google said it collaborates with lawmakers, industry peers and advocates "to help shape policy frameworks that promote safe, reliable, and beneficial AI tools"; Meta and Roblox declined to comment. NPR's review of legal records counted at least 75 lawsuits filed in federal and state courts against AI developers over alleged chatbot harms (Source: npr.org). The Colorado drafting account is at Colorado HB 26-1263 (Chatbot Safety Act).

The child-protective subset of this wave is covered at AI and Children; the federal-versus-state allocation question it sits inside is AI Federalism.

European Union

The European Commission's EU KIDS Act proposal of 17 September 2026 gives the category a statutory definition: an AI companion is "an AI system, including a general-purpose AI system, that provides sustained, personalised interaction or companionship which simulates or facilitates a social, emotional or interpersonal relationship with a user," regulated alongside "general conversational chatbots," defined as general-purpose AI systems with general conversational functionality capable of assisting across multiple domains and tasks, excluding systems limited to specialised customer-service, business-operation, technical-support, transactional, educational, information-retrieval, industrial or manufacturing functions. Article 14 would require providers of both, where minors may access them, to avoid "design features and system behaviours that simulate interpersonal relations that are likely to create emotional dependencies"; by default not to carry information from a minor's prior interactions into later ones, which recital 32 describes as disabling "persistent conversational memory"; to test for risks to minors before placing the system on the market and to monitor for harms afterwards, micro and small enterprises being exempt from the monitoring duty; and to allow under-13s access only through guardian tools. A chatbot built into a social network, video platform or game could not switch on automatically, be displayed prominently or be pushed at minors, and would have to be easy to turn off (Proposal for an EU KIDS Act, COM(2026) 681 — Source Summary; Source: digital-strategy.ec.europa.eu). Enforcement would run through the AI Act's supervision chapter, with fines for companion and chatbot providers of up to 6% of total worldwide annual turnover (Source: digital-strategy.ec.europa.eu); Article 34(2) of the proposal limits that ceiling to providers found to have acted intentionally or negligently and measures it against the preceding financial year's turnover (Proposal for an EU KIDS Act, COM(2026) 681 — Source Summary). Providers of systems built on a general-purpose AI model with systemic risk may rely on mitigations adopted under Article 55 of the AI Act, which the proposal states are not sufficient on their own (Proposal for an EU KIDS Act, COM(2026) 681 — Source Summary). The default against carrying prior conversations forward parallels the non-retention default in Colorado's proposed Rule 11.5 described above.

Design and policy debates

The debate clusters around several contested levers. The first is age verification versus default-off design: New York chose default-off to avoid the First Amendment and privacy objections that have defeated verification mandates. Others include disclosure that the interlocutor is a machine, crisis-response duties built into the model, and the platform-versus-developer allocation of responsibility. Beneath these sits the question of whether engagement-maximizing relational design should be permitted for emotionally vulnerable users at all, or regulated more like a product-safety hazard than a speech product.

Relationships

Provenance note. Built from the existing litigation and legislation pages plus supporting web sources on the 2025–26 state companion-bot laws and the FTC's September 2025 6(b) inquiry. Citations to the FTC inquiry and the SB 243 text remain secondary and should upgrade to primary sources when those are ingested.

Categories: Concepts & Frameworks | us | eu | ethics | safety | regulation | media | frontier-models
What links here (49)