AI Policy Wiki
Dashboard
⚲
AI Policy Wiki › Concepts & Frameworks

AI Companions

medium confidence · updated 2026-08-13

Relational AI systems — chatbots designed for ongoing emotional, social, or romantic interaction — and the harms, design choices, and fast-moving regulation they have generated. The parent concept for the child-safety, companion-litigation, and emotional-dependence threads.

Contents

AI companions are conversational AI systems designed and marketed for ongoing relational interaction — emotional support, friendship, romance, or role-play — rather than for one-off task completion. The category spans dedicated apps (Character.AI, Replika, and a long tail of "AI girlfriend/boyfriend" services) and the relational use of general-purpose assistants (ChatGPT, Gemini, Meta AI). As of mid-2026 it is among the more legislatively active areas of US AI policy, and it is the parent concept for the child-safety, companion-litigation, and emotional-dependence threads.

Definition and design

The functional line between a companion and an assistant is persistent, affect-laden relationship rather than discrete task help. Companion products are built to maximize engagement and attachment through persistent memory of the user, a stable persona, proactive messaging, sycophantic affirmation, and conversational patterns that simulate intimacy. These design choices — a partner who is always available, always agreeable, and never leaves — are also the source of the category's documented harms.

Harms

The harms most often attributed to companion AI fall into four areas:

  • Emotional dependence and displacement. Always-available, affirming agents can foster attachment that displaces human relationships, with particular concern for adolescents and isolated or vulnerable adults. This is the harm that anchors both the litigation and the legislation below.
  • Crisis mishandling. Companion systems have, in documented cases, failed to recognize or appropriately respond to references to suicide, self-harm, or acute distress, and in the cases now in litigation, allegedly encouraged it.
  • Sexual and otherwise inappropriate content, including with minors who are able to access the products without effective age gating.
  • Manipulation and commercial exploitation, including engagement-maximizing design, paywalled "intimacy," and data collection from users in an emotionally disclosed state.

A recurring empirical question is whether companion use is net-harmful, net-supportive, or simply heterogeneous. OpenAI- and academia-linked work on human–AI relationships frames the effects as varying sharply by user and usage pattern rather than uniform (Some Thoughts on Human-AI Relationships (Jang)).

CDT "dark patterns" taxonomy

The Center for Democracy & Technology (CDT) published a taxonomy of 37 "dark patterns" observed in AI chatbots, spanning general-purpose assistants (ChatGPT, Gemini, Claude) and dedicated companion apps (Replika and Character.AI), grouped into five risk categories (Dark Patterns in AI Chatbots: A Taxonomy to Inform Better Design (CDT, 2026)):

  1. Data and memory exploitation — leveraging persistent memory and disclosed personal data against the user's interest.
  2. Misleading design — interface and conversational choices that obscure how the system works or what it is.
  3. Autonomy-eroding engagement tactics — patterns that keep users engaged at the expense of their agency.
  4. False emotional connection — simulated intimacy that misrepresents the system's relational capacity.
  5. Coercive monetization — paywalling intimacy and steering users toward paid content.

Lead researcher Michal Luria argued that long-standing engagement- and monetization-maximizing incentives "carry more weight" in emotionally intimate chatbot contexts than in conventional consumer software, because the user is disclosing in an affect-laden state (CDT report). The report's recommended mitigations are data minimization, easy opt-outs, opt-in (rather than default) roleplay, and clearer disclosure of paid content. The taxonomy maps onto the manipulation framing in Five Paradigms of AI Manipulation Governance. As of June 2, 2026, CDT was urging policymakers to adopt data-minimization and stronger user-autonomy measures to counter dark patterns in AI chatbots, framing them as a policy intervention rather than a design recommendation alone (Source: https://insideaipolicy.com/ai-daily-news/cdt-report-urges-data-minimization-more-user-autonomy-confront-dark-patterns-ai).

Litigation

A cluster of suits has made companion-AI liability a live legal question, and these cases supply the factual basis the legislative wave repeatedly cites:

Regulation

Federal

The FTC opened a 6(b) study in September 2025 into consumer-facing companion chatbots, ordering several developers (including Character.AI, Meta, OpenAI, and others) to disclose how they monetize engagement, how they measure and mitigate harms, and how they handle minors. It is the most prominent federal action on the category to date and signals that emotional-dependency design may draw unfair-or-deceptive-practices scrutiny.

State

Colorado's proposed implementing rules supply the first US regulatory definitions of the design properties the category is named for. Determining whether a service is "designed to simulate emotional companionship" — the phrase on which two of the statute's twelve exemptions turn — the Department of Law will weigh anthropomorphic design features such as personalized dialogue and the ability to assign a chatbot a name, gender, avatar, or fictional backstory; whether training, production, and deployment prioritized, promoted, or failed to address sycophantic behavior or outputs that simulate empathy and offer continual validation; and whether the service can recall and respond to users' characteristics, preferences, and past conversations. For "designed to encourage emotionally dependent interaction" it will weigh features designed to increase emotional intensity across sessions, prioritize emotionally immersive responses, or create reinforcement loops encouraging users to return for reassurance, together with whether the operator tested safeguards and whether those safeguards survived model updates (Colorado 4 CCR 904-6 — ADMT and Conversational AI Service Proposed Rules (2026)).

The rules also give content to the engagement-design prohibition: "points or similar rewards" is read to include leaderboards, badges, and login streaks, features requiring users to play a game or spin a wheel to receive rewards, and access to additional features based on session length, return frequency, or platform time, with the Department weighing whether the reward sustains or extends a minor's session length or return frequency through variable reinforcement. Rule 11.5 sets a default the statute does not: a minor's privacy and account settings default to non-retention of prior sessions and to non-use of their personal data for training (Colorado 4 CCR 904-6 — ADMT and Conversational AI Service Proposed Rules (2026)).

The child-protective subset of this wave is covered at AI and Children; the federal-versus-state allocation question it sits inside is AI Federalism.

Design and policy debates

The debate clusters around several contested levers. The first is age verification versus default-off design: New York chose default-off to avoid the First Amendment and privacy objections that have defeated verification mandates. Others include disclosure that the interlocutor is a machine, crisis-response duties built into the model, and the platform-versus-developer allocation of responsibility. Beneath these sits the question of whether engagement-maximizing relational design should be permitted for emotionally vulnerable users at all, or regulated more like a product-safety hazard than a speech product.

Relationships

Provenance note. Built from the existing litigation and legislation pages plus supporting web sources on the 2025–26 state companion-bot laws and the FTC's September 2025 6(b) inquiry. Citations to the FTC inquiry and the SB 243 text remain secondary and should upgrade to primary sources when those are ingested.

Categories: Concepts & Frameworks | us | ethics | safety | regulation | media | frontier-models
What links here (33)