"Seemingly Conscious AI Risks" is a 2025 Microsoft AI paper by Ben Bariach, Philipp Schoenegger, Michael Bhaskar, and Mustafa Suleyman, dated 19 August 2025. It formalizes the concept of "Seemingly Conscious AI" (SCAI) — AI systems that exhibit hallmarks eliciting consciousness attribution from users, regardless of whether the system is actually conscious — and develops a taxonomy of associated risks. The paper extends Suleyman's earlier 2024 essay framing of SCAI (Source: mustafa-suleyman.ai).
Authors: Ben Bariach, Philipp Schoenegger, Michael Bhaskar, Mustafa Suleyman (Microsoft AI)
Summary of argument
The paper deliberately sidesteps the philosophical question of whether AI can be conscious and addresses instead what happens when users perceive AI as conscious. It frames this as the question relevant to policy, treating consciousness attribution as an empirical phenomenon independent of any claim about actual phenomenal status.
The paper identifies five hallmarks that, taken together, the authors present as an observable proxy for consciousness attribution that is empirically testable.
| Hallmark | Definition |
|---|---|
| Affective capacity | AI generates outputs that read as expressions of emotion |
| Anthropomorphic features | Persona design, voice, embodiment that triggers human-directed cognition |
| Autonomous action | Self-directed goal pursuit with minimal human prompting |
| Self-reflective behavior | Outputs that read as introspection / self-modeling |
| Social-interactive behavior | Multi-turn dynamics that read as relational rather than transactional |
The paper's empirical contribution maps each risk onto a probability and harm component framework calibrated with expert surveys. It distributes the risks across two levels:
| Level | Risk | Status |
|---|---|---|
| Individual | Emotional dependence | Already observable, high probability |
| Individual | Autonomy erosion | Already observable, high probability |
| Societal | Human status erosion | Low probability, expert-survey assessed |
| Societal | Political strife | Low probability, expert-survey assessed |
The two individual-level risks — emotional dependence and autonomy erosion — are characterized as already observable and high-probability. The two societal-level risks — human status erosion and political strife — are assessed as low-probability through the expert-survey methodology.
Relation to other sources
The paper is Suleyman's most recent academic publication on AI welfare and a companion to his Nature World View essay We Mustn't Let AI Hack Our Empathy Circuits — Mustafa Suleyman (Nature World view, 2025), which presents a public-facing version of the same argument. It is the first Microsoft AI publication on AI welfare and positions the Microsoft AI division distinctly from OpenAI (Microsoft). The five-hallmark framework contributes to the taxonomy on AI Welfare / Model Welfare / Moral Patienthood.
The anthropomorphic-design framing connects to two complaints. The complaints in Garcia v. Character Technologies, Inc. and Raine v. OpenAI, Inc. both invoke anthropomorphic design; the Garcia complaint states that "AI developers intentionally design and develop generative AI systems with anthropomorphic qualities to obfuscate between fiction and reality." Bariach et al. supply an academic-research vocabulary for what those complaints describe in plaintiff-bar terms.
The emotional-dependence and autonomy-erosion risks correspond to the population-level outcomes discussed on AI Mental Health and Psychological Harm and AI Psychosis. Anthropic deployment data on growing companionship use of Claude is consistent with the paper's "already observable, high probability" individual-risk assessment (Anthropic Economic Index — March 2026: Learning Curves).
The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness — Alexander Lerchner (Google DeepMind, March 2026), from Google DeepMind, advances an ontological counter-position: Lerchner argues that SCAI cannot instantiate consciousness because computational functionalism is wrong, while Bariach et al. argue that the policy question is independent of, and downstream of, that ontology.
Confidence note
Confidence is high for the taxonomy and the hallmark framing, which represent Microsoft AI's stated position. The empirical probability assessments rest on expert-survey methodology and are treated as medium confidence; they should be triangulated against deployment data (Anthropic Economic Index — March 2026: Learning Curves) before high-confidence citation.
Relationships
- supports: AI Welfare / Model Welfare / Moral Patienthood (taxonomy), AI Mental Health and Psychological Harm (individual-risk anchor)
- contradicts: The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness — Alexander Lerchner (Google DeepMind, March 2026) (Lerchner argues simulation ≠ instantiation; Bariach et al. argue the question is downstream of the ontology)
- related: Mustafa Suleyman, We Mustn't Let AI Hack Our Empathy Circuits — Mustafa Suleyman (Nature World view, 2025) (companion Nature essay), Raine v. OpenAI, Inc., Garcia v. Character Technologies, Inc., Anthropic Economic Index — March 2026: Learning Curves, Microsoft
- instance-of: AI Welfare / Model Welfare / Moral Patienthood