Author: Nita Farahany Source: https://staytuned.substack.com/cp/184600139 Published: January 13, 2026
An essay by Nita Farahany, published January 13, 2026, arguing that consumer AI chatbots functioning as authoritative health advisors occupy a legal grey zone: because they are not healthcare providers, they owe users no fiduciary duty even as tens of millions rely on them daily for health decisions. The essay introduces what it calls fiduciary AI as a policy gap and contends the migration of trust from accountable institutions to systems that disclaim responsibility is already at scale. Fuller framework coverage is at Fiduciary AI.
Summary of the argument
The essay sets the argument against a decline in trust in public health institutions. Trust in the CDC fell from 66% to 54% in under a year (KFF), a drop that Farahany notes extended even to Democrats. Into that gap, she argues, step consumer health chatbots: ChatGPT Health (OpenAI, launched January 7, 2026, timed to the JP Morgan Healthcare Conference) and Claude for Healthcare (Anthropic, days later). OpenAI says more than 230 million health-related questions are asked on ChatGPT each week — over 40 million per day — often outside normal healthcare hours.
Farahany characterizes the resulting behavior change as a shift from after-hours pediatrician hotlines and CDC websites to uploading a photo of a child's rash to ChatGPT at 2 a.m. and asking whether to drive to the ER. She cites a NEJM AI study reporting that people prefer AI medical responses to those written by doctors and rate even low-accuracy AI answers as valid, trustworthy, and satisfactory, which she links to acceptance of harmful advice and unnecessary medical attention.
The essay frames the regulatory situation as a gap between existing regimes. As Farahany states it:
We have HIPAA for clinical care. We have consumer protection laws for products. These intimate, personalized, always-available AI chatbots, which are now influencing health-care decision making of millions while disclaiming responsibility for them, fall into a legal grey zone. They aren't healthcare providers, so they have no fiduciary duty to their users.
She argues that what distinguishes these tools from prior AI health tools is the combination of fluency, personalization (drawing on uploaded medical records and wellness apps), always-on availability, and apparent authority, which lets the chatbots function as health authorities while remaining legally non-authoritative.
The essay positions itself as identifying a gap to be filled rather than as opposition to AI in healthcare. Farahany allows that chatbots can synthesize information, translate jargon, prepare patients for visits, and democratize access, but argues:
We are rapidly normalizing the transfer of trust from accountable institutions to systems that explicitly refuse accountability. We need to answer sooner, rather than later, what legal obligations should apply to tools that function as health authorities while claiming they are not one — especially when tens of millions of daily users already treat that product as a health advisor.
Key claims
- The legal grey zone. Farahany surveys why existing regimes do not reach these tools. OpenAI's position is that HIPAA does not apply because it is not a "covered entity." Anthropic offers a "HIPAA-ready" Claude for Healthcare for HIPAA-compliant organizations using Claude for Enterprise, but Farahany argues those protections likely do not apply to consumer use of Claude for Health. The FDA has clarified that many AI-enabled software tools fall outside medical-device regulation when clinicians can "independently review" the recommendations; Farahany argues this framing does not fit chatbot use that increasingly happens far from clinic walls. On privacy, she notes that both companies promise no foundation-model training on consumer health data, but observes the promises are only as good as their word and could change with the terms of service.
- Generalization beyond healthcare. The fiduciary AI argument generalizes to legal-advice AI, financial-advice AI, and any domain historically governed by fiduciary duty. Farahany describes the structure as identical: the AI functions as an authority while disclaiming the duty.
Provenance
Published January 13, 2026 on Farahany's Substack (https://staytuned.substack.com/cp/184600139) as an opinion essay advancing an original framework. Farahany's broader work on neurotechnology and mental privacy (Cognitive Liberty) informs the argument. As a single-source essay, it is treated as an attributed position rather than as independent factual evidence; the underlying data points (the KFF trust figures, the OpenAI usage figures, the NEJM AI study) are reported as Farahany cites them.
Relationships
- authored-by: Nita Farahany
- introduces: Fiduciary AI
- related: AI Mental Health and Psychological Harm, AI Liability, Healthcare — AI Deployment, Cognitive Liberty (Farahany's foundation)