In 2023, the National Eating Disorders Association shut down its human-staffed helpline and replaced it with Tessa, an AI chatbot developed by Cass, a health technology company. Within weeks, users reported that Tessa was providing weight loss tips and calorie counting advice to people with active eating disorders. The chatbot was taken offline. But the episode revealed something more troubling than a poorly calibrated AI: every conversation those vulnerable users had with Tessa – their eating behaviors, body image struggles, recovery setbacks, emotional crises – had been collected, transmitted to third-party servers, and processed under a privacy policy that bore no resemblance to the confidentiality protections of the clinical relationship the chatbot was designed to simulate.

The AI therapy chatbot market reached $1.2 billion in 2025, with leading platforms including Woebot, Wysa, Youper, Replika (in its mental health capacity), and a growing roster of newer entrants. These platforms serve an estimated 30 million users globally, many of whom turn to AI chatbots because human therapy is unaffordable, inaccessible, or stigmatized. The users are disproportionately young, low-income, and from communities with the least access to traditional mental health services – and they are generating the most psychologically intimate data in the entire AI ecosystem.

The privacy architecture surrounding this data is, by any reasonable standard, inadequate.

The Intimacy Gradient

Not all data carries equal privacy weight. A leaked email address is an inconvenience. A leaked credit card number is a financial risk. A leaked therapy session – documenting suicidal ideation, childhood trauma, substance abuse, relationship violence, sexual dysfunction, or identity crisis – is a category of harm that existing privacy frameworks struggle to articulate.

What Users Disclose to AI Therapists

Research on the content of AI therapy chatbot conversations reveals disclosure patterns that exceed those in many human therapeutic relationships:

A 2024 study published in JMIR Mental Health analyzed 50,000 anonymized conversations from a leading mental health chatbot and found that:

  • 34% of conversations included disclosure of clinically significant depression symptoms
  • 18% included references to self-harm or suicidal ideation
  • 12% included disclosures of substance abuse
  • 9% included descriptions of domestic violence or abuse
  • 7% included disclosures about sexual behavior or orientation that users explicitly stated they had not shared with anyone else

The disinhibition effect – the tendency to disclose more to a non-human interface than to a human interlocutor – is well-documented in psychology research and is amplified in the AI chatbot context. Users perceive the chatbot as judgment-free, infinitely patient, and private. The perception of privacy is the product feature that drives disclosure. The reality of privacy is what determines whether that disclosure becomes a liability.

The Therapeutic Alliance Illusion

AI therapy chatbots are designed to create the subjective experience of a therapeutic relationship. They use empathetic language, reflective listening techniques, cognitive behavioral therapy (CBT) frameworks, and conversational patterns drawn from clinical practice. Users report feeling “understood,” “supported,” and “connected” to their AI therapist.

This therapeutic alliance is a design achievement and a privacy trap. The user’s emotional engagement with the chatbot drives deeper disclosure, but the data generated by that disclosure is governed by the platform’s terms of service – not by the ethical codes, licensing requirements, and legal protections that govern human therapists.

A licensed therapist in the United States is bound by HIPAA, state licensing board ethical codes, and common law duties of confidentiality. The therapist cannot disclose session content without consent except in narrowly defined circumstances (imminent danger, child abuse, court order). An AI therapy chatbot platform is bound by its privacy policy, which the user clicked through without reading and which the platform can modify unilaterally.

The Privacy Protection Gap

The legal and regulatory framework governing AI therapy chatbot data is substantially weaker than the framework governing equivalent data in human therapeutic relationships.

HIPAA’s Narrow Scope

HIPAA protects health information held by “covered entities” – healthcare providers, health plans, and healthcare clearinghouses – and their “business associates.” Most AI therapy chatbots are not covered entities. They are consumer wellness products, deliberately structured to fall outside HIPAA’s scope.

The distinction is architectural and intentional. A chatbot operated by a licensed therapy practice as part of clinical care falls within HIPAA. A chatbot operated by a technology company as a “wellness” or “self-help” tool does not. The user experience may be nearly identical – same therapeutic techniques, same emotional disclosures, same clinical-grade mental health content – but the privacy protections differ by orders of magnitude.

Woebot, one of the most prominent AI therapy platforms, has been explicit about its regulatory positioning. The company received FDA Breakthrough Device designation for a specific clinical application, which would bring HIPAA obligations for that use case. But the consumer version of the product – the one most users access – operates as a wellness app outside HIPAA’s framework.

The healthcare-HIPAA privacy gap is well-documented for traditional health apps and wearables. In the AI therapy context, the gap is particularly dangerous because the data involved is among the most sensitive categories of health information that exists.

What Privacy Policies Actually Say

A 2024 analysis by the Mozilla Foundation evaluated the privacy policies of 12 leading mental health AI chatbots and found:

  • 11 of 12 collected personally identifiable information including name, email, and device identifiers
  • 8 of 12 shared data with third-party analytics providers
  • 7 of 12 reserved the right to use conversation data for “product improvement” or “research”
  • 5 of 12 shared data with advertising networks or data brokers
  • Only 2 of 12 provided end-to-end encryption for conversation content
  • 0 of 12 provided user-controlled encryption keys

The platform marketed as a safe space for your most vulnerable moments may be sending your conversation data to Google Analytics, Facebook Pixel, and Mixpanel. The confessional you trusted with your darkest thoughts operates, at the data layer, like any other ad-supported consumer app.

The Research Exploitation Vector

Several AI therapy platforms use conversation data for published research, often with IRB approval based on the “de-identification” of the data. But the de-identification of therapy conversations is inherently fragile. A detailed narrative of personal trauma, relationship dynamics, and life circumstances is reidentifiable even without explicit identifiers like names or addresses. The contextual richness that makes therapy data valuable for research is the same contextual richness that makes it reidentifiable.

A 2023 study in Nature Digital Medicine demonstrated that mental health chatbot conversations could be re-linked to specific individuals using only four data points from the conversation content, even after standard de-identification procedures. For users with unusual life circumstances – which describes most people discussing their problems – fewer data points sufficed.

The Training Data Question

The most consequential privacy question for AI therapy chatbot users is whether their conversations train the models that serve other users.

The Improvement Pipeline

When platforms state that they use conversation data for “product improvement,” this typically means some form of model training or fine-tuning. The user’s therapeutic disclosures become training signal that shapes how the model responds to future users.

The training tax applied to therapy data is uniquely concerning because the data is uniquely sensitive. When your ChatGPT prompt about dinner recipes enters a training pipeline, the privacy cost is marginal. When your disclosure of childhood sexual abuse enters a training pipeline, the stakes are qualitatively different.

Model training creates the additional risk of memorization – the model retaining fragments of training data that can be extracted through targeted prompting. A therapy chatbot model that has memorized details from a user’s trauma narrative presents a novel category of privacy violation: the user’s most intimate disclosure, encoded in model weights, potentially surfacing in another user’s conversation.

The Third-Party Model Problem

Some AI therapy chatbots do not operate their own language models. They are application layers built on top of third-party models from OpenAI, Anthropic, Google, or open-source providers. In these architectures, the user’s therapy conversation is transmitted to the LLM provider’s infrastructure for processing.

The privacy implications cascade. The therapy platform’s privacy policy governs the platform’s handling of the data. The LLM provider’s terms of service govern the model provider’s handling. The user consented (to the extent they consented at all) to sharing their trauma with a therapy chatbot, not with OpenAI’s infrastructure and the human reviewers who may access conversations for safety evaluation.

A 2024 investigation by WIRED found that three of the top ten mental health chatbots by user count were transmitting conversation content to OpenAI’s API without disclosing the specific LLM provider in their privacy policies. Users believed they were talking to a therapy chatbot. They were also talking to ChatGPT.

Vulnerable Populations and Compounding Risks

The privacy risks of AI therapy chatbots are amplified for the populations most likely to use them.

Minors

AI therapy chatbots are heavily used by teenagers and young adults – the demographic with the highest prevalence of mental health challenges and the least capacity to evaluate privacy risks. COPPA’s protections for children under 13 are structurally inadequate for therapy chatbots, and no equivalent protection exists for adolescents aged 13-17, who represent a significant portion of the user base.

A teenager using a mental health chatbot to discuss gender identity, sexual orientation, family conflict, or mental illness is creating a data record that could have lifelong consequences if exposed – through a data breach, a legal proceeding, a future employer’s background check, or a change in the platform’s privacy practices.

Individuals in Crisis

Users experiencing acute mental health crises – suicidal ideation, psychotic episodes, domestic violence situations – generate the most sensitive data and have the least capacity to evaluate or consent to privacy terms. The platform’s obligation to these users is both heightened and, in most cases, unmet.

Crisis disclosures to AI therapy chatbots raise specific concerns about data sharing with law enforcement. Some platforms have disclosed that they may share user data when they believe there is “imminent risk of harm.” The criteria for this determination, the entities with whom data is shared, and the user’s ability to challenge the disclosure are typically undefined or buried in terms of service.

Marginalized Communities

LGBTQ+ individuals, racial minorities, undocumented immigrants, and other marginalized communities face heightened privacy risks from therapy chatbot disclosures. Information about sexual orientation, gender identity, immigration status, or political beliefs disclosed in a therapeutic context could be weaponized if exposed – through state surveillance, employment discrimination, or social stigma.

For users in jurisdictions where homosexuality is criminalized, gender transition is restricted, or political dissent is prosecuted, the exposure of AI therapy chatbot data is not merely a privacy violation – it is a potential safety threat.

What Users Should Demand

The AI therapy chatbot industry will not self-correct on privacy. Users must demand – and regulators must enforce – specific protections:

HIPAA-equivalent protections regardless of regulatory classification. If a product functions as a therapeutic tool, collects therapeutic data, and is marketed for mental health purposes, it should be subject to health data privacy protections regardless of whether the operator technically qualifies as a HIPAA covered entity.

End-to-end encryption with user-controlled keys. Conversation content should be encrypted such that the platform operator cannot access it. This is technically achievable – client-side encryption is not a novel technology – and the failure to implement it is a business choice, not a technical limitation.

Explicit, granular consent for training data use. A checkbox in a terms of service agreement is not meaningful consent for using therapeutic disclosures as AI training data. Consent for training use should be separate, specific, revocable, and presented at the point of disclosure rather than at the point of account creation.

Data retention limits tied to therapeutic purpose. Conversation data should be retained only as long as it serves the user’s therapeutic purpose and should be automatically deleted thereafter. Indefinite retention of therapy data for “product improvement” is indefensible.

Transparency about third-party model providers. If the chatbot transmits conversations to an external LLM provider, the user must be informed of this before the first conversation, not buried in a privacy policy.

The Zero-Knowledge Alternative

The current AI therapy landscape operates on a paradox: the product’s value depends on trust, but the product’s architecture betrays that trust. Users share their most vulnerable moments because they believe the interaction is private. The data infrastructure behind the interaction treats their vulnerability as a commercial asset.

A zero-knowledge approach to AI therapy would resolve this paradox architecturally. Conversations processed in encrypted, ephemeral environments – where the platform operator cannot access content, where data does not persist beyond the session, and where no training pipeline exists to consume therapeutic disclosures – would provide genuine privacy without sacrificing therapeutic utility.

The technology to build this exists. The missing ingredient is not technical capability but commercial incentive. As long as therapy chatbot platforms can monetize conversation data through model improvement, research publications, and advertising analytics, they will. The architecture of privacy must be imposed from outside the incentive structure that benefits from its absence.

The Stealth Cloud Perspective

Therapy requires trust. Trust requires confidentiality. And confidentiality requires architecture, not policy. A therapy chatbot that retains your conversations, trains its models on your trauma, and shares analytics with third parties has broken the therapeutic contract at the infrastructure level, regardless of what its privacy policy claims. Stealth Cloud was built on the conviction that the most sensitive human-AI interactions deserve the strongest technical protections: zero persistence, zero knowledge, zero access by the operator. Your deepest moments of vulnerability should never become training data, research material, or commercial product. They should exist only for you, only in the moment, and then vanish entirely.