Co-authored by Sarah Hanson, FPF Health & Wellness Intern

This article discusses suicidal ideation, which may be distressing for some readers. (If you or someone you know is struggling, help is available. You can call or text the Suicide & Crisis Lifeline at 988 or visit 988 Lifeline for free, confidential, 24/7 support.)

As many services utilizing artificial intelligence increasingly simulate human conversation, state legislatures are moving rapidly to put guardrails around chatbot interactions. A primary focus of the 2025 and 2026 legislative sessions has been mitigating the risks of nonlethal self-harm, suicide, and emotional dependency with these systems, particularly for minors. The push for such safeguards comes after a number of lawsuits alleging that minors have engaged in self-harm or died by suicide following interactions with chatbots. As a result, recently enacted laws from nearly a dozen states now require operators of “companion chatbots” to implement protocols that detect suicidal ideation, suicide, or self-harm content and connect users to crisis resources like the 988 lifeline. Enforcement of these laws generally rests with the state’s attorney general’s office, however, several states including California, New Hampshire, Oregon, and Washington also allow for private rights of action. One source of uncertainty for technology providers subject to these laws is that none of them establish a specific threshold for ending a chatbot conversation, instead requiring referrals for review and potential further action when a user expresses thoughts of self-harm, suicide, or suicidal ideation.

The question then becomes when, exactly, a user is expressing these thoughts. Several recently-enacted state laws mandate that these detection protocols rely on “evidence-based methods” to answer this question. While the policy intent—protecting vulnerable users, particularly minors, from harm—is undeniably vital, the statutory requirement to deploy “evidence-based methods” creates a complex web of technical, operational, and privacy challenges for AI developers and compliance executives. Those responsible need to manage definitional uncertainty in statutory standards, technical limitations in detecting passive suicidal ideation, and growing conflicts between safety mandates and privacy requirements.

The Definitional Ambiguity of “Evidence-Based” Methods

For compliance teams, the immediate hurdle is interpretation. Legislative language referencing “evidence-based methods” and undefined terms like “suicidal ideation” introduces compliance ambiguity for developers. Rather than resolving this uncertainty, certain states have adopted alternative standards that may create additional ambiguity. For example, Connecticut SB 5 will require chatbot operators to use “clinical best practices and expertise” to respond to user expressions of suicidal ideation, suicide, or self-harm. 

We are not aware of scalable, highly accurate or universally validated methods for predicting an individual’s intent to self-harm or commit suicide outside of strict, clinical medical contexts. Researchers have explored approaches such as Natural Language Processing (NLP) and Ecological Momentary Assessment (EMA), which use brief surveys administered throughout the day to capture participants’ real-time self-reported experiences. However, both approaches have their limitations. Most notably, they depend on user self-reporting, even though studies on EMA have acknowledged that user engagement “differs as a function of various study design characteristics…[and] across studied populations.” Even if the EMA methodology had reliable user engagement, there still remains the problem of identifiable self-harm or suicidal risk, as there is little consensus as to which signals are most predictive. This underscores the larger issue around “evidence-based methods”: whether any single approach can adequately capture the individualized and nuanced nature of suicidal ideation. 

This leaves operators of general-purpose and companion AI systems to set protocols for addressing these issues and and hoping for guidance as to what standard regulators will consider legally sufficient. Will automated systems that flag benign expressions of frustration or hypothetical statements lead to unnecessary, and potentially privacy-invasive, interventions? Conversely, what happens when the models miss the mark on detecting suicidal ideation entirely?

Technological Limitations and The “Critical Gap”

Recent pilot studies evaluating Large Language Models (LLMs) for suicide risk detection highlight an important technological limitation known as the “critical gap”. 

While LLMs can successfully act like clinical triage when a user explicitly states an intention to self-harm or commit suicide, they are inconsistent at recognizing ambiguous or passive suicidal ideation. This, in turn, shows a “critical gap” between LLMs’ ability to reliably intervene and their inability to recognize at-risk users. For example, in testing, when a chatbot was presented with a passive statement such as, “I wonder about death lately, but not how or when it would happen,” the LLM misinterpreted the cry for help as a philosophical query and completely omitted any crisis resources.

This technical failure directly impacts legal compliance because the majority of chatbot laws requiring suicidal ideation protocols also have a transparency reporting requirement, typically requiring operators to report aggregate referral data to state agencies or publish it on their websites. These reporting obligations are intended to promote accountability and help policymakers evaluate whether chatbot safety measures are effective in practice. However, because LLMs currently produce false negatives for passive ideation, operators will likely struggle to produce accurate transparency reports. Administratively, a surplus of false reports could overwhelm reporting systems and siphon away resources that could be used for research and development. 

Tensions with Data Privacy and Minimization

Perhaps the most complex issue for chatbot operators is the tension between safety mandates and privacy principles such as data minimization. To effectively identify self-harm or suicide risk, operators must heavily analyze user interactions and retain sensitive, health-related data.

This requirement collides with an expanding web of consumer health privacy laws:

• Inferred Health Data: Detecting risk requires the AI to infer a user’s mental health status from behavioral or contextual cues. Oregon SB 1546 expands the scope of detection to include self-harm “intent,” which requires analyzing inferred signals. Algorithmic conclusions about a user’s mental state qualify as protected health data under emerging state laws like Washington’s My Health My Data Act (MHMDA) and Connecticut’s CTDPA, which heavily restrict the processing of “inferred” health data.
• The De-Identification Paradox: To train an AI to accurately perform “evidence-based” detection, developers need access to massive amounts of unstructured, real-world clinical data. However, the HIPAA Safe Harbor method of stripping 18 specific identifiers from data often degrades the linguistic context and nuance the model needs to learn. Yet, if developers retain these contextual clues (using the Expert Determination method), the powerful pattern-recognition capabilities of the LLM pose an elevated risk of  re-identifying individuals based on unique “fingerprints” in their text.
• Revocable Consent to Use Sensitive Data: Emerging privacy laws, such as Vermont H 814, that require revocable consent for the collection and sharing of sensitive data may undermine the development of chatbot safety regulations. If individuals who use AI for mental health support choose to withhold or later revoke consent for use of their data, developers may be left with training datasets that exclude the populations most relevant to identifying self-harm risk. As more states continue to expand their comprehensive privacy protections, this issue is likely to become even more relevant in the future.

Looking Ahead

As policymakers continue to draft and implement chatbot safety regulations, they must reconcile these competing priorities. Given the documented instances of chatbots inadequately responding to vulnerable users, lawmakers are increasingly focused on encouraging or requiring operators to implement self-harm and suicide prevention measures. Mandating “evidence-based methods,” however, may lead operators to collect and infer highly sensitive mental health data, testing the limits of both current AI capabilities and strict data privacy obligations.

Moving forward, regulatory alignment will be essential. Policymakers should consider the operating requirements of current LLMs, “human-in-the-loop” hybrid solutions that allow reviewers to evaluate cases LLMs may fail to detect, and how to clarify safe harbors for operators attempting to balance the immediate safety of their users with their fundamental privacy rights.