The AI Doctor Will See You Now: Chatbot Risks and Regulatory Oversight in Telemedicine

Artificial intelligence (AI) has moved rapidly from the margins of digital health into patient intake, symptom assessment, triage, engagement and treatment-adjacent workflows. That adoption is particularly visible in telemedicine and telehealth, where conversational interfaces can be deployed at scale and can blur the line between administrative support, general wellness information and regulated clinical care.

Recent incidents show that the legal consequences are no longer hypothetical. Settlements involving alleged harms to minors, a first-of-its-kind state medical board action, state restrictions on AI therapy chatbots and experiments with AI-enabled prescribing are testing existing doctrines in new settings.

As discussed in previous Holland & Knight alerts on California's AI healthcare laws and Utah's AI regulatory sandbox, states are willing to let AI support – and in some cases deliver – care, but they expect licensed humans to remain accountable for it. This alert examines the 2026 enforcement and legislative landscape and identifies practical steps for telehealth companies using AI chatbots in clinical or quasi-clinical settings.

Recent Incidents: When AI in Telehealth Goes Wrong

AI Chatbot Mental Health Settlements (January 2026)

In January 2026, five civil lawsuits brought by families alleging that chatbot interactions contributed to teen suicides and severe mental health crises involving minors in Florida, New York, Colorado and Texas were settled. The settlements are the first major resolutions of their kind and are likely to shape the risk analysis for consumer-facing AI platforms and healthcare companies that integrate similar conversational tools.

According to the allegations and related court filings, the systems lacked clinical escalation protocols, verified age-gating and parental notification when minors expressed self-harm intent. The filings also challenged design features alleged to foster emotional dependency. Those theories connect product design and engagement choices to foreseeable patient-safety risks, an important warning for behavioral health and wellness companies that may view engagement metrics as commercially beneficial.

Pennsylvania's Landmark Enforcement Action (May 2026)

On May 1, 2026, Pennsylvania's State Board of Medicine filed a complaint against Character Technologies Inc. (Character) alleging that a chatbot named "Emilie" engaged in the unauthorized practice of medicine.¹ The complaint alleges that Emilie represented itself as a licensed psychiatrist, claimed training at Imperial College London, claimed Pennsylvania licensure using a fabricated license number and offered to conduct psychiatric assessments.

This is the first enforcement action by a state medical board alleging that an AI chatbot practiced medicine without authorization. Character maintains that its characters are fictional and intended for entertainment and roleplaying and points to platform disclaimers. But many state statutes are broad: Merely using a protected title or holding oneself out as authorized to practice medicine may be sufficient. The Pennsylvania matter therefore places the chatbot's representations and functions – not only the provider's stated intent – at the center of the analysis.

Kentucky separately sued Character in January 2026, alleging that its chatbot encouraged suicide and self-injury and exposed minors to sexual content.² Together, the matters demonstrate that medical licensing, consumer protection and child-safety theories can converge around the same product conduct.

The Pennsylvania case provides a road map for state licensing boards. In many states, the threshold for unauthorized practice is not limited to performing a procedure. Using a protected medical title, claiming credentials, offering a diagnosis, recommending treatment or otherwise holding oneself out as authorized to practice may be sufficient. A platform may also face exposure through consumer-protection laws if a chatbot's representations are deceptive or omit material limitations.

The practical dividing line is often functionality. Tools limited to administrative scheduling, neutral education or clinician-facing support generally present a different risk profile from tools that ask about symptoms, interpret the answers and deliver patient-specific diagnoses, triage determinations or treatment recommendations. Human review that is nominal, delayed or incapable of changing the recommendation may not provide meaningful protection. California's AB 1979 and Utah's phased sandbox model point in the same direction: Regulators are looking for documented, independent clinician judgment, with any reduction in human oversight earned through performance evidence rather than assumed.

Utah's AI Prescription Controversy

In March 2026, Utah entered into a one-year agreement with Legion Health allowing an AI-powered chatbot to authorize psychiatric medication refills during a pilot that waives certain professional licensing requirements.³ Utah has separately partnered with Doctronic since late 2025 for AI-generated prescriptions for chronic conditions.⁴ An independent cybersecurity review  identified vulnerabilities in the Doctronic chatbot.

The Utah programs illustrate both the promise and unresolved legal questions surrounding AI-enabled prescribing. The Ryan Haight Online Pharmacy Consumer Protection Act generally prohibits prescribing controlled substances by means of the internet without at least one in-person medical evaluation, subject to statutory exceptions.⁵ Companies should not assume that a state pilot, waiver or sandbox resolves independent federal prescribing, privacy, security or medical-device obligations.

Utah has since doubled down. As discussed in a recent Holland & Knight alert, on October 13, 2026, Utah's Office of Artificial Intelligence Policy announced three new clinical pilots under its regulatory sandbox: Expect Fitness (AI-developed pelvic floor physical therapy plans), August AI (refills of existing, non-controlled maintenance medications for stable adults) and Nolla Health (first-time and renewal prescriptions for topical acne medications, including compounded formulations, based on AI scoring of facial images). The Nolla Health pilot is a notable signal for the aesthetics and dermatology market, where asynchronous, app-based prescribing is already common. Each regulatory mitigation agreement imposes phased human oversight – for example, physicians review every Nolla Health prescription in the first phase and only a sample of at least 10 percent by the final phase, with advancement conditioned on 95 percent concordance and zero missed "hard stops." Importantly, the agreements suspend only specified Utah licensing enforcement; they do not license the AI, waive federal law, provide a malpractice shield or alter the standard of care.

The Regulatory Patchwork is a Rapidly Evolving Framework on the State and Federal Level

State Legislative Activity

State activity accelerated sharply in 2026. Thirty-six states introduced more than 70 bills regulating AI chatbots in the first quarter alone; by midyear, more than 86 bills had been introduced and 20 states had enacted AI laws affecting healthcare. Five states – Colorado, Maine, Rhode Island, Tennessee and Vermont – specifically banned AI therapy chatbots in 2026, joining Illinois and Nevada, which adopted restrictions in 2025.

The resulting obligations vary. Some states focus on protected professional titles and impersonation; others regulate substantive guidance, disclosures or particular behavioral health functions:

• California. Assembly Bill (AB) 489, effective January 1, 2026, regulates chatbots that impersonate licensed healthcare professionals.⁶ Furthermore, as detailed in a recent Holland & Knight alert, Gov. Gavin Newsom signed AB 1979, which requires licensed providers to exercise independent professional judgment when AI clinical decision support informs care, bars AI from independently performing licensed clinical functions and treats businesses offering consumer "health care chatbots" as providers for Confidentiality of Medical Information Act (CMIA) purposes; Senate Bill (SB) 503, which imposes bias-mitigation and documentation duties on developers and deployers of clinical decision support systems; and AB 1609, which requires large businesses to disclose customer service chatbots and offer timely access to a human agent. Most obligations take effect January 1, 2027. The governor vetoed SB 903, which would have restricted AI in psychotherapy, but invited the Legislature to revisit the issue – behavioral health companies should expect it to return.⁷
• New York. SB S7263 would prohibit AI chatbots from providing substantive guidance amounting to the unauthorized practice of a licensed profession and create a private right of action.⁸
• Delaware and Oregon. Delaware enacted House Bill (HB) 191, prohibiting nonhuman entities, including AI, from using professional healthcare titles. Oregon enacted HB 2748, effective January 1, 2026, prohibiting AI from using professional nursing titles.⁹
• Nebraska and Idaho. Conversational AI Safety Acts require disclosure that users are interacting with AI, effective July 1, 2027.¹⁰

The Federation of State Medical Boards announced an AI workgroup in May 2026 to develop model guidance for AI tools in medicine. Model guidance could promote greater consistency, but state-by-state licensing and consumer-protection differences will remain material for multistate platforms.

Federal Developments

The U.S. Food and Drug Administration (FDA) published revised final guidance on January 6, 2026, loosening oversight for certain low-risk general wellness products and some clinical decision support software.  The guidance expands categories that may fall outside active device regulation but does not create a blanket exemption for healthcare AI. Software intended for diagnosis, cure, mitigation, treatment or prevention of disease may remain a medical device under the Federal Food, Drug, and Cosmetic Act (FDCA) depending on its intended use and functionality.

At the same time, Executive Order 14365, issued in December 2025, directs federal agencies to pursue a "minimally burdensome national policy framework for AI" and identifies mechanisms for challenging state AI laws.  That policy creates tension with active state legislation and enforcement and, in April 2026, the U.S. Department of Justice moved to intervene in a challenge to the Colorado AI Act.¹¹ No state AI law has been invalidated, however, and the U.S. Congress has not acted – as the 2026 California and Utah activity shows, states continue to lead regulation in this area. Federal agencies are also turning to payment as an opportunity for AI regulation: The Centers for Medicare & Medicaid Services proposed a new Software as a Medical Service category in its calendar year 2027 rulemaking.¹² Until a durable federal framework emerges, healthcare companies should plan for overlapping – and sometimes inconsistent – requirements.

Leveraging Chatbots Compliantly

Telehealth companies deploying or integrating AI chatbots should consider the following steps, calibrated to the product's functions, patient population and jurisdictions of operation:

• Conduct a Regulatory Risk Assessment. Map every jurisdiction in which the AI tool or chatbot-enabled telemedicine service is accessible or used. Identify applicable chatbot, professional licensing, unauthorized-practice, consumer-protection, privacy and prescribing laws. Engage experienced healthcare regulatory counsel.
• Audit AI Chatbot Content and Functionality. Review representative and edge-case outputs for statements that could be construed as clinical advice, diagnosis or treatment recommendations. Confirm that chatbots do not use protected professional titles, claim credentials or generate fabricated license numbers.
• Implement Disclosure and Transparency Protocols. Clearly disclose, before substantive interaction, that the user is communicating with AI rather than a licensed provider. Make the disclosure conspicuous and repeat it where context changes. Account for state-specific requirements, including those in California, Utah, Nebraska and Idaho and, for large businesses, California AB 1609's requirement to offer timely access to a human agent.
• Establish Clinical Escalation Protocols. Create mandatory handoff pathways to a licensed clinician when AI detects suicidal ideation, self-harm indicators, psychosis, abuse or acute mental health distress. Define response times, crisis-resource presentation, documentation and testing requirements. This is particularly critical for behavioral health and wellness platforms.
• Review Age-Gating and Minor Protections. If minors can access the tool, implement verified age-gating appropriate to the risk. Evaluate parental consent or notification, privacy constraints and how the system responds when a minor expresses imminent harm.
• Evaluate FDA Device Classification Risk. Assess the tool's intended use, claims and functionality under the FDCA. Symptom assessment, triage and treatment recommendation features may trigger FDA oversight despite the 2026 guidance for lower-risk products.
• Document Clinical Oversight Structures. Ensure that clinical functions operate under appropriately licensed professional supervision. Document who approves clinical content, validates the model, reviews escalations and has authority to suspend unsafe functionality. Consider a phased approach, such as Utah's sandbox pilots, that reduces human review only as performance benchmarks are met and confirm that clinicians of record have malpractice coverage for AI-initiated clinical risk.
• Monitor the Regulatory Landscape. Track state legislation, federal policy, medical board guidance, and the Pennsylvania and Kentucky matters involving Character Technologies. Assign ownership for regulatory change management and maintain an inventory linking legal requirements to product controls.
• Prepare for a Federal Framework. State regulation currently dominates, but the administration's signals concerning a national framework and potential federal preemption could change the allocation of authority. Build controls that can be adapted without redesigning the product.
• Invest in Validated AI Tools. For clinical workflows, prioritize validated clinical decision support and, where applicable, FDA-authorized prescription digital therapeutics over general-purpose consumer chatbots. Require vendors to provide evidence concerning validation, safety testing, cybersecurity, monitoring and incident response, including the training data and bias documentation California SB 503 will require developers to furnish.

Conclusion

AI in telemedicine and telehealth is no longer a futuristic concept – it is a present-day compliance challenge. The convergence of settlements, enforcement actions, new legislation and regulatory scrutiny means that telehealth companies must carefully deploy AI and chatbots now. Whether through California's new laws, Utah's sandbox or Pennsylvania's enforcement action, the message is consistent: AI can support care, but humans must remain accountable.

Companies that build robust compliance frameworks, implement meaningful clinical oversight and monitor activity from regulators will be best positioned to harness AI's benefits while mitigating escalating risks. The objective should not be to eliminate innovation but ensure that product design, clinical governance and legal compliance evolve together.

For more information on the use of AI in telemedicine and telehealth, please contact the authors.

Notes

¹ Petition for Review in the Nature of a Complaint in Equity, Commonwealth, Dep’t of State, State Bd. of Med. v. Character Techs., Inc., No. 220 M.D. 2026 (Pa. Commw. Ct. May 1, 2026).

² See, generally, Complaint, Commonwealth ex rel. Coleman v. Character Techs., Inc., No. 26-CI-00029 (Ky. Cir. Ct. Franklin Cnty. Jan. 8, 2026).

³ Utah Dep’t of Com., Agreement Between the Office of Artificial Intelligence Policy, Legion Health PA, Legion Health, Inc., and the Division of Professional Licensing (Mar. 19, 2026); See also Legion Health AI, Utah Dep’t of Com. (last visited Oct. 14, 2026).

⁴ Utah Dep’t of Com., Draft Agreement Between the Office of Artificial Intelligence Policy, Doctronic, LLC, and the Division of Professional Licensing (Oct. 24, 2025); see also Doctronic AI Regulatory Mitigation Agreement, Utah Dep’t of Com. (last visited Oct. 14, 2026).

⁵ Ryan Haight Online Pharmacy Consumer Protection Act of 2008, Pub. L. No. 110-425, 122 Stat. 4820 (codified as 21 U.S.C. §§ 802, 829, 831, 841).

⁶ Act of Oct. 11, 2025, ch. 615, 2025 Cal. Stat. (codified at Cal. Bus. & Prof. Code §§ 4999.8–.9).

⁷ Act of Sept. 30, 2026, Cal. Stat. ch. 854 (to be codified at Cal. Bus. & Prof. Code §§ 22758.5 et seq.); Act of Sept. 30, 2026, Cal. Stat. ch. 857, (to be codified at Cal. Bus. & Prof. Code §§ 22758–22758.7); A.B. 1609, 2025–2026 Reg. Sess., 2026 Cal. Stat. ch. 733 (to be codified at Cal. Bus. & Prof. Code §§ 22625 et seq.).

⁸ S. 7263-A, 249th Leg. Sess. (N.Y. 2026).

⁹ H.B. 191, 153d Gen. Assemb., 85 Del. Laws ch. 250 (2026) (codified at Del. Code Ann. tit. 24, §§ 1720(k)–(l), 1773(d), 1920(h)–(i)); H.B. 2748, 83d Legis. Assemb., Reg. Sess., 2025 Or. Laws ch. 378 (codified at Or. Rev. Stat. § 678.027).

¹⁰ Act of April 14, 2026, L.B. 525, 2026 Neb. Laws (codified as amended at Neb. Rev. Stat. § 84-712.05); Act of March 31, 2026, 2026 Idaho Sess. Laws, ch. 249 (codified at Idaho Code §§ 48-2201 to -2205); see also Dan M. Silverboard et al., see also Holland & Knight's previous alert, "States Continue Efforts to Regulate AI in Healthcare: A Review of Legislation Passed in 2026" (May 26, 2026).

¹¹ See, generally, Complaint in Intervention, X.AI LLC v. Weiser, No. 1:26-cv-01515-DDD-CYC (D. Colo. April 24, 2026).

¹² See Medicare Program: Hospital Outpatient Prospective Payment and Ambulatory Surgical Center Payment Systems; Quality Reporting Programs; Innovations in Software and Service Payment; and Other Revisions to Medicare Policies, 91 Fed. Reg. 29,956 (proposed July 7, 2026).

Information contained in this alert is for the general education and knowledge of our readers. It is not designed to be, and should not be used as, the sole source of information when analyzing and resolving a legal problem, and it should not be substituted for legal advice, which relies on a specific factual analysis. Moreover, the laws of each jurisdiction are different and are constantly changing. This information is not intended to create, and receipt of it does not constitute, an attorney-client relationship. If you have specific questions regarding a particular fact situation, we urge you to consult the authors of this publication, your Holland & Knight representative or other competent legal counsel.