Keeping Humans in the Loop: California Sets New Ground Rules for AI in Healthcare

During the 2025-2026 legislative session, California lawmakers prioritized the regulation of artificial intelligence (AI), including the use of AI in the healthcare industry. Of the dozens of AI bills considered, five key healthcare-related measures reached California Gov. Gavin Newsom's desk in September, with a September 30, 2026, deadline to sign or veto.

The governor signed three bills – Assembly Bill (AB) 1979, Senate Bill (SB) 503 and SB 947 – and vetoed two others: AB 2575 and SB 903. With these choices, human review, bias documentation and professional oversight are now a baseline for AI in California healthcare. Though the vetoed bills did not become law, Gov. Newsom commended the underlying policy goals in both veto messages and urged the California Legislature to revisit them.

This Holland & Knight alert summarizes the legislative backdrop and each of the five bills and provides practical considerations for providers, digital health companies, AI developers and investors.

The Backdrop: AB 1609 Puts Chatbots on Notice

On September 28, 2026, two days before the healthcare AI package, Gov. Newsom signed AB 1609 (Zbur) as part of a set of "commonsense" consumer bills. AB 1609 is not a healthcare-specific bill, but it previewed the theme that would define the governor's healthcare AI decisions: People – not bots – must remain accessible and accountable. The law applies to "large private businesses," defined as businesses with more than $500 million in gross annual revenue nationally that provide goods and services to customers. Many healthcare companies fall well within this category.

Covered businesses may not represent a customer service chatbot as human and must disclose when a customer is interacting with a chatbot rather than a person. They must also make a good-faith effort to connect a customer to a human agent within 15 minutes of a request during regular business hours (or schedule an appointment) and post customer service contact information on their websites.

Importantly for the industry, the law exempts hospital communications related to the provision, coordination, management, administration, payment or operation of healthcare services. Violations are subject to civil fines of up to $5,000 for an initial violation and $10,000 for each subsequent violation, though there is no private right of action. The law also includes exceptions for unforeseen circumstances, emergencies and channels not intended for general customer use. AB 1609 takes effect on January 1, 2027.

3 New Healthcare AI Laws

AB 1979: The Clinician Makes the Final Call

AB 1979 broadly requires health facilities, clinics and physicians to take "reasonable steps" to ensure that a licensed healthcare provider exercises independent clinical judgment in patient care whenever that care is informed by the output of a "clinical decision support system." AB 1979 defines a "clinical decision support system" (CDS system) as "an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decisionmaking related to timing of care, diagnosis, or treatment." In addition to this obligation, AB 1979 specifically prohibits healthcare providers from utilizing AI systems to: 1) independently perform clinical functions reserved for licensed professionals or 2) direct, guide, supervise or instruct unlicensed personnel in performing those functions. The law does contain exemptions for tools used for administrative functions such as scheduling and appointment reminders.

Violations of the act constitute unprofessional conduct (including the unlawful practice of medicine) subject to enforcement by the Medical Board of California and other professional boards. Notably, AB 1979 also amends the Confidentiality of Medical Information Act (CMIA) to treat any business that offers a "health care chatbot" to consumers as a healthcare provider for CMIA purposes. These businesses are therefore subject to CMIA's confidentiality duties and penalties (though only for CMIA purposes) but they are not subject to other regulations governing healthcare providers. AB 1979 takes effect on January 1, 2027.

SB 503: Bias Mitigation Becomes a Compliance Obligation

Also effective January 1, 2027, SB 503 requires developers and deployers of CDS systems (using the same definition as AB 1979) to make reasonable efforts to identify systems with known or reasonably foreseeable risks of "biased impacts" – i.e., diminished access, quality of care or outcomes based on protected characteristics such as race, sex or age – and to take reasonable steps to mitigate them. The law further requires developers to provide deployers with a statement of intended uses and foreseeable risks, along with documentation covering training data and demographic representativeness, performance and bias evaluation, data governance, intended benefits, and recommended use and monitoring.

SB 503 also requires deployers (defined to include healthcare providers) to monitor the CDS systems' effects and implement mitigation measures that address identified or "reasonably foreseeable" problems. Although deployers will likely welcome this new transparency, both developers and deployers will need to enhance their AI governance programs.

SB 947: Human Review of Automated Employment Decisions

Effective July 1, 2027, SB 947 regulates employers' use of automated decision systems – software that scores, classifies or recommends in ways that assist or replace human judgment about employees. Though not healthcare-specific, the law applies to hospitals, health systems and other large healthcare employers. Employers may not use these systems to violate labor or civil rights laws, infer an employee's protected status or retaliate against employees for exercising their rights. Software output alone cannot justify discipline or termination; a human must corroborate the decision with other evidence, such as personnel files, supervisor reviews or work product.

2 Vetoed Bills That Remain Worth Watching

SB 903: Limits on AI in Psychotherapy

SB 903 would have restricted the use of AI in psychotherapy to administrative and supplementary support. Patients would have had to provide informed consent – not a checkbox buried in terms of use – before AI recorded, transcribed or screened them for care. The bill would have barred marketing chatbot-delivered services as therapy or calling a chatbot a "therapist," required licensed professionals to review and approve AI-generated therapeutic decisions, diagnoses, treatment plans, emotion detection and triage, and prohibited direct psychotherapeutic communication by AI unless cleared by the U.S. Food and Drug Administration.

In his veto message, Gov. Newsom said the bill's definitions and scope were so broad that they would harm legitimate clinical tools never meant to replace psychotherapy. Nonetheless, behavioral health innovators should expect the California Legislature to return to this space.

AB 2575: Protections for Clinicians Who Override AI

AB 2575 would have prohibited retaliation or discrimination against workers based on their decision to rely on – or decline to rely on – a CDS system when exercising professional judgment and would have authorized them to file complaints with the labor commissioner. It also would have barred those who develop, modify, select or deploy CDS systems from arguing that a licensed professional's failure to override an output was a superseding cause of harm, limiting a key way in which AI vendors could shift liability to clinicians.

Gov. Newsom vetoed the bill, citing the impracticality of its "solely based on" evidentiary standard and the labor commissioner's lack of medical expertise to judge clinical standards of care.

Key Takeaways

Despite concerns voiced by President Donald Trump against state-by-state AI regulation, California continues to enact legislation governing the use of AI in healthcare. As with many states that have adopted similar legislation, the message from Sacramento is clear: AI can support care, but licensed professionals must maintain oversight.

With most obligations taking effect January 1, 2027, providers, digital health companies, AI developers and investors should act now to comply with these laws. A key step is adopting a comprehensive AI governance plan with policies designed to accomplish the following goals:

• Inventory AI Use Cases. Map where AI is used in clinical decisions, patient-facing chat, workforce management and back-office workflows, and assign a human decision-maker to each.
• Document Professional Judgment. Establish escalation and override protocols so licensed providers demonstrably retain independent judgment under AB 1979.
• Strengthen Vendor Diligence and Contracts. Request SB 503 documentation on training data, bias testing and monitoring, and allocate monitoring responsibilities contractually.
• Review Chatbots. Assess healthcare chatbot data flows under the CMIA and customer service bots under AB 1609's disclosure and human-access rules.
• Audit Human Resources Technology. Before July 1, 2027, ensure automated employment tools are paired with human review under SB 947.

Holland & Knight's Healthcare Team can help providers, digital health companies, AI developers and investors assess exposure, build AI governance programs and negotiate AI vendor agreements under these new laws.

For more information, please contact the authors or another member of the team.

Information contained in this alert is for the general education and knowledge of our readers. It is not designed to be, and should not be used as, the sole source of information when analyzing and resolving a legal problem, and it should not be substituted for legal advice, which relies on a specific factual analysis. Moreover, the laws of each jurisdiction are different and are constantly changing. This information is not intended to create, and receipt of it does not constitute, an attorney-client relationship. If you have specific questions regarding a particular fact situation, we urge you to consult the authors of this publication, your Holland & Knight representative or other competent legal counsel.