California Attorney General Issues New Legal Advisories on Artificial Intelligence
Highlights
- California Attorney General Rob Bonta recently issued two new Legal Advisories that offer further clarification on the application of California law(s) to businesses – including those in the healthcare industry – that develop, sell or use artificial intelligence (AI).
- The first advisory applies to the use of AI generally and reminds manufacturers and developers of AI systems that they and their systems must comply with California laws that protect consumers from various unfair business practices, and the other applies specifically to healthcare entities that use AI.
- This Holland & Knight alert focuses primarily on the healthcare advisory and how healthcare entities must ensure compliance with the relevant California laws when using AI.
From the West Coast Healthcare Desk
California Attorney General (AG) Rob Bonta issued two new Legal Advisories on Jan. 13, 2025, to provide further clarity on the application of California law(s) to businesses, including those in the healthcare industry, that develop, sell or use artificial intelligence (AI).
The first advisory applies to the use of AI generally, reminding manufacturers and developers of AI systems that they and their systems must comply with California laws concerning unfair business practices, false advertising, discrimination, bias and abuse of data.1 Furthermore, in line with growing public concern regarding health entities' use of patient data, the advisory also reinforces the state's constitutional right to privacy afforded to consumers, such as the right to know whether a business collects personal information and how that information is being used.2
The second advisory is titled "Legal Advisory on the Application of Existing California Law to Artificial Intelligence in Healthcare" (HC Advisory). As the title suggests, the HC Advisory applies specifically to healthcare entities that use AI, including healthcare providers, manufacturers, insurers and all other companies that interact with the healthcare system.3 Though acknowledging the many benefits that AI can have in improving patient and population health, the AG states that misuse of AI can result in violations of California consumer protections laws, as well as California healthcare laws – including restrictions on the corporate practice of medicine (CPOM). This Holland & Knight alert focuses on the HC Advisory.
AI in the Healthcare Sector
As noted above, the HC Advisory focuses specifically on the use of AI in the healthcare sector. The takeaway from the HC Advisory is clear: The AG may take enforcement action against creators, marketers and users of AI systems if those AI systems result in a violation of California law. This includes California's Unfair Competition Law, state antidiscrimination laws and laws governing the practice of medicine.
The HC Advisory provides examples of where the creation, marketing and use of an AI system could potentially violate the Unfair Competition Law. As background, California's Unfair Competition Law prohibits unlawful, unfair and fraudulent business practices, as well false advertising.4 In the HC Advisory, the AG takes the position that it may be unlawful under the Unfair Competition Law to 1) falsely advertise the accuracy or utility of AI systems, or 2) create, market or disseminate an AI system that does not comply with federal or state laws, including civil rights and privacy laws.
Separately, the AG contends that the creation and/or use of an AI system that adversely impacts a protected classes of individuals (e.g., race, sex, color, religion age, disability) is violative of California antidiscrimination laws, including Government Code Section 11135, which prohibits discrimination by entities and individuals who receive state support (e.g., Medi-Cal funding). As an example, the AG points to instances of what is referred to as "algorithmic bias" or "algorithmic discrimination," which occurs when the use of AI systems, software or algorithm disproportionately results in the denial of services to protected classes of individuals.
Significantly, the AG asserts that the deployment of AI by medical providers must comply with laws regarding CPOM. According to the AG, California law mandates that only human medical professionals are allowed to practice medicine and that the delegation of the practice of medicine to AI could potentially constitute a violation of California's laws prohibiting CPOM. The AG specifically states that "using AI or other automated decision tools to make decisions about patients' medical treatment, or to override licensed care providers' determinations about what a patient's medical needs are, may violate California's ban on the practice of medicine by corporations and other 'artificial legal entities.'" The import of this guidance is that California physicians may use AI to assist in their medical practice but must make the ultimate decision regarding patient care. Additionally, an entity that attempts to override a physician's plan of care by referring to AI-generated recommendations runs the risk of violating California laws on CPOM, which also constitute violations of the Unfair Competition Law.
Other notable positions taken in the HC Advisory are as follows:
- As described in a previous Holland & Knight alert, health insurance plans cannot use AI to deny, delay or modify healthcare services based in whole or in part on medical necessity (e.g., through the prior authorization process).5
- Providers must consider whether to include the use of AI in their required informed consent forms. The AG did not take a position on this but flagged it to providers to determine.
- Providers and businesses that provide electronic medical records systems must ensure that the use of AI (or AI functionality) is consistent with the requirements for the California Confidentiality of Medical Information Act and limit access and improper use of sensitive information, including reproductive and sexual health information.
Conclusion
Overall, the advisories issued by the AG, including the HC Advisory, serve as a good reminder for healthcare entities that they must ensure compliance with the relevant California laws when implementing and utilizing AI. In addition to the laws discussed above, various other tort, environmental, business and public health laws should also be considered when implementing AI systems. Healthcare providers and companies must make certain that new and emerging technologies fully comply with existing laws and regulations. Companies should engage legal counsel with extensive knowledge in navigating the complex California laws that apply to AI tools that are intended for use in healthcare.
Notes
1 California Legal Advisory: Artificial Intelligence Holds Great Potential and Risks; Ca. Bus. & Prof. Code, § 17200 et seq.
2 Ca. Civ. Code, § 1798.140(v).
3 California Legal Advisory: Artificial Intelligence in the Healthcare Sector.
4 Cal. B&P Code, § 17200 et seq.
5 Health & Saf. Code, § 1367.01, subd. (k)(1); Ins. Code, § 10123.135, subd. (j)(2).
Information contained in this alert is for the general education and knowledge of our readers. It is not designed to be, and should not be used as, the sole source of information when analyzing and resolving a legal problem, and it should not be substituted for legal advice, which relies on a specific factual analysis. Moreover, the laws of each jurisdiction are different and are constantly changing. This information is not intended to create, and receipt of it does not constitute, an attorney-client relationship. If you have specific questions regarding a particular fact situation, we urge you to consult the authors of this publication, your Holland & Knight representative or other competent legal counsel.