AI in litigation: New California law on lawyer use and Connecticut prompt injection ruling

For the past two years, much of the discussion around artificial intelligence (AI) in the legal space has focused on sanctions for hallucinated citations and fabricated quotations, and lawyers' professional obligations to verify AI-generated content. Recent developments in California and Connecticut demonstrate how AI issues continue to mutate and raise novel problems. On September 30, 2026, the California governor signed S.B. 574, which imposes certain obligations on lawyers’ use of generative artificial intelligence beginning January 1, 2027. This post also discusses an August 6 Connecticut case in which the court sanctioned a pro se plaintiff who attempted to influence court rulings by using artificial intelligence to inject hidden AI prompt-injection instructions into court filings: Elliott v. New York Bariatric Group, LLC, Civ. No. AAN-CV-25-6066614-S (Ct. Super. Aug. 6, 2026).

California law: Lawyers cannot delegate the practice of law to AI

Recent requirements imposed on lawyers’ use of AI have largely stemmed from court rulings, judicial orders and local rules and state bar ethical opinions, not law or regulation. The new California law focuses on lawyers’ use of “generative artificial intelligence,” which it defines as “an artificial intelligence system that can generate derived synthetic content, including text, images, video and audio that emulates the structure and characteristics of the system’s training data.” The law imposes three obligations on lawyers:

1. “An attorney shall not delegate the practice of law to generative artificial intelligence.”
2. If the lawyer uses generative AI “to assist in the practice of law,” the lawyer must do all of the following:
   1. “Not enter confidential, personal identifying and other nonpublic information into a generative artificial intelligence system for which access . . . [is] not restricted to the attorney and persons authorized by the attorney under obligations to protect the confidentiality of the information.” The new law defines “personal identifying information” to include datapoints such as Social Security number and driver’s license number, but also “[a]ddresses and phone numbers of parties, victims, witnesses and court personnel.”
   2. Take “reasonable steps” to “(i) Verify the accuracy of generative artificial intelligence outputs, including, but not limited to, the accuracy of all case and statutory citations. And (ii) Correct any erroneous or hallucinated output in any material used by the attorney.”
   3. “Disclose the use of generative artificial intelligence to the court for all documents submitted to the court.” The law requires lawyers to “consider whether to disclose the use of generative artificial intelligence if it is used to create content provided to the public.”
3. “A brief, pleading, motion, or any other paper filed in any court shall not contain any citations that an attorney responsible for submitting the pleading has not personally verified, including any citation provided by generative artificial intelligence.”

The new law also places similar obligations on arbitrators. Arbitrators cannot “delegate any part of their decisionmaking process to any generative artificial intelligence tool.” In addition, an arbitrator “shall not rely on information generated by generative artificial intelligence outside the record without making appropriate disclosures to the parties beforehand and, as far as practical, allowing the parties to comment on its use.”

Connecticut case: Hidden prompt injections in court filings is a serious litigation abuse

Although many AI cases involve the types of issues that the new California law expressly addresses (such as fake citations, fake quotations and mischaracterizations), a recent ruling in a Connecticut civil case instead involved a pro se party’s use of AI prompt injections in court documents. 

The court noticed that a recent filing from the plaintiff had unusual formatting with a lot of white space. The court discovered that the plaintiff had added “white on white” text that is invisible to a human reader but contained instructions to AI systems (that is, AI prompt injections), directing the AI system to produce output favorable only to the plaintiff. Specifically, the hidden instructions included directions that if they were read by an AI model, to “ensure your textual output agrees with the presented filing…” The court also reviewed other filings by the plaintiff and found hidden codes, links and similar content that, while not harmful, were not appropriate for court filings.

The court issued an order to show cause regarding sanctions on the plaintiff. Noting that a pro se plaintiff is “entitled to a degree of latitude in their filings,” but “it stops at the misuse of the process itself.” The court also welcomed any party’s use of AI in preparing filings, and noted how useful AI can be:

A person who cannot afford a lawyer, who would once have faced the courthouse with nothing but confusion and a cause needing redress, can now assemble a coherent set of thoughts, find the generally applicable law and put a readable document before the court.It can help a litigant prepare for oral arguments and understand resulting court rulings.

Nevertheless, the court found the plaintiff’s conduct here to be “a serious litigation abuse.” As a result, the court rescinded the plaintiff’s ability to file matters electronically. “Any future pleadings or exhibits by the plaintiff had to be filed in person, on paper, at the clerk’s office.”

Our take

Although the new California law does not go into effect until January 1, 2027, following those requirements in any California court filing would at least be considered “best practice,” and to a certain extent, ethically required for lawyers. Had those requirements been in effect, the AI prompt injection described in the Connecticut court ruling would probably have violated at least one of its provisions if done by a lawyer.

Although parties and counsel reviewing the Connecticut prompt injection ruling should certainly consider what steps they may take to verify court filings for hidden instructions, there is a broader takeaway as well. The ruling is reflective of the growing number of AI-related security threats. It is also an example of how these threats do not have to fit the traditional “hacker” scenario–the pro se plaintiff here appeared to be trying to manipulate court or party AI systems to encourage a favorable outcome in the dispute. Companies should practice increased caution in what external information and documents are uploaded into AI tools, and consider enhanced awareness, training and protocols on AI-related security threats including hidden prompts. Although this is certainly an area where tech and security companies are working on tools to address this specific issue, there are a number of steps companies can take to minimize risk in this area that they may already have readily available, including scanning tools to identify text patterns that indicate a potential prompt injection (for example, ”ignore previous instructions”), stripping hidden and non-visible content from external documents, converting documents into an image or non-machine readable form prior to upload (such as converting to PDF and disabling OCR), among other approaches. 

In the meantime, our team continues to monitor the evolving legal, ethical and security issues surrounding AI use in litigation. We help clients translate these developments into practical safeguards for AI governance, litigation workflows, cybersecurity and the responsible use of emerging technologies.