AI-generated Research Error Exposes Law Firm to Client Harm
In May 2025, two law firms were sanctioned after submitting a court filing containing artificial intelligence (AI)-generated citations that appeared credible but were entirely fabricated, a phenomenon commonly known as “hallucinations.” The incident arose in the case Lacey v. State Farm Gen. Ins. Co. when the plaintiff’s attorneys filed a supplemental discovery brief challenging whether certain documents held by the insurer were protected from disclosure.
The law firms working together on this matter were Ellis George LLP and K&L Gates LLP. The brief, which was submitted to Special Master Michael Wilner, contained approximately nine inaccurate legal citations out of 27 legal sources. Ellis George LLP later acknowledged using several AI tools to draft the brief, including Google Gemini, Westlaw Precision and CoCounsel. The firm shared the document with colleagues at K&L Gates LLP without disclosing its use of AI, who then incorporated the citations into the final filing without verification. Concerningly, Wilner sent back the brief, flagging two suspicious citations. The attorneys filed a “corrected” version that still contained six AI-generated errors.
Wilner found that the attorneys had “collectively acted in a manner that was tantamount to bad faith.” The flawed brief was struck from the record, and the firms were ordered to disclose the misconduct to their client and pay $31,100 in legal fees.
Why This Case Matters
This case illustrates several key considerations as they pertain to the risk of AI in professional settings, as follows:
- Tool quality is not a defense. Rather than relying on consumer‑grade chatbots or untested products, the attorneys used professional legal AI tools designed and marketed specifically for use by lawyers, including Westlaw Precision and CoCounsel. Despite this, the brief still contained inaccurate and entirely fabricated citations, demonstrating that even tools intended for professional use are not immune to error. While AI can support drafting and research, its output must be carefully reviewed before use.
- The incident harmed multiple parties. While sanctions were imposed on the law firms, the incident also harmed the client. Specifically, Wilner denied the discovery motion, depriving the client of the opportunity to obtain potentially valuable evidence and weakening their position in the case. This highlights that when AI-generated errors go unchecked, the consequences can affect both those who use the tools and those who rely on the output.
- A single AI‑assisted error can trigger multiple risks simultaneously. The law firms’ failure to verify AI‑generated citations led to a range of consequences, including reputational damage, loss of client trust, and potential legal and financial exposure, including the possibility of professional liability claims. This illustrates how a single AI error can trigger multiple risks at once.
- Consequences may extend beyond the immediate outcome. In similar cases, courts have imposed additional measures in relation to AI‑related errors, including sanctions on individual lawyers and, in some instances, referral to the bar. This highlights how AI-related misuse can lead to broader professional scrutiny beyond the initial incident.
- The frequency and plausibility of AI hallucinations are increasing. Wilner noted that the fabricated citations appeared sufficiently credible that they could have been included in a judicial order, describing the possibility as “scary.” Compounding these concerns, documented instances of AI‑generated hallucinations in court filings have grown steadily since 2023, demonstrating that AI-generated errors are not easy to identify without careful review.
Other Notable Cases
The following examples further illustrate how AI-related errors have arisen in legal practice:
- In Mata v. Avianca, Roberto Mata, a passenger, filed a personal injury claim against the airline Avianca after being struck by a serving cart on a flight. When Avianca moved to dismiss the claim, the plaintiff’s lawyers used ChatGPT to prepare an opposition to the motion. However, the brief cited six entirely fabricated cases with fictional judicial quotations. When one of the attorneys later asked ChatGPT whether the cases were real, the chatbot incorrectly confirmed that they were, compounding the initial error. The court imposed a fine and required the attorneys to send letters of apology to their client and each judge whose name had been falsely cited in the brief.
- In Johnson v. Dunn, Frankie Johnson, an incarcerated plaintiff, filed a civil rights claim against Jefferson Dunn, the former commissioner of the Alabama Department of Corrections. During the proceedings, plaintiff’s counsel flagged citations in two defense motions as appearing to be fabricated, and the court independently confirmed the errors. One of the defense attorneys later admitted to having used ChatGPT to obtain citations without verification, despite his firm having clear policies in place warning against unsupervised AI use. The court removed the attorneys from the case, formally criticized their conduct and referred them to the state bar. They were also required to disclose the sanctions order to clients and opposing counsel.
- In Boston v. Williams, four women, including Selena Boston, filed a civil lawsuit against comedian Katt Williams following an alleged altercation outside a nightclub. When the defendant applied for summary judgment to dismiss the case, the plaintiff’s lawyer filed a response in opposition. However, the brief was prepared by the lawyer’s daughter using AI and was not properly reviewed before submission. It contained significant citation errors: 17 of 24 citations were found to be inaccurate or nonexistent, many of which were consistent with AI-generated hallucinations. The court required the lawyer to disclose the misconduct to her clients, and, for the next five years, to the court in any current or future cases in the same jurisdiction. She was also referred to the state bar.
Risk Management Considerations
Organizations should consider the following risk management measures to reduce their exposure to AI-related claims:
- Verify every AI-generated citation. Employees should review AI outputs to ensure citations are accurate and relevant to the context and jurisdiction, always verifying them against authoritative sources. They should also remain mindful of AI’s limitations, as outputs that sound plausible are not always accurate or reliable.
- Implement a tiered AI use policy. Organizations should establish clear policies governing AI use, distinguishing between acceptable uses (e.g., internal summaries), restricted uses (e.g., research requiring verification) and prohibited uses (e.g., inputting sensitive or confidential information). These should be clearly communicated across the organization and supported by training on ethical use, including an understanding of the tools’ risks.
- Disclose AI use to stakeholders. Organizations should clearly communicate their use of AI, including updating relevant agreements or communications to reflect how such tools support service delivery. They should obtain informed consent where appropriate and recognize that some stakeholders (e.g., those in highly regulated industries) may restrict or prohibit AI use. Transparent disclosure can help manage expectations and reduce the risk of professional liability claims.
- Establish formal AI governance structures. Organizations should implement formal governance frameworks to oversee AI use, including creating a cross-functional oversight team comprising legal, compliance and IT representatives. Governance should extend beyond written policies to include active monitoring of how AI tools are used in practice, robust workforce training, practical controls (e.g., approval requirements for certain uses and restrictions on uploading sensitive data), and clear documentation of compliance and enforcement.
- Train employees on how AI can fail. Organizations should provide training on the limitations of AI tools, including the risk of hallucinations, which can appear credible and be difficult to detect. Training should emphasize why such errors can survive superficial review and reinforce the importance of verifying outputs, as employees remain responsible for the accuracy of their work.
Professional Liability Insurance and AI
Professional liability insurance, also known as errors and omissions (E&O) insurance, typically covers claims alleging negligence, errors, or omissions in professional services. However, many pre‑2023 policies do not expressly address AI‑related risks, creating uncertainty as to the extent of coverage for AI‑assisted work. Insurers are now increasingly updating policy language, which may introduce exclusions or other limitations on claims arising from the use or failure of AI tools. Even where coverage applies, it typically does not extend to court‑ordered sanctions and fines, which exceeded $30,000 in the case of Ellis George LLP and K&L Gates LLP. Coverage may also be capped, with lower sublimits applying to AI‑related claims in some cases. It may be further limited by policy language requiring services to be performed only by natural persons.
Organizations can reduce the likelihood of uncovered losses by carefully reviewing their current policy language, including any sublimits and exclusions. Where gaps exist, firms should work with an insurance broker to evaluate the merits of endorsements or standalone AI coverage. Organizations should also review their governance and risk management practices related to AI, as deficiencies in these areas may affect the availability of coverage.
We Can Help
While AI can help law firms drive efficiencies, its use in professional settings introduces significant risk. Increasing instances of AI generated hallucinations in court filings reflect a broader pattern of AI related risk across sectors, prompting insurers to reassess coverage and, in some cases, introduce exclusions or limitations. Organizations should carefully review policy language to ensure that coverage aligns with their use of AI tools.
Contact us today for more risk management and coverage solutions.