The following article was originally published by the American Health Law Association.

A Stanford professor who served as an expert witness saw his court testimony dismissed after AI-generated citations in his filing turned out to be fabrications. Legal proceedings now face a cautionary example because this case questions Minnesota’s prohibition on AI-created election deepfakes. 

The Hancock Case: When AI Undermines Expert Credibility 

This legal case examines Minnesota’s prohibition of electoral influence through AI-generated, deepfake technology. The state called upon Jeffrey T. Hancock from Stanford University, who specializes in communication and artificial intelligence, to demonstrate that deepfakes represent a real danger to democratic systems. Hancock revealed that he relied on ChatGPT-4 to assist in creating his expert filing [1]. 

What happened next was revealing: The AI system created numerous, completely invented citations that appeared legitimate but pointed to sources that didn’t exist. According to Forbes reporter Lars Daniel, attorneys for Minnesota State Representative Mary Franson and YouTuber Christopher Kohls, who contested the deepfake law on First Amendment grounds, found bogus citations and alerted the judge. The judge responded to the false citations by removing Hancock’s testimony from the record because he considered Hancock’s credibility destroyed.  

Precedent: The Mata v. Avianca Case 

Mata v. Avianca, Inc. (2023) 

Case Overview: Roberto Mata filed a lawsuit against Avianca Airlines, claiming injuries caused by a serving cart during a flight. The case was removed to federal court under the Montreal Convention, which governs international air travel disputes. Avianca filed a motion to dismiss, arguing that Mata’s claims were time-barred [2]. 

Key Issue: The plaintiff’s attorney, Steven Schwartz, used ChatGPT to draft the opposition to the motion to dismiss. The filing included citations to several judicial decisions that were later revealed to be entirely fabricated by ChatGPT. 

Court’s Decision: 

  • Judge Castel imposed sanctions on Schwartz and his law firm for submitting non-existent cases in court filings and failing to verify their validity. 
  • The court emphasized that attorneys have a gatekeeping responsibility under Rule 11 of the Federal Rules of Civil Procedure to ensure the accuracy of their submissions. 
  • The sanctions included a $5,000 penalty and mandatory notifications to all judges falsely cited in the fabricated opinions. 
  • The court underscored the broader implications of relying on AI tools without proper verification, highlighting the erosion of trust in legal proceedings. 

The Nuanced Problem of AI-Generated Citations 

AI-generated citation problems affect more than completely invented references. Some instances of AI citation issues present more complex and concerning challenges. 

Misattributed Real Research 

AI systems can accurately reference valid research but make mistakes attributing these findings to incorrect authors or publication dates. For example, an AI system might incorrectly attribute research data to “Williams et al..” (2022) when these findings were originally published by “Thompson et al..” (2020). The misattribution proves particularly hard to identify because the referenced content is real but located elsewhere than what the AI states. 

The Hancock case court documents exposed this precise issue. During cross-examination, Hancock acknowledged that his citations included accurate research findings but assigned them to the wrong researchers or journals, leading to a confusing blend of genuine information with mistaken sourcing.

Inaccessible Valid Sources 

A major problem is citing legitimate sources that remain inaccessible to researchers: 

  • Paywalled academic journals: Most academic papers require payment or academic institution subscriptions to access, which creates verification difficulties for those who do not have access. 
  • Proprietary databases: References to law cases or statutes are often found in proprietary databases such as LexisNexis or Westlaw, which require paid subscriptions for verification. 
  • Conference proceedings: Research that appears at academic conferences can be authentic but typically lacks broad online indexing or availability. 

In his analysis of the Minnesota case, Luis Rijo observed that “many of the references Hancock mentioned were articles from specialized communication journals which needed institutional subscriptions for readership access.” Although some attorneys first suspected these sources as fabricated publications, further examination showed they were authentic articles that could not be verified without access to specific databases. 

This creates a dangerous gray area: Refusing a citation because it lacks immediate internet accessibility could invalidate authentic evidence. This situation allows completely fabricated references to hide behind assertions of “limited accessibility.” 

AI Hallucinations: The Mechanisms Behind AI Hallucinations & Their Importance in Legal Proceedings 

I’ve spent years working as a testifying and consulting expert in statistics and data science, and I’ve seen both sides of this coin. AI tools have revolutionized how we analyze evidence, but they have also introduced problems I never had to worry about with traditional methods. These AI hallucinations aren’t just academic concerns—they’re affecting the outcomes of real cases and impacting people’s lives. 

The way these hallucinations happen isn’t complicated once you break it down. These large language models are essentially prediction machines trying to guess what text should come next based on patterns they’ve seen before. They’re not trying to be truthful—they’re trying to sound convincing. See the problem? 

When ChatGPT or similar systems generate responses, they prioritize what sounds plausible over what’s true, like a smooth-talking friend who can make anything sound believable. The text flows well and feels internally consistent, even when it’s entirely made up. This becomes a nightmare in legal settings where you need rock-solid citations, not just plausible-sounding ones. 

What makes this worse is that there’s no built-in fact-checker. When I’m researching something for court, I’m constantly cross-referencing, verifying, and questioning my sources. These AI systems don’t automatically check themselves against LexisNexis, Westlaw, or other authoritative sources. They’re working in their own bubble based on training data, with no ability to fact-check against the real world unless they’ve been specifically programmed to do so. 

Researchers call another problem the “echo chamber effect,” a fancy way of saying AI systems can reinforce patterns they’ve seen in their training data, including mistakes or unverified claims that appeared frequently. Since they can’t distinguish between facts and speculative fluff, they present both with the same confidence level. It’s like citing a peer-reviewed journal and a random blog  post with equal authority. 

There is also the issue of the knowledge cutoff. Most AI systems stop learning at a certain date, meaning they’re biased toward older information that might be outdated by now. In rapidly evolving areas of law or science, this limitation can lead to references that miss recent developments or precedents. 

When these problems show up in legal settings, they create several headaches: 

  • False citations: This is the most obvious problem—completely invented cases, statutes, or academic papers that don’t exist. Unless you’re familiar with that specific domain, these fabrications look identical to real sources. 
  • Misattributed citations: Even trickier are instances where the AI correctly describes real research or legal principles but attributes them to the wrong source. This is particularly deceptive because the content seems solid when you check it, but the attribution is wrong. 
  • Unverifiable citations: Then there are references to potentially legitimate sources behind paywalls or in specialized databases. This ambiguity can be exploited to give fake credibility to arguments while making them hard to verify. 

The worst part is how this damages credibility. Once a judge spots AI-generated errors in an expert’s testimony, they naturally question everything else that expert has said. It’s like finding one cockroach in a restaurant: You assume the whole kitchen is contaminated. This credibility damage extends beyond the specific errors to affect all claims, even the thoroughly verified ones. 

The Hancock case is a perfect example. Here’s a respected Stanford professor with genuine expertise, and the court threw out his entire testimony because of AI-generated citations. His insights about deepfake threats might have been valuable, but we’ll never know because AI undermined his credibility. The irony that an AI expert was undone by AI-generated misinformation drives home how widespread and subtle this problem is. 

Standards for Responsible Application of Artificial Intelligence in Legal Environments 

In the Minnesota case Judge Provinzino presided over, he advised future attorneys to ask witnesses whether AI helped draft their declarations and to confirm any content produced by AI. The court highlighted that, although AI offers valuable advantages for legal practice, lawyers must still apply independent judgment and critical thinking skills instead of unthinkingly relying on AI systems’ information. 

This case, together with new best practice standards, requires experts and attorneys to: 

  1. Verify all AI claims: Every AI-generated claim should be independently validated, as each citation, forensic analysis, and data point requires confirmation before being submitted to the court, ensuring both its existence and correct attribution. 
  2. Maintain database access: Expert witnesses and law firms must subscribe to relevant academic and legal research databases to verify references properly. 
  3. Document verification efforts: When using paywalled or hard-to-access sources as citations, document how you verified these materials and consider appending excerpts or screenshots to your report. 
  4. Educate on AI limitations: Legal professionals, including judges and attorneys, require training in AI literacy to understand possible errors in data existence and source attribution. 
  5. Require transparency: The judiciary must enforce requirements for clear disclosures about how AI has been applied in legal arguments and expert testimony. 

Courts Must Be Wary of AI “Assistance” 

Let’s be honest—this case should be a wake-up call for everyone involved in the legal system. As someone who regularly testifies in court, I find these developments deeply troubling. AI can be incredibly useful, but we can’t blindly trust what it spits out, especially in legal proceedings where the stakes are so high. 

The bottom line is that we need to be more careful. Whether you’re a lawyer, an expert witness, or a judge, you need to recognize that AI-generated information—whether it’s entirely made up, misattributed, or just hard to verify—poses real risks to the integrity of our legal system. 

As AI becomes more common in courtrooms and legal research, we need to focus on the fundamentals: fact-checking, verification, and transparency. These aren’t just abstract principles; they’re the foundation of reliable forensic work and fair legal proceedings. 

Speaking as someone who regularly works as a testifying and consulting expert, my advice is simple: Use AI as a tool, not a replacement for your judgment. When I’m working with data analysis tools, I never just accept the output without checking it. The same should go for AI: It can help us work more efficiently, but we still need to apply the same critical thinking and verification methods we’ve always used. 

Before checking the reference list below, try to guess which citations in this article might be fabricated. Some references have deliberately been fabricated for demonstration purposes to illustrate the very problem this article discusses. Try to determine which ones are legitimate and which are not. 

References 

Legitimate References 

  1. Daniel, L. (2025). The Irony—AI Expert’s Testimony Collapses Over Fake AI Citations. Forbes. https://www.forbes.com/sites/larsdaniel/2025/01/29/the-irony-ai-experts-testimony–collapses-over-fake-ai-citations/ 
  2. Rijo, L. (2025). Minnesota court rejects expert testimony tainted by AI-generated citations. PPC Land. https://ppc.land/minnesota-court-rejects-expert-testimony-tainted-by-ai-generated-citations/ 
  3. State of Minnesota v. Kohls, 609.771 (Minnesota District Court, 2025). 

Fabricated References (For Demonstration Purposes) 

  1. Richards, J. (2024). AI in the Courtroom: When the Tool Becomes the Trap. Journal of Digital Evidence and Legal Technology, 18(2), 112-128. 
  2. Morenstein, A. & Thompson, P. (2023). Artificial Intelligence and Expert Witness Testimony: Guidelines for Preventing Hallucinated References. Technology Law Review, 45(3), 289-311. 
  3. West, J. (2024). Generative AI in Legal Settings: Empirical Evidence and Practical Guidelines. Journal of Law and Artificial Intelligence, 5(1), 23-47. 
  4. Hancock, J. T., & Williams, S. (2023). Deepfakes in Political Discourse: Measuring Public Perception and Electoral Impact. Communication Research Quarterly, 42(4), 315-334.