AI Policy Wiki
Dashboard
⚲
AI Policy Wiki › Concepts & Frameworks

AI-Fabricated Citations

high confidence · updated 2026-09-19

References, quotations, witnesses and authorities that a generative model invents and that pass unverified into a document of record — court filings, administrative decisions, peer-reviewed papers, student work — together with the professional-responsibility, judicial and publishing responses to the pattern.

Contents

An AI-fabricated citation is a reference, quotation, named witness or legal authority that a generative model produces and that does not exist, or that exists but does not say what it is cited for. The pattern is distinct from model hallucination in general because the failure is completed by a human: the output reaches a document of record — a court filing, an administrative decision, a peer-reviewed paper, a student submission — because nobody verified it. The model-side mechanism is treated at Sycophancy and Hallucination.

The two defect types

Accounts of individual episodes tend to collapse two different failures that the records distinguish.

The first is an authority or source that does not exist — an invented case, a reference to a paper never written, a named witness who is not in the record. The second is a real source misrepresented: a real case cited for a proposition it does not support, or words attributed to a person who exists but did not say them. The New Mexico Supreme Court's contempt order in In re Stephen D. Aarons separates them on the face of its findings, recording four wholly fabricated witnesses alongside false testimony attributed to three people who do appear in the record, and separately recording two real New Mexico cases whose holdings were misrepresented (In re Stephen D. Aarons — Dispositional Order of Direct Contempt (N.M. Sup. Ct., 2026)).

Incidence

A running compilation of court filings flagged for AI hallucinations had recorded just over 1,000 filings as of July 2026, a figure commentators set against roughly 65 million new US cases filed annually (Source: hardcoresoftware.learningbyshipping.com).

The measured rate is higher and better quantified in biomedical publishing. An audit of approximately 2.5 million biomedical papers published between January 1, 2023 and February 18, 2026 in PubMed Central's open-access corpus verified 97.1 million references and identified 4,046 confirmed fabricated citations across 2,810 papers. The rate rose from 1 in 2,828 papers in 2023 to 1 in 458 in 2025 to 1 in 277 in the first seven weeks of 2026 — a twelve-fold increase over three years, or roughly four per 10,000 papers in 2023 against 51.3 in the fourth quarter of 2025 and 56.9 in early 2026. At the time of the audit 98.4% of affected papers had not been retracted. The authors dated the inflection to mid-2024, when models capable of generating realistic-looking references came into wider use, and argued that "each reference implicitly asserts that a verifiable source exists and supports the claims being made," so that when references point to non-existent studies "readers, reviewers, and policy makers are unable to evaluate the evidence" (Source: nature.com; retractionwatch.com; see AI Content Saturation (\"AI Slop\") for the full audit record).

Where the fabrications land

Advocates' filings. The largest recorded category, and the one the filings compilation measures.

Adjudicators' own decisions. On September 2, 2026 the Supreme Court of India set aside a ₹425.27-crore customs penalty order in Vijay Ghanshyam Gadiya v. Union of India after finding that the order, issued by the Additional Commissioner of Customs in Surat, relied on non-existent case law, fake citations and AI-hallucinated legal ratios (Source: sci.gov.in). The distinction from advocates' filings is that the fabricated authorities sat inside the state's own reasoning imposing a penalty, rather than in material the state was asked to evaluate.

Published research. The biomedical audit above, and arXiv's response to submissions carrying invented references.

Products whose value proposition is citation. Perplexity has produced fabricated citations in multiple instances, citing sources that did not support the claim or fabricating attribution — the same failure arising in a system built to supply sources.

Student and professional training work. Treated at AI in Education.

Institutional responses

Professional-responsibility standards

Bar ethics guidance addresses the pattern through duties lawyers already owe. Formal Opinion 2024-004 of the State Bar of New Mexico's Ethics Advisory Committee, issued September 24, 2024, states that the use of generative AI "does not change their fundamental duties under the Rules of Professional Conduct," and construes Rule 16-303 NMRA (candor toward the tribunal) and Rule 16-804(C) NMRA (dishonesty, fraud, deceit or misrepresentation) to require that a lawyer using generative AI to draft a court filing "independently verify the accuracy and sufficiency of all research, citations, and analysis conducted by the AI tool." The lawyer may not rely on the same tool to check its own work, because such tools "can be misled by their own prior hallucination," and must correct errors the tool makes; a failure to do so is described as "a knowing misrepresentation of the law." Following a court's own rules on AI use and disclosure is placed within the duty of candor. The opinion gives Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), as its example, and states that it is advisory only and binds neither lawyers, the disciplinary board nor any tribunal (State Bar of New Mexico Formal Ethics Advisory Opinion 2024-004 — Using Generative Artificial Intelligence in the Practice of Law).

The opinion's verification language is framed around research, citations and analysis, and its example concerns sources that did not exist. It does not separately address AI-generated statements of the factual record, the second defect type described above, although the text of Rule 16-303(A)(1) it quotes reaches false statements "of fact or law."

A federal court in New Mexico has drawn on such opinions in upholding sanctions. In Dehghani v. Castro (D.N.M. May 9, 2025), the district court affirmed Rule 11 sanctions against an attorney whose brief, drafted by a contract attorney, cited six cases that did not exist and thirteen that did not support the propositions stated; the sanctions included a $1,500 fine, a one-hour CLE program on legal ethics in writing or the use of AI in writing, and self-reports to the New Mexico and Texas state bar disciplinary boards. The court quoted the New Mexico opinion and Opinion No. 705 (February 2025) of the Professional Ethics Committee of the State Bar of Texas as "instructive," while noting that "it is not certain that AI was in fact used here" (Source: caselaw.findlaw.com).

Sanctions

The New Mexico Supreme Court's order in State of New Mexico v. Oscar Renee Sandoval; In re Stephen D. Aarons, No. S-1-SC-40845, is among the heavier sanctions recorded. The Court held the attorney in direct contempt, ordered a $5,000 payment to the State Bar of New Mexico Client Protection Fund within thirty days, referred the matter to the Disciplinary Board, and barred the attorney from appearing before the Court pending the Board's outcome; all briefing in the underlying murder appeal was stricken and the public defender appointed (In re Stephen D. Aarons — Dispositional Order of Direct Contempt (N.M. Sup. Ct., 2026)). The Court grounded the contempt finding in Rule 12-312(D) NMRA — failure to comply with the briefing requirements of Rule 12-318 NMRA being grounds for sanctions — and in its inherent authority of superintending control under Article VI, Section 3 of the New Mexico Constitution.

Two findings in that order locate the aggravating conduct away from the fabrication itself. The attorney admitted he did not verify the factual claims or legal authority before signing and filing, and did not tell his client either of that failure or that show-cause proceedings were under way; the Court's conclusion that he "demonstrated a lack of remorse and a lack of concern for his client" rests on those admissions rather than on the use of the model. The order is nonprecedential by its own election under Rule 12-405(B) NMRA, and it does not cite the State Bar's Formal Opinion 2024-004 or any Rule of Professional Conduct.

Judicial rule-making

Article 19 of China's Supreme People's Court Opinions on Lawfully Adjudicating AI-Related Dispute Cases, issued September 7, 2026, requires litigation participants who use AI to prepare court filings or case-search reports to verify the authenticity and accuracy of the law, judicial interpretations and cases cited before submission, to disclose the AI assistance when filing, and to bear responsibility for the accuracy of the content. The same article directs courts to dismiss claims, and to impose fines or detention, where a party fabricates case facts using evidence obtained by deleting or altering generated-content labels, by specific instruction input, selective presentation of results or adversarial interference (Supreme People's Court Opinions on Lawfully Adjudicating AI-Related Dispute Cases (Fa Fa [2026] No. 10); China — SPC Opinions on Lawfully Adjudicating AI-Related Dispute Cases (法发〔2026〕10号)). The combination of a verification duty, a disclosure duty and an allocation of responsibility to the filer is the most complete rule of its kind recorded.

Evidentiary presumptions

UC Berkeley School of Law's Summer 2026 AI policy replaces AI-detection tooling with per-se evidentiary presumptions keyed to specific model failure modes, so that a hallucinated citation raises a presumption of prohibited AI use. The approach anchors enforcement on observable artifacts rather than on statistical detection of machine authorship.

Publishing gatekeeping

arXiv tightened its endorsement rules on January 21, 2026: new researchers must obtain a personal endorsement from a previously approved researcher, and an academic email address is "no longer a sufficient credential for determining minimum research competence." Thomas G. Dietterich, vice chair of the editorial advisory council, announced that authors submitting AI-generated fake citations would be banned for a year (Source: blog.arxiv.org).

Positions and framing

Steven Sinofsky has argued that institutional device bans in law schools repeat Harvard Law School's 1982 ban on portable computers in exams, characterizing them as a recurring response to new technology rather than a durable policy, and uses the hallucinated-filings compilation to argue the rate is small: "just over 1,000 cases" against a denominator of "about 65,000,000 new cases filed every year," comparing AI-fabricated citations to attorneys' long-standing practice of "stretching the facts in briefs" (Source: hardcoresoftware.learningbyshipping.com).

Jessica Silbey and Woodrow Hartzog's treatment of AI "slop" connects fabricated citations to authorship questions litigated elsewhere, drawing a distinction between slop and "meaningful human authorship" and linking the disclosure obligations courts have imposed after AI-fabricated citations to the standard for human authorship in copyright (AI Slop (Silbey and Hartzog, 2025)).

Open questions

  • Whether the recorded rate of fabricated citations in court filings is rising, falling or flat. The filings compilation is a count rather than a rate, and no denominator of AI-assisted filings is published.
  • Whether verification duties of the kind Article 19 imposes change filer behaviour, and how compliance would be observed. No enforcement record exists yet.
  • Whether sanctions are converging on a scale. The recorded penalties range from reprimand to contempt with a court-appearance bar, and no jurisdiction has published a tariff.
  • Whether retraction practice in biomedical publishing will reach the 98.4% of audited papers with fabricated references that remained unretracted at the time of the audit.
  • Whether the Disciplinary Board proceedings to which the Aarons matter was referred will apply Rules 16-303 and 16-804(C) NMRA as construed in Formal Opinion 2024-004, which the contempt order does not cite, and whether that opinion's verification duty is read to reach AI-generated statements of the factual record as well as citations.

Relationships

Categories: Concepts & Frameworks | global | technical | legal | ethics | safety | academic
What links here (13)