Blog · Data custody

When AI makes up sources: what the court cases teach everyone

· 2 min read

Courts have now dealt with more than 1,400 cases in which AI errors, most often invented case citations, ended up in legal filings. That figure comes from a tracker maintained by Damien Charlotin at HEC Paris, reported by Scientific American in May. Lawyers are the most visible example because courts check citations. The same thing happens, unchecked, in reports, board papers and grant applications everywhere.

What has been happening in court

The tracker was adding "around 350, 400 decisions a quarter", according to Scientific American. Penalties have grown with it. Norton Rose Fulbright's 2026 update describes courts applying the sanctions rules they already have. As the Fifth Circuit put it, "the same sanctions rules apply".

Some cases are striking. Scientific American describes an attorney sanctioned in April by the Alabama Supreme Court who, after being warned about one false citation, cited a nonexistent case again at the end of the very next sentence. Researchers quoted in the article call the pattern "cognitive surrender": people deferring their thinking to a system that sounds confident.

Why it keeps happening

A language model is built to produce fluent text. Given a question, it produces something that looks like a good answer, and a citation looks like part of a good answer. Unless the system is built to answer only from sources it can actually point to, it can produce a reference that looks right and does not exist.

Warnings alone have not stopped it, because the pressures are constant: deadlines, volume, and a tool that is usually right. The fix has to be in how the system works, not only in how carefully people use it.

What a trustworthy answer looks like

The general principles are in why AI answers need sources. In short:

  • Every claim points to a stored source that a person can open.
  • The source is the real record, kept with its receipt: who published it, when it was fetched and whether it has changed. See data provenance.
  • No source, no claim. If nothing in the evidence supports a statement, the system says so instead of filling the gap.
  • A person decides what goes out under their name.

What Prism Labs is doing

This is the rule at the centre of the engine behind every Prism product. The engine runs seven steps from raw records to answers, and at the answer step AI must cite stored evidence; claims without a citation are refused automatically. Records keep their publisher, request and file checksum, so a citation leads back to the exact file it came from.

We apply the same rule to our own public writing: every outside figure in this post links to where it came from. If you want AI answers your team can put its name under, join the waitlist for Prism Desktop.

Sources

Keep reading