An AI hallucination is an output from a generative AI model that sounds right but is not: a confident answer, a statistic, a quote or a source the model produced that matches no real fact or document. The term is used mainly for generative language models, the tools that write text. The hard part is that a hallucination reads just as fluent and assured as a correct answer, so you cannot spot it by how it sounds. You can only spot it by checking.

The two ways it goes wrong

A 2023 survey in ACM Computing Surveys split the problem into two kinds, which is useful when you decide how to defend against it. A model can contradict a source you gave it, which researchers call unfaithful, or it can contradict the real world, which they call not factual. The first is a summary that adds a detail your document never contained. The second is an invented fact about the world. Both read as confident, and both count as hallucinations.

Why they happen

The root cause sits in how the tool works. A generative language model is built to produce the most likely next words given your request, not to verify that those words are true. When it knows the answer, the most likely text is also correct. When it does not, the most likely text is a confident guess that fits the pattern, and the model has no internal signal telling it which situation it is in.

There is a second reason that is less obvious. OpenAI researchers argued in a 2025 paper that the way models are trained and tested rewards guessing over honesty. On a typical benchmark, a wrong answer and an "I don't know" both score zero, while a lucky guess scores a point. A student who guesses on every exam question outscores one who leaves blanks, so the model learns the same habit: when unsure, answer anyway. The fix they point to is to stop penalising honest uncertainty, which helps but does not remove the behaviour.

A real case, with real consequences

In 2023 a federal court in New York sanctioned two lawyers who had filed a brief citing six court decisions that did not exist. The citations came from ChatGPT, which produced the case names and quotes and, when asked, confirmed they were real. The court imposed a 5,000 dollar penalty in its Opinion and Order in Mata v. Avianca (S.D.N.Y., 22 June 2023).

For any company, the lesson is blunt: the tool gives no warning when it invents something, and the professional who submits its output unchecked owns the mistake. A law firm felt it first, but a fabricated figure in a financial report or an invented policy in a customer answer does the same damage.

Where this bites companies

Hallucinations do the most damage where the output looks authoritative and goes out without a second check:

  • A figure or a percentage in a report that the model invented to fill a gap.
  • A legal, tax or policy citation that points to a rule that does not say what the model claims.
  • A customer answer that promises a feature, a price or a return policy the company never offered.
  • Code that calls a function or a library that sounds real but does not exist.

Each of these is cheap to catch early and expensive to clean up later, so a habit of verifying beats fixing things after they ship.

How to reduce the risk

You cannot remove hallucinations, but you can cut them to a level your business can live with. The steps that work are practical and reinforce each other:

  • Ground the model in your own documents. A tool that answers from your files, and cites them, hallucinates far less than one answering from memory alone.
  • Ask for sources, then check them. Treat any citation, number or quote as unverified until you have seen the original.
  • Keep a person in the loop for anything that counts. Decisions about money, people, law or clients get human sign-off before they leave the building.
  • Start where the stakes are low. Use AI first on drafts and internal work, where an error is caught and corrected rather than shipped.
  • Train the team to verify. People who understand why the tool invents things know which parts to check, which is what AI literacy teaches.

Who owns these rules matters as much as the rules themselves, which is why hallucination risk belongs inside your wider AI governance rather than left to each person's judgement.

Frequently asked questions

What is an AI hallucination in simple terms?
An AI hallucination is an output from a generative AI model that sounds right but is not: a confident answer, a statistic, a quote or a source that the model produced and that matches no real fact or document. The text reads as fluent and sure of itself, which is exactly what makes it hard to catch.

Why do AI models hallucinate?
A generative language model is built to produce the most likely next words, not to check whether they are true. OpenAI researchers argued in 2025 that the way models are trained and tested also rewards confident guessing over admitting uncertainty, because a wrong guess and an honest "I don't know" both score zero on a typical test, so the model learns to guess.

Can AI hallucinations be fixed completely?
No. They can be reduced a lot with grounding in trusted documents, source checks and human review, but not eliminated, because generating plausible text is how the model works. The safe assumption is that any important output can contain a hallucination until a person verifies it.

Sources