The Decoder/Why does AI make things up?

Why does AI make things up?

An AI hallucination is a confident answer that is not true: the model produced the most likely sounding text rather than a checked fact, because it was trained and scored on producing text, not on knowing when to stop.

Mstimaj

By Fatima Jalloh • published 7 October 2026 • updated 30 September 2026 • free, no account

A language model does not look things up. It predicts the next piece of text from patterns in what it read. When the pattern is strong, the prediction is right. When the pattern is thin, it still produces something fluent, because fluent is what it was rewarded for. OpenAI's own researchers put it plainly in 2025: training and testing reward a confident guess over an honest blank, so models learn to guess.

Why does ChatGPT invent a fake citation?

Because a citation is a pattern too. The model has seen millions of references and knows exactly what one looks like: two surnames, a year, a plausible journal, a page range. When you ask for a source it does not have, the easiest text to produce is a reference shaped object. Nothing inside the model checks whether the paper exists. Citations, case names, statistics and URLs are the four places where this happens most, because all four have a rigid shape and a huge space of plausible fillers.

The fix is not to ask the model harder. It is to give it the document and ask it to quote from it, or to let it search and read the page before it answers.

Can I trust AI answers?

Trust them the way you trust a well read colleague speaking from memory. Reliable on the common and the general, less reliable on the specific, the recent and the numerical, and never a substitute for the source when the answer matters. The benchmarks put the best models at a few made up details per hundred summaries when they are given the text to summarise. Without the text, on open questions about obscure facts, the rate is much higher and nobody publishes a clean number for it.

The practical rule: anything you would need to cite, check. Anything that would cost money or a relationship if wrong, check.

How do I stop AI from making up sources?

Four things that work, in order of effect. Give it the material: paste the document, attach the file, or turn on web search, so the answer is grounded in text it can see. Give it permission to say it does not know; Anthropic’s own guidance is to state this in the prompt, and the OpenAI paper argues the whole problem starts with models being penalised for it. Ask for quotes before conclusions: have it pull the exact sentences first, then answer from those. And ask for the source next to every claim, then open one. If the first link is wrong, treat the rest as unchecked.

Which AI hallucinates the least?

On the one public test that is updated regularly, Vectara’s summarisation leaderboard, the spread between the leading models is small: the best sits under 2 percent and the widely used ones sit between 3 and 4 percent, as of late September 2026. That test measures one narrow task, summarising a document it was given, which is the easy case. It says little about open questions. The honest answer is that the differences between top models are smaller than the difference between asking with a source and asking without one.