Why AI Makes Things Up

A lawyer with thirty years behind him asked an AI for cases to support his client in court. It gave him six. They had names, they had dates, they had judges, they had quotations with page numbers. Every one of them was invented. That was three years ago, and it keeps happening. There is a database that counts it.

As of yesterday it holds two thousand, two hundred and seventeen court decisions where somebody filed something an AI made up. Fifteen hundred and fourteen of those are in the United States. It was about two hundred in the middle of last year.

One careless man does not explain that. This happens to people who are paying attention, in a setting where being wrong costs money, and sometimes a licence. The word everyone uses for this is hallucination. I am not going to use it, because a hallucination is something you see that is not there, and nothing here is seeing anything.

Most people say it lied to them. That is closer. It is still not right, because lying means knowing the truth and choosing to say something else. It does not know it is wrong. Here is the part that makes everything else make sense. The thing you are typing to does not have a filing cabinet. It cannot go and look something up and come back.

What it has is a very good sense of which words tend to follow which other words, built out of an enormous amount of text. When you ask a question, it produces the answer that is shaped like a correct answer. Shaped like. That is the whole thing.

A real legal citation has a case name, a volume number, a court, and a year. All four of those have a shape. The machine can produce all four of them perfectly, without any of them pointing at a case that exists. In the same way that you could write a completely convincing address on an envelope. Correct street format. Correct postcode format. For a street that was never built.

The postman is the one who finds out. And it arrives in the same confident voice it uses when it is right. That is the part that catches people. There is no wobble in the made up citation. No hedge, no I think, no slight pause. It reads exactly like the four that were real.

This next part I find genuinely interesting, and it is not a bug somebody forgot to fix. Researchers at OpenAI published a paper on exactly this question, and the argument is short enough to say in one breath. These models are graded like students sitting an exam.

Think about a multiple choice test where a wrong answer costs you nothing. Four options. You have no idea. You leave it blank, and you get zero. You guess, and you are right one time in four. So take a hundred questions you do not know. Leave every one blank, and you score zero.

Now guess on all hundred. One in four of those guesses comes out right by luck, so twenty five of them are correct. Twenty five out of a hundred, for knowing nothing. Twenty five marks. For knowing nothing. Purely for being willing to answer.

Now make that student extremely good at guessing, because it has read more than you ever will, and give it tens of millions of questions. The arithmetic does not change. Guessing wins. That is what the grading does. The tests that decide which model is best mostly count how many answers came out right.

They award nothing at all for saying, I do not know. So a model that admits uncertainty scores worse than a model that produces a confident guess. And the one that scores better is the one that gets released, written about, and bought. The paper has a phrase for it. An epidemic of penalising uncertain responses.

So when it hands you a fake source with a straight face, it is not malfunctioning. It is doing the thing it was rewarded for, for years, by every benchmark that mattered. You do not need to check everything. That would make the tool useless, and you would stop using it by Thursday.

What you need is to know which four situations are different from the rest. One. Anything with a number in it. Prices, dates, percentages, dosages, measurements, distances, how many grams of sugar are in the thing. A number is the easiest thing in the world for it to produce, and the hardest thing for you to spot as wrong, because a wrong number looks exactly like a right one.

Two. Anything it says somebody else said. Quotations, citations, studies, court cases, who wrote what and when. This is where the lawyers got caught. If it names a source, treat the source as a claim. Not as evidence. Three. Anything about the last few months.

The model learned from text that stopped at some point, and that point is further back than you think. Ask it about something recent, and it will very often answer anyway, out of the shape of what a recent answer sounds like. Four. Anything you are about to put your name on.

Sent to a client. Filed with a court. Posted publicly. Handed to your manager. Not because an error is more likely there. Because an error costs more there. Everything else, the first draft, the rewrite, the email you were dreading, the thing you are only using to think out loud with, you can leave alone. You will notice if that goes wrong.

So, how do you check. Three steps, and you can do all three while the kettle boils. Ask it for the source. Not, are you sure. Are you sure only tells you whether it will fold when challenged, and folding under pressure has nothing to do with whether it was right the first time.

Ask it for the source. Say it plainly. Where did this come from, and give me a link I can click. Open the link. Actually open it. A link that does not load, or that loads something about a different subject entirely, is your answer. This one catches most of it on its own.

Search the key fact by itself. Take the number, the name, the quotation, and go looking for it somewhere that is not the thing that just told you. If it is real, it exists in more than one place. Source. Open. Search. Thirty seconds, and you only spend them on the four things.

The people in that database were not stupid, and most of them were not lazy either. They were doing the thing the tool quietly invites everybody to do, which is to accept an answer that looks precisely like a correct answer. The habit worth building is not suspicion. Suspicion is exhausting, and it will send you back to doing everything by hand inside a week.

The habit is knowing which four things to slow down for, and spending your thirty seconds there, instead of spreading them thin across everything. The tool is not going to start telling you when it does not know. It was graded for years on how often it was right, and never once on whether it knew where its own knowledge stopped.

That part is still your job. It is a small job. It is thirty seconds. But nobody else in the process is doing it for you. Subscribe. No permission required.