It Is Not Lying to You
Lying means knowing the truth and choosing otherwise. It does not know the case is fake, and it does not know that it does not know.
The short answer
It was not lying to him. That word carries an accusation the machine cannot earn, and reaching for it sends you looking for the wrong thing.
What lying would require
To lie, you have to know the true answer and decide to say a different one. Three things have to be true at once: you know the fact, you know your statement contradicts it, and you choose the statement anyway.
None of those three hold here. It does not know the case is fake. It does not know that it does not know. And nothing in what it produces marks the difference between the answer it is sure of and the answer it assembled out of thin air.
That last one is the part that costs people money. There is no tell. The invented citation does not arrive hedged, or slower, or softer. It reads exactly like the four that were real.
What it is doing instead
It was trained on patterns, so it produces the answer that is shaped like a correct answer.
A legal citation has four parts: a case name, a volume number, a court and a year. Each has a familiar shape. A machine that has read a great many of them can produce all four in the right order and the right format without any of them pointing at a case that exists.
It is not trying to be accurate. It is trying to fill the gap you handed it, and the shape of a correct answer is the only material it has for that.
Why the word matters
If you believe it lied, you start watching for the moment it decides to deceive you, and you will watch forever, because that moment never comes. You also start treating it as untrustworthy in general, which usually ends with going back to doing everything by hand.
If you understand that it cannot see the difference itself, you do something smaller and more useful: you check the handful of answers where being wrong would actually cost you something.
Check yourself
Name the three things a lie requires.
Knowing the true answer, knowing your statement contradicts it, and choosing the statement anyway.
Which of those three does the machine have?
None of them. It does not know the answer is false, and it does not know that it does not know.
Why can you not catch this by listening to its tone?
The confident tone is part of the pattern it learned. It sounds the same whether the answer is real or invented.
What is it actually producing?
An answer shaped like a correct answer. The format and the order of the parts, without a guarantee that the parts point anywhere.
What is the practical consequence of using the wrong word?
You end up watching for a moment of deception that never arrives, instead of checking the few answers where an error would cost you.
Words used here
Why it makes things up • Token • AI agent
Go deeper
The full version of this, including why the training rewards a guess over an admission of uncertainty, the four kinds of claim worth checking and a thirty second check, is in Why AI Makes Things Up.
What she says
A lawyer handed a judge six cases to back up his argument. All six were invented by an AI. It was not lying to him. Lying means knowing the truth and choosing otherwise. It does not know it is wrong.
It produces the answer that is shaped like a correct answer. It was trained on patterns. So it fills in what a right answer usually looks like. It is not trying to be accurate. It is trying to fill the gap you handed it.
Case name, volume, court, year. Every shape perfect. None of them real. Said in exactly the voice it uses when it is right. Subscribe. No permission required.
Comments
No comments yet. Ask what did not make sense.