Why does it make things up?

It is not lying to you.

Lying means knowing the truth and choosing to say something else. A language model does neither. It does not keep a record of what is true.

The problem is not honesty. It is that the model has no built-in way to check what is true.

"Hallucination" is not a perfect name either. Nothing has broken. The model is doing what it was built to do.

It learned what sounds right, not what is true.

Remember lesson 02. Hide the next word. Guess it. Get corrected. Repeat this trillions of times.

The model was rewarded for predicting likely text. It was not directly checking whether each answer was true.

A believable wrong answer can look just as natural as a correct one.

Facts are spread out, not stored.

There is no list of facts inside the model. What it learns is spread across billions of numbers called weights.

Common information leaves a stronger pattern. Rare information leaves a weaker one. So knowing and not knowing are not separated by a clear line.

That is why it knows common facts well but may struggle with something specific to your company.

Watch the answers get weaker.

Move from something written about millions of times to something rarely mentioned. The answers do not suddenly stop. They slowly become less reliable.

The problem is that the writing can still sound just as confident.

Watch the last bar. The answer gets weaker, but the confidence still sounds the same.

It always has to pick something.

At every step, the model scores possible next tokens and picks one. There is no built-in option that means I do not have enough information.

So when the right answer is weak or missing, another answer can still win simply because it looks plausible.

This is why fake citations can look so real. The model has learned what a real citation should look like.

Good writing does not mean correct writing.

People often sound unsure when they are unsure. We naturally use that as a clue.

AI does not work that way. A correct answer and a wrong answer can both sound equally smooth and confident.

If you spot the fake, it is usually because you know the topic. The writing itself may give you no warning.

People preferred confident answers.

In later training, people compared answers and chose the better one. Clear and confident answers were often preferred over uncertain ones.

That pushed models toward sounding helpful and complete, even when uncertainty would sometimes be safer.

Nobody asked the model to bluff. It was a side effect of rewarding answers people liked.

Do not make it remember. Give it the source.

Instead of asking the model to answer from memory, give it the information it needs and ask it to answer from that.

Also let it say when the answer is missing. Then check that any source it gives you is real.

This approach has a name. It is called RAG, and that is what we explore next.

Confidence means nothing

Tone is generated by the same process as substance. A sure-sounding answer is not evidence of a correct one.

Rare topics are riskiest

The thinner the training coverage, the more the model is filling in — and your niche is thin coverage.

Specifics are the warning

An oddly precise clause number on an obscure question is the shape of invention, not the shape of certainty.

It cannot audit itself

Asking “are you sure?” runs the same machinery again. Agreement is not verification.

Grounding beats prompting

No instruction makes it know something. Putting the source in the context changes what it is doing.

Give it an exit

Explicit permission to answer “not in the documents” is one of the few things that measurably reduces invention.