Why AI makes mistakes
Prediction is not knowledge
A language model produces likely continuations from patterns. A likely sentence can be grammatical and relevant without corresponding to a checked fact.
1Try it yourself
Simulation game
Hallucination hunt
Stamp each claim: Trap or Trust. Confident voice ≠ true.
Quiz show
“Sydney is the capital of Australia.”
2Learn the idea
Read
The core idea
See it
Confidence is a tone — verify before you act
A language model produces likely continuations from patterns. A likely sentence can be grammatical and relevant without corresponding to a checked fact.
Read
A practical lens
Use this three-part method:
- Separate fluency from evidence. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
- Identify claims that describe the outside world. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
- Ask what source or tool could actually confirm each claim. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
Read
Worked example
Walk through A travel assistant states that a small museum opens every Monday, even though the museum has always closed on Mondays.. Label three moments where “Prediction is not knowledge” changes what you trust: (1) the first fluent answer, (2) the first missing source or permission, and (3) the decision a human must own. Write the before/after task so the model only does the slice that evidence supports. Keep one sentence that states how this page’s idea differs from a generic “AI is smart/dumb” score.
Read
Common traps and better moves
- Treating a polished paragraph as a database record. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
- Assuming specificity proves research. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
- Confusing a useful draft with a verified answer. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
Read
Build the habit
Before you close the tab, capture a reusable habit for Prediction is not knowledge inside Why AI makes mistakes: name the observable check, the evidence you would open, and the stop condition. Rehearse it once on a low-stakes example, then once on a higher-stakes variant. The habit succeeds when you can explain the check without reopening this lesson. Target outcome: Explain why plausible language generation does not guarantee factual accuracy.
Go deeper
Before you start
Why this matters
A travel assistant states that a small museum opens every Monday, even though the museum has always closed on Mondays.
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
Local focus for Prediction is not knowledge (Why AI makes mistakes): write the smallest test that would falsify a confident claim on this page, name the evidence you would open first, and note who must approve if the cost of being wrong is more than a redo. Keep the note under ten lines so you will actually reuse it.
All responses are required.