Page 8 of 8~116 min topic

Why AI makes mistakes

Mastery check: trust with evidence

Mastery means designing a repeatable trust process: define acceptable evidence, verify high-impact claims, document uncertainty, and stop when support is inadequate.

~13 min this pageMastery check

1Try it yourself

Simulation game

Hallucination hunt

Stamp each claim: Trap or Trust. Confident voice ≠ true.

Quiz show

SOUNDS SURE

Sydney is the capital of Australia.

2Learn the idea

Read

The core idea

See it

Why fluent answers can still be wrong
01Predict ≠ lookupSounds like an answer
02Web is messyFacts + fanfic mix
03No embarrassmentCan sound sure
04Prompt trapAsked to invent detail

Confidence is a tone — verify before you act

Mastery means designing a repeatable trust process: define acceptable evidence, verify high-impact claims, document uncertainty, and stop when support is inadequate.

Read

A practical lens

Use this three-part method:

  1. Write a claim-checking standard. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
  2. Assign sources and reviewers to important claim types. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
  3. Measure errors and improve the workflow over time. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.

Read

Worked example

Walk through You are responsible for a weekly AI-assisted briefing that combines summaries, dates, quotations, and recommendations.. Label three moments where “Mastery check: trust with evidence” changes what you trust: (1) the first fluent answer, (2) the first missing source or permission, and (3) the decision a human must own. Write the before/after task so the model only does the slice that evidence supports. Keep one sentence that states how this page’s idea differs from a generic “AI is smart/dumb” score.

Read

Common traps and better moves

  • Relying on memory after the briefing ships. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
  • Tracking only obvious errors reported by others. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
  • Calling a workflow safe because it worked last week. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.

Read

Build the habit

Before you close the tab, capture a reusable habit for Mastery check: trust with evidence inside Why AI makes mistakes: name the observable check, the evidence you would open, and the stop condition. Rehearse it once on a low-stakes example, then once on a higher-stakes variant. The habit succeeds when you can explain the check without reopening this lesson. Target outcome: Explain why plausible language generation does not guarantee factual accuracy.

Go deeper

Before you start

Why this matters

You are responsible for a weekly AI-assisted briefing that combines summaries, dates, quotations, and recommendations.

In the wild

See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.

Check your understanding

Page assessment

Answer from memory. Completion is saved from this evidence, not from opening the next page.

Local focus for Mastery check: trust with evidence (Why AI makes mistakes): write the smallest test that would falsify a confident claim on this page, name the evidence you would open first, and note who must approve if the cost of being wrong is more than a redo. Keep the note under ten lines so you will actually reuse it.

1. Explain the page’s core distinction without using the word “smart.”
2. Which fact, source, permission, or test would most change your judgment in the opening scenario?
3. Name one low-consequence use where a light check is enough and one high-consequence use where independent review is required.
4. What should a responsible user do when the available evidence cannot support the requested conclusion?

All responses are required.