Prediction: your first ML idea
Check the scoreboard numbers
A green printout is not enough — validation means the buckets add up and match a hand count.
1Learn the idea
Read
Validate against a hand baseline
On paper, mark each round:
| Round | Truth | Score | Guess (≥0.5?) | Bucket | | ----- | ----- | ----- | ------------- | ------ | | 1 | 1 | 0.8 | 1 | TP | | 2 | 0 | 0.3 | 0 | TN | | 3 | 1 | 0.6 | 1 | TP | | 4 | 1 | 0.9 | 1 | TP | | 5 | 0 | 0.2 | 0 | TN |
Hand total: TP=3, FP=0, TN=2, FN=0. Accuracy = 5/5 = 1.00. That is your baseline. Code must match it — not invent a vibier story.
Read
Run and compare
truth = [1, 0, 1, 1, 0]
scores = [0.8, 0.3, 0.6, 0.9, 0.2]
threshold = 0.5
preds = [int(score >= threshold) for score in scores]
pairs = list(zip(truth, preds))
tp = pairs.count((1, 1)); fp = pairs.count((0, 1))
tn = pairs.count((0, 0)); fn = pairs.count((1, 0))
accuracy = (tp + tn) / len(pairs)
print(preds, f"accuracy={accuracy:.2f}", tp, fp, tn, fn)
Expected evidence:
[1, 0, 1, 1, 0] accuracy=1.00 3 0 2 0
Read
What this metric does **not** prove
Beating the hand baseline on five practice rounds does not prove:
- the game works on thousands of real messages,
- every phone or every friend’s dataset,
- that accuracy alone is enough (always show TP/FP/TN/FN).
Write “proved” and “not proved” as two columns in your notebook. Validation without that split turns into bragging.
Read
Denominator discipline
Always write the denominator: (tp + tn) / len(pairs) with len(pairs) = 5. A percent with no “out of what” can hide a tiny sample. Ask: if I only measured three rounds, would 100% feel the same?
Also try one negative thought experiment: report accuracy but hide FP and FN. Why is that unfair in a school spam demo or a shot-make game?
Read
Validation ritual you can repeat
Before you move on, do this three-step ritual out loud with a partner:
- Read the printed
predsand point to each truth value. - Tick which bucket each pair belongs in.
- Confirm the printed
tp fp tn fnandaccuracy=1.00match your ticks.
If any step fails, stop. Do not “fix it later” by adding features. Validation is the gate that keeps later pages from inventing a softer target. Write the ritual in your notebook so page 8’s ship checklist can point back to it.
Go deeper
Before you start
Why this matters
Pretend you are the referee after five spam-or-not rounds. Before trusting accuracy=1.00, write how you would check: Do TP+FP+TN+FN equal 5? Does (TP+TN)/5 match the printed accuracy? What would a decorative win look like — a shiny number with no bucket check?
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.