Page 4 of 8~104 min topic

Prediction: your first ML idea

Check the scoreboard numbers

A green printout is not enough — validation means the buckets add up and match a hand count.

~13 min this pageEvaluation

1Learn the idea

Read

Validate against a hand baseline

On paper, mark each round:

| Round | Truth | Score | Guess (≥0.5?) | Bucket | | ----- | ----- | ----- | ------------- | ------ | | 1 | 1 | 0.8 | 1 | TP | | 2 | 0 | 0.3 | 0 | TN | | 3 | 1 | 0.6 | 1 | TP | | 4 | 1 | 0.9 | 1 | TP | | 5 | 0 | 0.2 | 0 | TN |

Hand total: TP=3, FP=0, TN=2, FN=0. Accuracy = 5/5 = 1.00. That is your baseline. Code must match it — not invent a vibier story.

Read

Run and compare

truth = [1, 0, 1, 1, 0]
scores = [0.8, 0.3, 0.6, 0.9, 0.2]
threshold = 0.5
preds = [int(score >= threshold) for score in scores]

pairs = list(zip(truth, preds))
tp = pairs.count((1, 1)); fp = pairs.count((0, 1))
tn = pairs.count((0, 0)); fn = pairs.count((1, 0))
accuracy = (tp + tn) / len(pairs)
print(preds, f"accuracy={accuracy:.2f}", tp, fp, tn, fn)

Expected evidence:

[1, 0, 1, 1, 0] accuracy=1.00 3 0 2 0

Read

What this metric does **not** prove

Beating the hand baseline on five practice rounds does not prove:

  • the game works on thousands of real messages,
  • every phone or every friend’s dataset,
  • that accuracy alone is enough (always show TP/FP/TN/FN).

Write “proved” and “not proved” as two columns in your notebook. Validation without that split turns into bragging.

Read

Denominator discipline

Always write the denominator: (tp + tn) / len(pairs) with len(pairs) = 5. A percent with no “out of what” can hide a tiny sample. Ask: if I only measured three rounds, would 100% feel the same?

Also try one negative thought experiment: report accuracy but hide FP and FN. Why is that unfair in a school spam demo or a shot-make game?

Read

Validation ritual you can repeat

Before you move on, do this three-step ritual out loud with a partner:

  1. Read the printed preds and point to each truth value.
  2. Tick which bucket each pair belongs in.
  3. Confirm the printed tp fp tn fn and accuracy=1.00 match your ticks.

If any step fails, stop. Do not “fix it later” by adding features. Validation is the gate that keeps later pages from inventing a softer target. Write the ritual in your notebook so page 8’s ship checklist can point back to it.

Go deeper

Before you start

Why this matters

Pretend you are the referee after five spam-or-not rounds. Before trusting accuracy=1.00, write how you would check: Do TP+FP+TN+FN equal 5? Does (TP+TN)/5 match the printed accuracy? What would a decorative win look like — a shiny number with no bucket check?

Check your understanding

Page assessment

Answer from memory. Completion is saved from this evidence, not from opening the next page.

1. Do the four counts sum to 5?
2. Does accuracy match (tp+tn)/5 by hand?
3. Did you list what five-round perfection does not prove?
4. Would you accept accuracy without buckets? Why or why not?

All responses are required.