Trace: spam filter or sports-shot toy numbers

Scenario A — spam filter: Practice texts look easy; the flexible model hits 99% train and 84% val. You reduce tree depth (simpler rules). New scores: 93% train, 91% val. Gap shrinks from 15 to 2 points. Prefer the second model for estimated generalization.

Scenario B — sports-shot toy: Same numbers, different story: the flexible model memorized one gym’s quirks; the simpler model keeps more skill on a new court.

Numbered state transitions:

1. Capture input — labeled practice set + held-out val set; version your notebook.
2. Apply mechanism — flexible fitting can absorb signal and noise; val labels do not update weights.
3. Record controls — capacity / tree depth, split, early-stop patience.
4. Inspect signal — gap = 0.99 − 0.84 = 0.15.
5. Compare to baseline — flexible vs simpler (93/91, gap 0.02).
6. Store evidence — both score pairs, control change, and a “does not prove worldwide performance” note. No real classmate message text in the log.

Change one assumption: validation labels were noisy, or the friend set comes from a different language, or you accidentally tuned on val for a week. Mark which steps break. Silent “still 99% train” is not enough — say whether the original ship decision still holds.

Hold everything fixed and imagine val is 88% instead of 84%. Gap becomes 11 points — still large. Or imagine simpler model lands at 90/89. Decision may stay the same. Write both beside the original so a reviewer sees the first divergence point.

A worked trace is not a marketing story. It is a lab notebook page another student can re-run. Use short labels: Input v1, Flexible fit, Gap 0.15, Simpler depth, Gap 0.02, Decision: ship simpler. If you catch yourself writing “the AI understood spam,” rewrite to “the flexible fit reduced training error, including possible quirks; validation stayed lower.”

Close the page by stating the unit and denominator again: percentage points on this labeled toy set. That discipline keeps the dramatic 15-point gap from floating free of meaning when you paste it into a slide.