Held-out quiz and a friend’s new examples

Useful evaluation for overfitting questions:

1. Held-out quiz (validation) — labeled examples kept aside while fitting.
2. Final test — used rarely, after you stop knob-twisting.
3. Friend tries different examples — new phrasings, new gym lighting, new handwriting; a social check that your notebook was not the whole world.

Report train and validation together. A lone 99% is incomplete evidence. Prefer the pair 93/91 over 99/84 when the gap shrinks and the task still matters.

• Split before training; write the split rule in the notebook.
• Compute generalization gap = train − val (example start: 0.99 − 0.84 = 0.15).
• Decide an acceptance rule in advance: “ship if val ≥ 90% and gap ≤ 5 points,” not “ship if the demo looks cool.”
• After one friend trial set, note domain shift: their examples may be harder; that is information, not sabotage.

A good val score on 40 club messages does not prove city-wide spam performance. Say the denominator. Say who labeled the answers. Say whether the friend set overlapped practice.

• Prefer measured baselines over persuasive screenshots.
• Keep versions so you can reproduce the score.
• Abstain from big claims when evidence is outside your contract (tiny n, samey data, peeked test).

Ask a partner to bring ten examples you have not ranked yet. You score them once, record train/val/friend numbers, and resist the urge to immediately retune until the friend set also looks perfect — that would recreate the failure mode from the previous page. Thank them for hard examples; difficult cases are gifts.

Write the evaluation contract before you see results: metrics, minimum validation score, maximum gap, and a clear “we will not claim…” line. Afterward, compare results to the contract without moving the goalposts. If you must change the contract, say so explicitly and treat it as a new experiment version, not a silent rewrite of history.