Page 4 of 8~96 min topic

Overfitting playground

Perfect practice vs okay on a surprise quiz

Choosing a model is choosing which score you refuse to worship — the practice trophy or the surprise-quiz truth.

~12 min this pageTradeoffs

1Learn the idea

Read

The tradeoff in plain sight

Too flexible → can nail practice (even quirks) → larger chance the surprise quiz falls.

Too simple → may miss real patterns → both practice and surprise quiz look weak (underfitting).

Balanced → practice looks strong-but-not-magical; surprise quiz stays close.

Numbers to keep:

  • Flexible: train 99%, val 84% → gap 15.
  • Simpler: train 93%, val 91% → gap 2.

If your club poster only shows 99%, you are advertising the trophy that is easiest to fake by memorization. If you show both numbers, people can weigh the tradeoff.

Read

Sports and spam versions

Sports-shot toy: memorizing one gym’s lighting quirks can yield near-perfect practice makes. On another court, accuracy drops. A simpler “arc + release timing” rule might lose a few practice points and win on new courts.

Spam toy: memorizing one phishing phrase from the practice file can inflate practice accuracy. New phish wording arrives tomorrow; the memorizer fails. A simpler “urgent + ask for password” pattern may generalize better.

Read

What you tolerate

Every tradeoff needs a tolerated loss. Example: “We accept practice dropping from 99 to 93 if validation rises from 84 toward 91.” If you cannot name the tolerated regression, you are not ready to pick.

Also trade time and complexity: a wiggly model may be harder to explain to your teacher. A simpler rule is easier to defend in a project write-up.

Read

Choosing in public

On a club poster, print both score pairs side by side. Add one sentence: “We chose the smaller gap because surprise-quiz honesty beats practice bragging.” Invite a challenge: a classmate may prefer the 99% toy for a game high-score. That preference is fine for entertainment — name it — but do not smuggle it into a “ready for school inbox” claim.

Practice the awkward sentence: “Our practice score got worse on purpose.” It sounds wrong until you show the validation lift. Tradeoffs become teachable the moment you stop hiding the number that went down. If the tolerated regression is unknown, the change is not ready for a consequential workflow — even a classroom one that filters messages or grades hints.

Go deeper

Before you start

Why this matters

Would you rather score 100% on a practice packet you already saw, or 90% on a surprise quiz with new wording? Write your pick and one sentence why. Then ask: which pick would you trust more for tomorrow’s real test?

Check your understanding

Page assessment

Answer from memory. Completion is saved from this evidence, not from opening the next page.

1. Why can a lower practice score be the better choice?
2. What is the generalization gap in the 99/84 example?
3. What does underfitting look like in school terms?
4. Why must “tolerated regression” be named before celebrating?

All responses are required.