How a flexible model memorizes quirks

A flexible model has enough room — many knobs, deep trees, wiggly curves — to capture real signal and accidental noise. During fitting, the process keeps improving the practice score. If remembering a quirk lowers practice error, the process is rewarded for it.

A held-out validation set estimates “will this work on new questions?” because those labels did not steer the updates. That is why teachers use unseen quizzes: to check understanding, not photocopy memory.

Compact anchor: generalization gap ≈ training score − validation score. Example: 0.99 − 0.84 = 0.15. Units matter: say “15 percentage points on this spam/shot task,” not a mysterious floating 0.15 with no story.

You train a “will this free throw go in?” toy on 40 filmed shots from one gym. The flexible version notices the clock always showed 3:15 in makes — a quirk of when the team practiced. Practice accuracy soars. On a new gym at a different time, the clock cue vanishes and validation falls. The mechanism is not magic betrayal; it is rewarding quirk-fitting on the practice set.

Simpler hypotheses, stopping early, stronger “keep it simple” pressure, and more varied examples all shrink the room available for quirks. They do not guarantee perfection; they change what the mechanism is allowed to cling to.

At each step ask: What entered? What got stored? What is still uncertain? A shiny final score can hide a model that only memorized today’s worksheet font.

Trace cause and effect rather than collecting fancy words. First identify the state before fitting. Next identify what the update actually changes. Finally identify what reaches the next quiz and what remains uncertain. That separation prevents a category error: treating a convenient “accuracy: 99%” screen as proof the system understood spam, shots, or science ideas.

Ask which quantities were measured, which were estimated, and whether an average hides a subgroup (only one gym, only one phishing style). If the mechanism cannot explain a surprising validation drop, inspect boundary conditions — label noise, tiny n, leaked test rows — before twisting knobs at random. Write one sentence beginning “During fitting, the process is rewarded when…” and finish it without using the word magic.