Feedback loops and drift

Model outputs can shape the data collected next. Feedback loops amplify earlier choices, while drift changes the relationship between old examples and current reality.

Use this three-part method:

1. Distinguish user preference from exposure effects. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
2. Track data and performance over time. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
3. Add exploration, correction, and human review where loops could narrow outcomes. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.

Walk through A news recommender promotes sensational stories, receives more clicks on them, and then reads those clicks as proof that users prefer sensational news.. Label three moments where “Feedback loops and drift” changes what you trust: (1) the first fluent answer, (2) the first missing source or permission, and (3) the decision a human must own. Write the before/after task so the model only does the slice that evidence supports. Keep one sentence that states how this page’s idea differs from a generic “AI is smart/dumb” score.

• Feeding every model decision back as a true label. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
• Waiting for overall accuracy to collapse. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
• Ignoring people who leave and therefore stop producing feedback. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.

Before you close the tab, capture a reusable habit for Feedback loops and drift inside Data fuels AI: name the observable check, the evidence you would open, and the stop condition. Rehearse it once on a low-stakes example, then once on a higher-stakes variant. The habit succeeds when you can explain the check without reopening this lesson. Target outcome: Explain how training examples and labels shape learned patterns.