Bias and fairness
Diagnose common failures: school risk scores
Failures: proxy features, unbalanced evaluation, and “we treated everyone the same.”
1Learn the idea
Read
Samness is not fairness
See it
If examples leave people out, the model can leave them out too
Name failures specifically. Asha Mensah refuses the single bucket “the AI messed up” when discussing school risk scores at Lumen Youth Programme. Separate data problems, task-framing problems, interface problems, and governance problems. Each needs a different repair, and only some involve retraining—keep the standing case (audit an application screening tool for unfair disparities) in the room.
During a real interruption at Lumen Youth Programme, Asha Mensah stress-tests “Samness is not fairness” on school risk scores: one queued question, one hurried call, one hallway challenge. If the idea only works in a quiet workshop, it will not survive the standing case (audit an application screening tool for unfair disparities).
Read
Proxies that smuggle postcode in
Build a tiny failure gallery with school risk scores and medical triage data. For each, describe a plausible confident mistake, the first human who should notice at Lumen Youth Programme, and a fix that is not “ask it again.” Plausibility matters: cartoon failures do not train judgment for Asha Mensah.
Count something crude about school risk scores—misses last week, minutes lost, or people affected—and write the number beside medical triage data. Asha Mensah needs that comparison before anyone at Lumen Youth Programme declares victory on the standing case (audit an application screening tool for unfair disparities).
Read
Triage data that underrepresents some clinics
List warning signs around school risk scores that justify slowing public claims even if a pilot continues privately: missing owners, no logged overrides, identical outputs for dissimilar people, vendors who will not state training scope. When several signs coincide, freeze marketing language tied to the standing case (audit an application screening tool for unfair disparities).
On “Triage data that underrepresents some clinics”, Asha Mensah edits language about school risk scores the way an editor would: strike “sentient,” “infallible,” and “just a tool” wherever they hide responsibility inside Lumen Youth Programme. medical triage data stays nearby as a plain-language control.
Read
Dashboards that hide slice errors
Write one stop condition for school risk scores with authority attached—a named role at Lumen Youth Programme who can pause use. Stop conditions without authority are theatre. Asha Mensah gets initials on the page before the next launch review, and uses medical triage data to show what “pause” looks like in a simpler system.
For “Dashboards that hide slice errors”, a second person at Lumen Youth Programme challenges Asha Mensah’s note on school risk scores and asks whether medical triage data already solves most of the need with less mystery. That challenge is part of finishing the standing case (audit an application screening tool for unfair disparities), not a delay tactic.
Go deeper
Before you start
Why this matters
Invent one confident wrong output for school risk scores that would look fine in a screenshot. Asha Mensah classifies the miss as data, framing, interface, or governance—and says which fix comes first. Repeat once for medical triage data with a different class.
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.