Page 5 of 8~104 min topic

Bias and fairness

Diagnose common failures: school risk scores

Failures: proxy features, unbalanced evaluation, and “we treated everyone the same.”

~13 min this pageDiagnose common failures — plausible mistakes and warning signs

1Learn the idea

Read

Samness is not fairness

See it

Skewed data → skewed guesses
Mostly group AFew group B
Learnercopies the skew
Uneven resultscheck who is hurt

If examples leave people out, the model can leave them out too

Name failures specifically. Asha Mensah refuses the single bucket “the AI messed up” when discussing school risk scores at Lumen Youth Programme. Separate data problems, task-framing problems, interface problems, and governance problems. Each needs a different repair, and only some involve retraining—keep the standing case (audit an application screening tool for unfair disparities) in the room.

During a real interruption at Lumen Youth Programme, Asha Mensah stress-tests “Samness is not fairness” on school risk scores: one queued question, one hurried call, one hallway challenge. If the idea only works in a quiet workshop, it will not survive the standing case (audit an application screening tool for unfair disparities).

Read

Proxies that smuggle postcode in

Build a tiny failure gallery with school risk scores and medical triage data. For each, describe a plausible confident mistake, the first human who should notice at Lumen Youth Programme, and a fix that is not “ask it again.” Plausibility matters: cartoon failures do not train judgment for Asha Mensah.

Count something crude about school risk scores—misses last week, minutes lost, or people affected—and write the number beside medical triage data. Asha Mensah needs that comparison before anyone at Lumen Youth Programme declares victory on the standing case (audit an application screening tool for unfair disparities).

Read

Triage data that underrepresents some clinics

List warning signs around school risk scores that justify slowing public claims even if a pilot continues privately: missing owners, no logged overrides, identical outputs for dissimilar people, vendors who will not state training scope. When several signs coincide, freeze marketing language tied to the standing case (audit an application screening tool for unfair disparities).

On “Triage data that underrepresents some clinics”, Asha Mensah edits language about school risk scores the way an editor would: strike “sentient,” “infallible,” and “just a tool” wherever they hide responsibility inside Lumen Youth Programme. medical triage data stays nearby as a plain-language control.

Read

Dashboards that hide slice errors

Write one stop condition for school risk scores with authority attached—a named role at Lumen Youth Programme who can pause use. Stop conditions without authority are theatre. Asha Mensah gets initials on the page before the next launch review, and uses medical triage data to show what “pause” looks like in a simpler system.

For “Dashboards that hide slice errors”, a second person at Lumen Youth Programme challenges Asha Mensah’s note on school risk scores and asks whether medical triage data already solves most of the need with less mystery. That challenge is part of finishing the standing case (audit an application screening tool for unfair disparities), not a delay tactic.

Go deeper

Before you start

Why this matters

Invent one confident wrong output for school risk scores that would look fine in a screenshot. Asha Mensah classifies the miss as data, framing, interface, or governance—and says which fix comes first. Repeat once for medical triage data with a different class.

In the wild

See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.

Check your understanding

Page assessment

Answer from memory. Completion is saved from this evidence, not from opening the next page.

1. In Asha Mensah’s scene, what bounded task does school risk scores perform at Lumen Youth Programme?
2. Which observation would most change your judgment about school risk scores, and why?
3. How should medical triage data alter the quality bar or the language you use?
4. Who can correct a miss before harm spreads, and what authority do they need?
5. How does this page advance the case: audit an application screening tool for unfair disparities?

All responses are required.