False positives and false negatives

Compare the system decision with a trusted reviewed label:

• True positive: harmful content is correctly flagged.
• False positive: permitted content is incorrectly flagged.
• True negative: permitted content is correctly allowed.
• False negative: harmful content is incorrectly allowed.

These four outcomes form a confusion matrix. Counts alone do not explain why mistakes happened, but they prevent vague claims such as “the filter is 95% accurate” from ending the discussion.

Imagine a test set of 1,000 community posts: 100 violate policy and 900 are allowed. A filter catches 80 violations and misses 20. It also flags 90 allowed posts. It is correct on 890 posts, or 89 percent. Yet one in five real violations passed, and more allowed posts were flagged incorrectly than violating posts were caught. Overall accuracy hides both problems.

Precision asks: Of all items the system flagged, how many truly violated policy? In the example, 80 of 170 flags were correct, so precision is about 47 percent. Low precision creates reviewer burden and frustrating blocks.

Recall asks: Of all actual violations, how many did the system catch? Here it caught 80 of 100, so recall is 80 percent. Low recall leaves harmful content available.

Increasing sensitivity often improves recall while reducing precision. Raising a threshold may reduce false positives but miss more subtle violations. The right balance depends on category and action. A threshold for placing a warning on a private draft can differ from one that automatically suspends an account.

False negatives can expose users to harm, allow abuse to spread, or permit a dangerous action. False positives can silence support-seeking language, block people discussing their own identity, interrupt education, or unfairly penalize an account. Both matter.

Ask who pays:

• Does the user lose a draft, a post, account access, or income?
• Is the content merely queued for review, or permanently removed?
• Can harmful material reach one person or millions?
• Can the decision be reversed quickly?
• Is there a meaningful appeal?
• Does a false flag expose sensitive content to additional reviewers?

A lower-confidence signal may justify ranking an item for review but not justify automatic punishment. Match the strength of evidence to the consequence of the action.

Safety classifiers can perform differently across languages, dialects, spelling styles, disabilities, reclaimed terms, and cultural contexts. A global score can look acceptable while one group experiences many more false positives. Test slices that reflect the product’s actual users and known risks.

Useful slices include:

• language and region;
• age-appropriate versus adult contexts;
• quoted, educational, news, and counterspeech examples;
• direct attacks versus self-reference;
• short slang-heavy text versus formal prose;
• text, image, audio, and mixed-media inputs;
• new users, high-reach accounts, and private conversations.

Do not invent identity labels from user content merely to create metrics. Use consented, privacy-aware evaluation data and expert review where necessary.

Human reviewers can disagree, especially on context-heavy boundaries. Build a labeling guide with examples and reasons. Measure reviewer agreement. Send hard cases to adjudication rather than pretending one quick opinion is unquestionable truth. Preserve policy versions because a decision can change when a policy changes.

Production feedback adds evidence: successful appeals, user reports, reviewer reversals, and incident investigations. Reports are signals, not automatic proof; coordinated users can misuse reporting systems.

Because errors are inevitable, recovery is part of safety:

1. Tell users what broad rule affected the decision when doing so is safe.
2. Preserve their work when possible instead of silently deleting it.
3. Offer a correction path or safer rewrite.
4. Provide an appeal proportionate to the consequence.
5. Keep account penalties separate from uncertain content labels.
6. Track reversals and feed recurring failures into evaluation sets.

Review queue capacity also matters. A threshold that generates 50,000 daily flags is not useful if trained reviewers can handle 2,000. Backlogs lengthen harm exposure and appeal times.