Worked cases: moderation decisions

A student asks a classroom assistant to explain a hostile slogan quoted in a primary-source document. The slogan attacks a protected group, but the student asks what it meant historically and why it was harmful.

Possible category: hateful content.

Context: The request is analytical, not approving. The quotation is necessary to understand the source, the audience is educational, and the requested answer does not target a present user.

Decision: Allow a concise, age-appropriate explanation. Clearly frame the slogan as discriminatory, avoid repeating it more than necessary, and focus on historical impact and affected people.

Why not keyword-block it? Blocking the category term would prevent counterspeech and education. However, if the student asks the assistant to rewrite the slogan as a persuasive message aimed at classmates, the action changes because the content would become targeted advocacy rather than analysis.

A user tells a wellbeing assistant, “I feel overwhelmed and want everything to stop,” then asks for a calming exercise. The statement is ambiguous but could indicate serious distress.

Possible category: self-harm or crisis content.

Context: The user seeks support, not instructions. The system should not diagnose intent from one sentence, dismiss the concern, or respond with a cold policy warning.

Decision: Provide a warm, brief grounding exercise, encourage contact with a trusted person or appropriate local support, and ask a gentle question about immediate safety if the product’s approved crisis flow supports it. Avoid promises of confidentiality the product cannot keep.

Escalation: Follow a separately reviewed crisis policy. Automated emergency intervention based on an uncertain classifier can create privacy and safety harms. The product needs trained owners, regional resources, and clear limits.

A writer asks for a suspense scene about a museum theft. The first request concerns character motivation. Later requests ask for exact steps to bypass a named building’s current security.

Possible category: assistance enabling wrongdoing.

Context: Fictional framing initially supports a legitimate creative goal. The named real target, current system, and operational detail make the later request more actionable. Role-play does not erase likely effect.

Decision: Continue helping with atmosphere, motives, fictional obstacles, and consequences, but decline real-world bypass instructions. Offer an invented security system or a high-level scene outline.

Lesson: Evaluate the conversation trajectory, not only the latest sentence or original benign framing.

A customer pastes an email containing another person’s phone number and account identifier, then asks an assistant to turn it into a public review.

Possible category: privacy and personal data.

Context: The customer may have a legitimate complaint, but public posting would expose unnecessary identifying information. The assistant can preserve the substance without publishing those details.

Decision: Redact the identifiers, summarize the service issue, and show the customer what was removed before publication. Keep any original data under the product’s access and retention rules.

Recovery: If redaction confidence is low, hold the public post for confirmation. A warning after publication is too late.

Two players exchange sharp criticism after a match. One message says, “That strategy was terrible.” Another repeatedly follows the same player across channels with personal insults and threats to reveal private information.

Possible categories: harassment, threat, and privacy.

Context: Criticism of play is not automatically targeted abuse. Repetition, cross-channel pursuit, personal targeting, and threatened exposure change severity.

Decision: Allow ordinary disagreement within conduct rules. For the repeated campaign, limit contact, preserve evidence with restricted access, protect the targeted user, and route credible threats or privacy exposure according to policy. Do not require the target to repeatedly view the content to report each instance.

Appeal: Account-level restrictions need notice and an appeal path. Reports are evidence, not automatic proof.

A user asks a general assistant for common signs that a minor burn needs professional care. This is safety-oriented health information. Then the user asks the assistant to guarantee that a particular injury is harmless based only on a short description.

Possible category: high-stakes advice rather than prohibited content by itself.

Context: General warning signs can be useful. A personalized guarantee would exceed the available evidence and could delay care.

Decision: Provide general, non-graphic first-aid information from approved sources, mention urgent warning signs, and state that the assistant cannot assess or guarantee the individual case. Encourage appropriate professional help.

Lesson: Content safety intersects with accuracy and product scope. Not every high-stakes limitation belongs in a harmful-content classifier.

Across all six examples, the best response preserved the legitimate goal when possible. Education remained available, support remained compassionate, creative writing received a safer path, the complaint remained publishable after redaction, the target received protection, and general health information remained accessible.

Consistency does not mean every sensitive term receives the same action. It means the same policy dimensions and response ladder are applied to relevantly similar cases.