Harmful categories and policy layers

Products organize policies differently, but learners should recognize several recurring families:

• Violence and threats: credible threats, praise or encouragement of violence, and material that could intensify real-world danger.
• Hate and targeted harassment: attacks, dehumanization, intimidation, or repeated abuse aimed at people, especially based on protected characteristics.
• Sexual harm and exploitation: non-consensual material, sexual exploitation, and any sexual content involving minors. These areas require strict rules and trained escalation.
• Self-harm and crisis content: encouragement, instructions, or romanticization, distinguished from supportive conversation, prevention, and recovery.
• Wrongdoing and dangerous instructions: help that substantially enables fraud, evasion, weapon construction, unauthorized access, or other harmful acts.
• Privacy and personal data: exposure, inference, or misuse of sensitive identifying information.
• Deception and manipulation: scams, impersonation, coordinated manipulation, or synthetic media presented in a misleading way.
• Age-inappropriate or graphic material: content that may be permitted in one controlled setting but unsuitable for a younger or unexpected audience.

Categories can overlap. A threatening message may also contain targeted harassment and private information. The system should record the relevant concerns without forcing every case into exactly one box.

Think of policy as four stacked layers.

Layer 1: baseline boundaries

These are rules the organization intends to apply everywhere: for example, never facilitate sexual exploitation of minors, never expose certain private identifiers, and never provide highly actionable assistance for imminent violence. Baseline rules provide a minimum floor across products.

Layer 2: product policy

The product’s purpose determines what it needs to permit. A crime-fiction editor, school tutor, and multiplayer game have different legitimate uses. Product policy defines allowed and disallowed behavior for that setting. It should describe the user experience as well as the category: block a request, limit detail, add a warning, reduce distribution, or route it for review.

Layer 3: audience and region

Age, language, culture, access mode, and local requirements affect risk. A child-directed service needs tighter defaults and age-appropriate explanations. Regional law may require specific reporting or takedown procedures. These differences should be explicit policy choices, not improvised assumptions made by individual moderators.

Layer 4: case context

The final layer considers intent, target, immediacy, detail, and framing. A request for prevention guidance differs from a request to carry out harm. A documentary quote differs from an endorsement. A vague fictional statement differs from a specific, credible threat. Context does not excuse every case; it helps apply the rule accurately.

When evaluating a case, ask:

1. Who is affected? Is there an identifiable target or vulnerable audience?
2. What is the user trying to accomplish? Education, prevention, support, entertainment, persuasion, or harmful action?
3. How actionable is the requested output? General background differs from optimized steps, exact quantities, or evasion advice.
4. How immediate is the risk? Is the situation hypothetical, ongoing, or urgent?
5. How is the content framed? Reporting and condemning abuse differs from praising it.
6. How will it be delivered? Private response, public post, recommendation, or real-world action?

No single dimension settles every case. Intent can be unclear or falsely stated. Evaluate the content’s likely effect as well as the user’s claimed purpose.

A policy needs an action ladder. One useful ladder is:

• Allow: provide the normal response.
• Allow safely: answer the legitimate part while limiting risky detail.
• Add friction: warn, require confirmation, restrict sharing, or reduce recommendation.
• Transform: redact private data, blur imagery, or produce a non-graphic summary.
• Escalate: send ambiguous, severe, or legally sensitive cases to trained reviewers.
• Refuse or remove: do not generate, publish, or retain the prohibited content.
• Emergency process: use a separately governed path for credible imminent-risk signals.

An emergency path needs legal, privacy, and operational review. A model’s uncertain guess should not automatically contact authorities or expose a user’s conversation.

Examples teach more than slogans. For each rule, include an allowed case, a disallowed case, and a borderline case with the reason. Keep examples practical and non-graphic. Update them when appeals, incidents, or new product features reveal confusion.

Avoid a “topic equals ban” structure. A rule such as “block violence” would suppress news, history, prevention, and victim support. A better rule distinguishes depiction, endorsement, threat, operational assistance, educational context, and audience.