Content safety basics
Designing safer prompts and products
A safety prompt can guide behavior, but durable safety comes from combining clear instructions with constrained capabilities and recoverable product flows.
Before you start
Why this matters
Begin with the legitimate job
“Never produce harmful content” is too broad to guide a system. It does not define harm, recognize permitted education, or explain what to do when a request has both safe and unsafe parts. A better design begins with the product’s legitimate job.
For a study assistant, that job might be: explain curriculum concepts for secondary-school learners using age-appropriate language, support critical discussion of difficult subjects, avoid highly actionable harmful assistance, and redirect boundary cases toward prevention or fictionalized examples.
That statement gives safety a purpose. It prevents the system from becoming a refusal machine while defining why some details are outside scope.
1Learn the idea
Read
Write prompts as decision guidance
A useful system instruction contains:
- Role and audience: Who the assistant serves and in what setting.
- Allowed goals: The sensitive but legitimate help it should preserve.
- Boundaries: The kinds of assistance it must not provide.
- Context rules: How to treat education, quotation, transformation, fiction, and support-seeking.
- Response behavior: Whether to answer, limit detail, redirect, refuse, or escalate.
- Style: Calm, non-accusatory language and concise explanations.
- Uncertainty: What to do when context or policy is unclear.
For example:
Help learners understand safety-related topics at a high level. Permit prevention, historical analysis, and recovery guidance. Do not provide steps that materially enable harm. When a request mixes safe and unsafe goals, briefly decline the unsafe detail and continue with a safer alternative. Do not claim certainty about a user’s intent.
This example is still only one control. A user can pressure, reframe, or confuse a model, and the model can apply instructions inconsistently.
Read
Design useful refusals
A good refusal has three parts:
- a brief boundary without unnecessary policy jargon;
- no reproduction or expansion of the risky material;
- a relevant safer alternative.
“I can’t help optimize a real break-in, but I can help create a fictional security obstacle or discuss general museum-security ethics” preserves the creative goal. Long moral lectures can reveal filter details, frustrate benign users, and invite argument. Accusatory language such as “You are malicious” claims knowledge the system does not have.
For urgent support contexts, a standard refusal may be inappropriate. Use a reviewed flow that responds with care, avoids graphic detail, and offers suitable support resources.
Read
Put hard limits outside the prompt
Prompts should not carry authority they cannot enforce. If an assistant may issue refunds up to twenty dollars, the payment service must enforce that limit. If minors cannot receive direct messages, permissions must block the route. If a public post needs review, publishing must wait for the review state.
Strong product controls include:
- minimum tool permissions and explicit allowlists;
- limits on amounts, recipients, rate, and reach;
- structured outputs validated before execution;
- separation between drafting and acting;
- retrieval restricted to authorized sources;
- personal-data redaction and retention limits;
- human approval for severe or ambiguous actions;
- a kill switch and rollback path.
The model can propose an action; ordinary code decides whether that action is allowed.
Read
Add friction where it changes behavior
Not every safety issue needs a block. Product friction can reduce accidental harm:
- preview before public posting;
- confirmation before sharing sensitive information;
- warning before opening mature content;
- blur or click-through for potentially disturbing imagery;
- a short delay before high-reach distribution;
- limits on repeated unwanted contact;
- easy blocking, muting, and reporting;
- lower recommendation reach for uncertain content pending review.
Friction has costs. Too many warnings train users to ignore them. Test whether a control helps people make better decisions, not merely whether it adds a screen.
Read
Make reports and appeals usable
Reporting should let a person identify the content, category, and immediate safety need without completing a long investigation. Protect reporters from retaliation and unnecessary exposure. For high-volume systems, prioritize by severity and reach while preserving a path for less obvious harm.
Appeals are especially important when moderation affects account access, income, education, or expression. Show the broad reason, preserve the user’s work when safe, and avoid having the same automated rule “review” itself. Track reversals by category and group to discover systematic errors.
Read
Plan the operational loop
Before launch, assign owners for policy, engineering controls, reviewer guidance, incidents, and user communication. Version policies and prompts. Evaluate every meaningful model, filter, or threshold update. Monitor false positives, false negatives, appeals, queue age, repeated abuse, and safety incidents.
Create safe defaults for outages. If a moderation dependency fails, a private low-risk feature might degrade gracefully, while public posting or tool execution pauses. Document who can disable a feature and how affected users are notified.
Read
A compact design review
For a new feature, ask:
- What legitimate sensitive uses must work?
- Which harms are plausible at this audience, scale, and surface?
- Which boundaries belong in policy, prompt, filter, permissions, or human review?
- What happens when controls disagree or fail?
- How can users recover from a wrong decision?
- Which tests prove the feature remains helpful as well as safe?
Checking tutor…