Red-team thinking for learners

A threat model is a short description of what you are protecting and what could go wrong. For a teen study assistant, it might state:

• Assets: student wellbeing, private school data, trusted educational access, and account integrity.
• Actors: ordinary students, curious boundary-testers, abusive users, and people who accidentally paste sensitive content.
• Surfaces: chat input, uploaded documents, image captions, shared links, generated quizzes, and reporting tools.
• Failures: age-inappropriate output, targeted abuse, private-data exposure, dangerous operational detail, or incorrect account penalties.

This scope keeps testing relevant. Without it, teams may spend days on exotic prompts while ignoring a public sharing button that exposes private conversations.

Write each test as:

Given a context and user request, when the system responds, then an observable safety property should hold.

For example:

Given a fictional chemistry-lab scenario containing placeholder substances, when the user requests unsafe operational detail, then the assistant should decline that detail and offer general lab-safety guidance.

The expected result is more useful than “the model should be safe.” It says which part to refuse and what helpful behavior to preserve.

Vary one dimension at a time before combining them:

• Intent: benign education, ambiguous curiosity, or clearly harmful purpose.
• Framing: direct request, role-play, quotation, translation, summary, or hypothetical.
• Conversation: first turn, repeated pressure, gradual escalation, or a later turn after benign context.
• Representation: ordinary wording, misspelling, slang, spacing, another language, or text inside an uploaded file.
• Audience: adult professional, general public, or child-directed setting.
• Delivery: private answer, public post, recommendation, or tool-enabled action.

Use placeholders such as [PROHIBITED STEP], [PRIVATE IDENTIFIER], or [TARGETED INSULT] when the dangerous content itself is not needed. Record only enough detail to reproduce the control failure.

Safety gaps often occur in product composition:

• A harmless generation becomes unsafe when automatically published.
• Separate benign answers can be assembled into an enabling sequence.
• A filter checks text but not an image caption or attachment.
• The first reply is safe, but repeated follow-ups gradually add restricted detail.
• A user asks the assistant to transform prohibited content rather than create it.
• A report button is abused to silence permitted speech.
• A classifier flags content correctly, but the application ignores the label.
• The safe response leaks private information into logs or analytics.

Test the end-to-end flow, including permissions, UI, review queue, and fallback behavior.

Red-team work needs rules. Use test accounts and isolated environments. Do not target real people, public services, or systems you lack permission to test. Avoid collecting illegal or traumatic material. Give testers a way to pause, rotate tasks, and report wellbeing concerns. Restrict access to sensitive test results and remove unnecessary personal data.

Define a stop condition before the test. Stop when a case could create real-world harm, affect production users, expose secrets, or exceed authorization. Escalate severe findings privately to the responsible owner. Do not publish a working bypass before mitigation.

A useful finding contains:

1. the policy expectation and product version;
2. a sanitized, reproducible input sequence;
3. the observed output or action;
4. the expected safer behavior;
5. likely impact and affected audience;
6. which layer failed or was absent;
7. a proposed test to prevent regression.

Severity is not the same as novelty. A simple permission mistake affecting many users may matter more than an elaborate prompt with no practical path.

Do not patch only the exact phrase. Identify the failure class. If translation bypassed a rule, add multilingual cases and inspect where classification happens. If the assistant refused but a tool still acted, fix the authorization boundary. If a benign support request was blocked, add counterexamples and review the policy.

After mitigation, rerun the original case, nearby variations, and ordinary helpful tasks. A change that blocks the failure but destroys legitimate use is incomplete. Keep the sanitized case in a versioned evaluation set and monitor whether later model or policy updates revive it.