Prompt injection & AI security
Anticipate failure modes
Name failures by their mechanism in prompt injection on the tool-using support agent, not with a generic hallucination label.
1Learn the idea
Read
Response design
See it
Confidence is a tone — verify before you act
For each severe prompt injection failure on the tool-using support agent, define stop condition, safe state, owner, and lasting prevention. Rollback only works if prior prompts, indexes, and models remain available. “Send to a human” needs queue capacity and context—not just a button name.
Run one tabletop on the tool-using support agent for prompt injection: inject a defect, verify detection, contain, recover, and keep the blameless trace.
Read
Make it operational
After the tabletop, store the injected prompt injection defect for the tool-using support agent as a regression fixture. If the same failure later reaches users silently, your detection story was aspirational. Detection without a fixture tends to rot for prompt injection.
Also pin one numeric memory from this prompt injection chapter: risk ≈ probability of successful injection × impact of available capability; reducing tool privilege cuts impact even when detection is imperfect That number is not decoration; it is a template for how claims about prompt injection on the tool-using support agent should look in design docs. Scoped specifically to prompt injection / tool-using support agent / failure-modes.
Read
Common mix-ups
People confuse prompt injection with neighboring buzzwords when debugging the tool-using support agent. Before changing prompts, ask whether the broken stage was evidence gathering, the prompt injection judgment itself, validation, or the product action. Fixing the wrong stage creates folklore (“we tried prompt injection and it failed”) that blocks the next team on the tool-using support agent. Scoped specifically to prompt injection / tool-using support agent / failure-modes.
Read
Rehearsal (prompt-injection/failure-modes)
Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to prompt injection rather than generic AI advice.
Read
Rehearsal (prompt-injection/failure-modes)
Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to prompt injection rather than generic AI advice.
Read
Rehearsal (prompt-injection/failure-modes)
Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to prompt injection rather than generic AI advice.
Go deeper
Before you start
Why this matters
Invent an incident for the tool-using support agent involving prompt injection. What earliest signal should fire before users complain?
Indirect injection via ticket
Detect with retrieved text says ignore policy. Respond by treat retrieved text as data; delimit.
Tool exfiltration
Detect with agent emails secrets. Respond by block sensitive tools; DLP on egress.
Delimiter breakout
Detect with user fakes system tags. Respond by don’t parse trust from plain text alone.
Approval fatigue
Detect with humans auto-approve. Respond by risk-base which actions need humans.
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.