Instrument the injection regression suite

Instrument the prompt-injection regression suite for a document-grounded agent so a run records enough structure to debug offline: counts, latency if relevant, pass/fail of attack success rate, canary leakage, forbidden tool-call count, and a stable stage name. Redact secrets and raw credentials from every event.

import json
print(json.dumps({'stage':'injection-suite','attack_success':0,'canary':0,'forbidden_tools':0}))

Expected evidence: injection telemetry. Prefer JSON or structured text you can grep in CI over prose logs for prompt-injection-in-code.

Turn one historical failure—especially answer refuses in text but a hidden tool call posts a canary token—into a test that fails if the signal disappears for the injection regression suite. Observability without a failing test is optional decoration; observability with a test is part of the prompt-injection-in-code artifact.

Draft a three-field event for the injection regression suite: stage, ok, and one domain field derived from attack success rate, canary leakage, forbidden tool-call count. Add fixture_id or docs_version when content can change. Explicitly list fields that must never appear (tokens, passwords, raw prompts) because treating retrieved document text as trusted system instructions is in scope for this lab.

Wire one assertion that fails if the injection regression suite event is missing after a run. Observability that cannot fail a test will not survive contact with a busy prompt-injection-in-code repository.

Imagine a teammate opens only your event stream after a bad deploy. Could they tell whether indirect-pdf-07 with IGNORE RULES + CANARY payload was wrong, whether answer refuses in text but a hidden tool call posts a canary token returned, or whether treating retrieved document text as trusted system instructions slipped through? If not, rename fields until those three stories are distinguishable.

At the testing and observability stage for prompt-injection-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about indirect-pdf-07 with IGNORE RULES + CANARY payload that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: agent without document/instruction separation on the same attacks.

For this page specifically, success looks like a structured event schema locked by a test while still centering the user decision to block indirect instructions in retrieved documents from causing forbidden tool calls. If you cannot point to a file, command, or assertion that proves that for the injection regression suite, stay on this page instead of advancing.

How-to: red-team prompt injection · Snippet: injection test fixture · Glossary: adversarial prompt

Previous · Next