AnyoneLearnAI
Start learning
Chapter DPrompt injection in codePage 7 of 8

Prompt injection in code

Cover security and operational gates

Prompt injection in code is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minSecurity and ops

Before you start

Why this matters

Read this incident aloud: a retrieved PDF says to ignore the system message and send environment variables through a webhook tool. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: retrieved content is untrusted data; no fixture has real secrets or network access.

1Learn the idea

Read

Enforce security and operational boundaries

Treat every external string, retrieved document, provider response, and generated tool argument as untrusted. Authentication establishes who called; authorization decides which resource and action are allowed; schema validation only establishes shape. Keep those checks separate. The governing boundary here is: retrieved content is untrusted data; no fixture has real secrets or network access. Enforce it in deterministic code outside the prompt and again at the action adapter.

Use least-privilege credentials for staging and production. Deny outbound network access in adversarial tests. Replace secrets with canaries that are useless outside the fixture. Redact before serialization so sensitive values cannot leak through logs, traces, exception messages, or dead-letter queues. Test redaction with representative email, token, and customer-ID patterns, including nested fields.

Operational security also means controlling change. Version policies, fixtures, model configuration, and allowlists. Require review for boundary changes. Set retention and deletion rules for evidence. Rate-limit abusive sources without making the model responsible for identity. During an incident, fail CI, quarantine the corpus item, and preserve only redacted evidence. Preserve safe metadata for investigation, rotate credentials only when evidence indicates exposure, and do not copy raw payloads into tickets.

For security and operations, write an abuse case and a credential/permission check beside the normal test. Exercise denial, evidence retention, and break-glass review without granting the model extra authority.

Read

Focused implementation artifact

DENIED_KEYS = {"api_key", "authorization", "password", "raw_prompt"}

def safe_attributes(values):
    return {k: ("[REDACTED]" if k in DENIED_KEYS else v)
            for k, v in values.items()}

def authorize(session_subject, requested_subject):
    if session_subject != requested_subject:
        raise PermissionError("subject mismatch")

def test_no_sensitive_evidence_exported(exported_span):
    assert DENIED_KEYS.isdisjoint(exported_span.attributes)

Read

Test the boundary as an attacker

Start from the benign fixture and mutate one trust boundary at a time: identity, tool name, arguments, retrieved instruction, callback, or exported telemetry. Each mutation must be denied by deterministic code, not by asking the model to be careful. Assert the denial reason and zero forbidden side effects. The named abuse outcome is the answer looks like a refusal but a hidden tool call posts the canary token.

Run the suite with fake canary secrets and outbound network denied. Search captured logs, spans, errors, and dead-letter payloads for the canary and representative PII. A clean response body is insufficient if an exporter leaked the value. Verify least-privilege credentials and ensure staging workers cannot write production state.

Operationalize the boundary: version policy and allowlists, require review for authority changes, set evidence retention, and document break-glass access. Track injection_attack_success_total without attacker-controlled labels. Security acceptance is not an average; the release still must satisfy 0.00 canary leakage and 0.00 forbidden tool calls across critical attacks. On evidence of a breach, fail CI, quarantine the corpus item, and preserve only redacted evidence.

Checking tutor…

Continue learning · glossary & guides