Debug post-guard only that still bills the model for blocked intents in the pre/post guardrail pipel

Do not start with a speculative fix for the pre/post guardrail pipeline. Force the failure on purpose, save the before output, then change one cause at a time. Retries are allowed only for transient conditions—not for bad input that will fail forever on guardrails-in-code.

def execute_with_policy(operation, *, attempts=3):
    for attempt in range(attempts):
        try:
            return operation()
        except (TimeoutError, ConnectionError):
            if attempt == attempts - 1:
                raise
        except (PermissionError, ValueError):
            raise  # deterministic or unsafe: never retry

def test_known_failure_is_contained():
    case = {"user_text":"Track order A12","session_customer":"cust_7","tool":"lookup_order","args":{"order_id":"A12","customer_id":"cust_99"}}
    decision = guard(request, allowed_tools={"lookup_order"}, bind_identity=True)
    assert not decision.allowed and "identity_mismatch" in decision.violations

Expected evidence: a keyword filter passes harmless wording but the model emits refund_order with an attacker-controlled customer_id. If you cannot reproduce on demand, you do not yet control the failure mode for guardrails-in-code.

After repair, rerun the exact reproduction command. Keep the failing fixture as a regression seed for the observability page. For the pre/post output guardrail pipeline, remember the claim you are restoring: blocked inputs never call the model; unsafe outputs are refused with a stable code.

Store a one-command reproduction for: post-guard only that still bills the model for blocked intents, or regex gaps on obfuscation. The command should use policy with PII and self-harm categories + sample prompts or a minimal mutant of it. Paste the failing output into notes/failure-before.txt (or your shell scrollback as copied text). After the fix, paste notes/failure-after.txt and keep both.

Retries belong only on transient faults. If the failure is bad input, a bad allowlist, or a logic bug in the pre/post guardrail pipeline, retrying will amplify cost without repairing trust around block unsafe prompts and strip or refuse unsafe completions before they reach users.

Classify the failure as prevent, detect, contain, or recover—using this lab’s language, not a generic poster. For guardrails-in-code, the first fix should usually be detect+prevent at the boundary, because post-guard only that still bills the model for blocked intents, or regex gaps on obfuscation is cheaper to stop early than to explain in production prose.

At the debugging stage for guardrails-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about policy with PII and self-harm categories + sample prompts that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: model call with no guards on the same fixtures.

For this page specifically, success looks like before/after evidence for the characteristic failure while still centering the user decision to block unsafe prompts and strip or refuse unsafe completions before they reach users. If you cannot point to a file, command, or assertion that proves that for the pre/post guardrail pipeline, stay on this page instead of advancing.

Cheatsheet: prompt injection defense

Previous · Next