Guardrails in code
Measure whether the pre/post guardrail pipeline works
Page 4 turns “it ran” into executable checks for the pre/post output guardrail pipeline.
1Learn the idea
Read
Make the metric executable
Translate the claim into assertions or a tiny eval harness. The metric to protect is: block rate, false-block samples, escape rate on red-team set. Always record the denominator (how many cases) beside any rate. A percentage without a denominator is marketing, not measurement.
Read
Run the checks
cases=[('exfil key', 'block'),('normal faq','allow')]
assert cases[0][1]=='block' and cases[1][1]=='allow'
print('guardrail labels ok')
Expected evidence: guardrail labels ok. A passing assertion proves only the behavior it names; broader usefulness still needs the chapter’s full limits.
Read
Say what the metric does not prove
Be explicit: beating the baseline (model call with no guards on the same fixtures) on this fixture does not prove behavior under post-guard only that still bills the model for blocked intents, or regex gaps on obfuscation. Label observations separately from conclusions so the next page inherits honest evidence about the pre/post guardrail pipeline.
Read
Lab notebook: denominator discipline
Compute block rate, false-block samples, escape rate on red-team set with the denominator written beside the rate every time. For this chapter, the evaluation set is intentionally tiny; that is allowed only if you say so in the evidence. Compare against model call with no guards on the same fixtures before celebrating.
Add one negative case aimed at post-guard only that still bills the model for blocked intents, or regex gaps on obfuscation. A suite with only happy cases cannot protect the pre/post guardrail pipeline when the characteristic failure appears in review.
Read
Worked judgment
If a check is expensive or flaky, shrink it until it is deterministic on policy with PII and self-harm categories + sample prompts. Flaky green builds teach the team to ignore gates. Record what this page does not prove so security-ops and mastery-ship inherit honest limits.
Read
Why this stage matters for the pre/post guardrail pipeline
At the evaluation stage for guardrails-in-code, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about policy with PII and self-harm categories + sample prompts that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: model call with no guards on the same fixtures.
For this page specifically, success looks like metrics with explicit denominators and a negative case while still centering the user decision to block unsafe prompts and strip or refuse unsafe completions before they reach users. If you cannot point to a file, command, or assertion that proves that for the pre/post guardrail pipeline, stay on this page instead of advancing.
Go deeper
Before you start
Why this matters
Write one independent check that would catch a fake pass for this lab. Prefer a check tied to block rate, false-block samples, escape rate on red-team set over a check that only asserts “no exception.”
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.