AnyoneLearnAI
Start learning
Chapter DEval metrics labPage 7 of 8

Eval metrics lab

Cover security and operational gates

Eval metrics lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minSecurity and ops

Before you start

Why this matters

Read this incident aloud: a canary prompt improves median latency but quietly reduces grounded answers for billing questions. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: telemetry excludes prompt text and customer identifiers; only pseudonymous request IDs are retained.

1Learn the idea

Read

Enforce security and operational boundaries

Treat every external string, retrieved document, provider response, and generated tool argument as untrusted. Authentication establishes who called; authorization decides which resource and action are allowed; schema validation only establishes shape. Keep those checks separate. The governing boundary here is: telemetry excludes prompt text and customer identifiers; only pseudonymous request IDs are retained. Enforce it in deterministic code outside the prompt and again at the action adapter.

Use least-privilege credentials for staging and production. Deny outbound network access in adversarial tests. Replace secrets with canaries that are useless outside the fixture. Redact before serialization so sensitive values cannot leak through logs, traces, exception messages, or dead-letter queues. Test redaction with representative email, token, and customer-ID patterns, including nested fields.

Operational security also means controlling change. Version policies, fixtures, model configuration, and allowlists. Require review for boundary changes. Set retention and deletion rules for evidence. Rate-limit abusive sources without making the model responsible for identity. During an incident, stop canary promotion, compare retriever spans, and roll back only if the quality breach persists. Preserve safe metadata for investigation, rotate credentials only when evidence indicates exposure, and do not copy raw payloads into tickets.

For security and operations, write an abuse case and a credential/permission check beside the normal test. Exercise denial, evidence retention, and break-glass review without granting the model extra authority.

Read

Focused implementation artifact

DENIED_KEYS = {"api_key", "authorization", "password", "raw_prompt"}

def safe_attributes(values):
    return {k: ("[REDACTED]" if k in DENIED_KEYS else v)
            for k, v in values.items()}

def authorize(session_subject, requested_subject):
    if session_subject != requested_subject:
        raise PermissionError("subject mismatch")

def test_no_sensitive_evidence_exported(exported_span):
    assert DENIED_KEYS.isdisjoint(exported_span.attributes)

Read

Test the boundary as an attacker

Start from the benign fixture and mutate one trust boundary at a time: identity, tool name, arguments, retrieved instruction, callback, or exported telemetry. Each mutation must be denied by deterministic code, not by asking the model to be careful. Assert the denial reason and zero forbidden side effects. The named abuse outcome is an average-latency chart stays green while the p95 and billing groundedness collapse.

Run the suite with fake canary secrets and outbound network denied. Search captured logs, spans, errors, and dead-letter payloads for the canary and representative PII. A clean response body is insufficient if an exporter leaked the value. Verify least-privilege credentials and ensure staging workers cannot write production state.

Operationalize the boundary: version policy and allowlists, require review for authority changes, set evidence retention, and document break-glass access. Track rag_grounded_success_ratio without attacker-controlled labels. Security acceptance is not an average; the release still must satisfy quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0. On evidence of a breach, stop canary promotion, compare retriever spans, and roll back only if the quality breach persists.

Checking tutor…

Continue learning · glossary & guides