AnyoneLearnAI
Start learning
Chapter DCost optimization labPage 7 of 8

Cost optimization lab

Cover security and operational gates

Cost optimization lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minSecurity and ops

Before you start

Why this matters

Read this incident aloud: duplicate shipping questions consume flagship-model tokens even though a cached small-model answer passes quality checks. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: cache keys exclude raw identity and sensitive text; regulated intents always use the approved model path.

1Learn the idea

Read

Enforce security and operational boundaries

Treat every external string, retrieved document, provider response, and generated tool argument as untrusted. Authentication establishes who called; authorization decides which resource and action are allowed; schema validation only establishes shape. Keep those checks separate. The governing boundary here is: cache keys exclude raw identity and sensitive text; regulated intents always use the approved model path. Enforce it in deterministic code outside the prompt and again at the action adapter.

Use least-privilege credentials for staging and production. Deny outbound network access in adversarial tests. Replace secrets with canaries that are useless outside the fixture. Redact before serialization so sensitive values cannot leak through logs, traces, exception messages, or dead-letter queues. Test redaction with representative email, token, and customer-ID patterns, including nested fields.

Operational security also means controlling change. Version policies, fixtures, model configuration, and allowlists. Require review for boundary changes. Set retention and deletion rules for evidence. Rate-limit abusive sources without making the model responsible for identity. During an incident, promote routing by intent, retain a holdout, and automatically disable it if quality or retries breach budget. Preserve safe metadata for investigation, rotate credentials only when evidence indicates exposure, and do not copy raw payloads into tickets.

For security and operations, write an abuse case and a credential/permission check beside the normal test. Exercise denial, evidence retention, and break-glass review without granting the model extra authority.

Read

Focused implementation artifact

DENIED_KEYS = {"api_key", "authorization", "password", "raw_prompt"}

def safe_attributes(values):
    return {k: ("[REDACTED]" if k in DENIED_KEYS else v)
            for k, v in values.items()}

def authorize(session_subject, requested_subject):
    if session_subject != requested_subject:
        raise PermissionError("subject mismatch")

def test_no_sensitive_evidence_exported(exported_span):
    assert DENIED_KEYS.isdisjoint(exported_span.attributes)

Read

Test the boundary as an attacker

Start from the benign fixture and mutate one trust boundary at a time: identity, tool name, arguments, retrieved instruction, callback, or exported telemetry. Each mutation must be denied by deterministic code, not by asking the model to be careful. Assert the denial reason and zero forbidden side effects. The named abuse outcome is cost per call falls after routing, but retries increase and cost per successful answer rises.

Run the suite with fake canary secrets and outbound network denied. Search captured logs, spans, errors, and dead-letter payloads for the canary and representative PII. A clean response body is insufficient if an exporter leaked the value. Verify least-privilege credentials and ensure staging workers cannot write production state.

Operationalize the boundary: version policy and allowlists, require review for authority changes, set evidence retention, and document break-glass access. Track llm_cost_per_success_usd without attacker-controlled labels. Security acceptance is not an average; the release still must satisfy reduce cost by 25% while quality drops no more than 1 percentage point. On evidence of a breach, promote routing by intent, retain a holdout, and automatically disable it if quality or retries breach budget.

Checking tutor…

Continue learning · glossary & guides