AnyoneLearnAI
Start learning
Chapter DCost optimization labPage 5 of 8

Cost optimization lab

Handle failures and retries

Cost optimization lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minFailure handling

Before you start

Why this matters

Read this incident aloud: duplicate shipping questions consume flagship-model tokens even though a cached small-model answer passes quality checks. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: cache keys exclude raw identity and sensitive text; regulated intents always use the approved model path.

1Learn the idea

Read

Reproduce failures and debug safely

Reproduce cost per call falls after routing, but retries increase and cost per successful answer rises without editing the prompt first. Load the frozen fixture, set a known release, replace network calls with the failing response, and capture one trace. Confirm that the failure happens twice. If it does not, the reproduction is not controlled enough to support a fix. Debug in transaction order: input acceptance, normalization, retrieval or routing, model/tool decision, output validation, then side effects.

Classify the fault before retrying. Timeouts, 429s, and temporary 5xx responses may be retryable with capped exponential backoff and jitter. Schema violations, authorization mismatches, unsafe tool requests, and failed quality assertions are not transient; retrying repeats risk and cost. Use a maximum attempt count and a deadline. For side effects, retry only behind an idempotency key or transactional outbox.

A fix is complete only when the reproduction becomes a permanent regression case. Add a negative assertion so the old unsafe or incorrect behavior cannot return silently. Preserve the trace ID and policy version in the test output, but redact payloads according to the same production policy. The operational response is to promote routing by intent, retain a holdout, and automatically disable it if quality or retries breach budget.

For failure handling, start with the named reproduction and add controlled timeout, malformed output, duplicate delivery, and forbidden-action variants where relevant. Demonstrate that retries cannot multiply side effects.

Read

Focused implementation artifact

def execute_with_policy(operation, *, attempts=3):
    for attempt in range(attempts):
        try:
            return operation()
        except (TimeoutError, ConnectionError):
            if attempt == attempts - 1:
                raise
        except (PermissionError, ValueError):
            raise  # deterministic or unsafe: never retry

def test_known_failure_is_contained():
    case = {"intent":"shipping_faq","model":"flagship","input_tokens":2100,"output_tokens":180,"cache_hit":false,"latency_ms":2400,"quality_pass":true,"price_version":"2026-07"}
    choice = router.choose(intent, complexity=0.18, cache_eligible=True)
    assert report.cost_per_success <= baseline * 0.75 and report.quality_delta >= -0.01

Read

Diagnose before retrying

Run the named reproduction—cost per call falls after routing, but retries increase and cost per successful answer rises—with a frozen clock and recording adapters. Capture the ordered calls and one trace. Re-run it to prove determinism, then locate the first stage whose actual output differs from its contract. Change only that stage. Prompt tuning is not a substitute for an authorization, idempotency, schema, or accounting fix.

Inject a timeout before any side effect and another after the dependency reports success. The first may be retried under a deadline; the second requires reconciliation because blind retry could duplicate work. Prove attempt count, backoff cap, and final error classification. Deterministic policy failures must make one attempt. Side effects require an idempotency key or transactional outbox.

Turn the reproduction into a permanent test and assert the old behavior is absent. Recompute cost per successful answer under retries so hidden attempts do not improve the denominator. Emit llm_cost_per_success_usd with error_type from a bounded enum. If containment fails, promote routing by intent, retain a holdout, and automatically disable it if quality or retries breach budget.

Checking tutor…

Continue learning · glossary & guides