Trace a worked example

Goal: improve the invoice-extraction service using model selection without breaking protected slices.

Mechanism reminder for model selection: Define task cases and hard gates; score candidates on quality, latency, cost, context needs, tool/schema support, and vendor constraints; keep a fallback model.

Baseline and shock: If model L hits 96% field-F1 at $0.12/1k docs and model H hits 97% at $0.55/1k, the +1 pt may lose unless errors are extremely costly.

Tradeoff in play for model selection: Larger models often raise quality and cost/latency together. Tiny models need more scaffolding. Switching vendors can win price and lose schema features.

Ship decision: Choose model L with schema constraints and a line-item sum check; keep H as fallback for low-confidence docs only.

Rollback triggers for model selection must cite field-level F1, schema-valid rate, $/1k docs, p95 latency, truncation rate, fallback success. If you cannot name a tolerated regression on the invoice-extraction service, do not promote the change.

Keep the model selection decision record beside the invoice-extraction service code paths that implement it. Future you will not remember why a default exists unless the evidence is linked from the config for model selection.

Also pin one numeric memory from this model selection chapter: If model L hits 96% field-F1 at $0.12/1k docs and model H hits 97% at $0.55/1k, the +1 pt may lose unless errors are extremely costly. That number is not decoration; it is a template for how claims about model selection on the invoice-extraction service should look in design docs. Scoped specifically to model selection / invoice-extraction service / worked-trace.

People confuse model selection with neighboring buzzwords when debugging the invoice-extraction service. Before changing prompts, ask whether the broken stage was evidence gathering, the model selection judgment itself, validation, or the product action. Fixing the wrong stage creates folklore (“we tried model selection and it failed”) that blocks the next team on the invoice-extraction service. Scoped specifically to model selection / invoice-extraction service / worked-trace.

Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to choosing a model rather than generic AI advice.

Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to choosing a model rather than generic AI advice.

Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to choosing a model rather than generic AI advice.

For choosing a model, add acceptance test 1: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 2: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 3: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 4: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 5: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 6: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 7: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 8: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 9: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 10: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 11: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 12: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 13: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 14: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 15: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 16: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 17: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 18: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 19: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 20: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 21: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 22: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 23: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 24: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 25: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 26: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 27: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 28: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 29: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 30: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.

For choosing a model, add acceptance test 31: evidence source, threshold, and signer. Keep it unique to this chapter's scenario.