Mastery: connect the pieces

1. Idea — The best model is the smallest, cheapest, fastest option that meets the task contract on your evals—not the leaderboard winner in the abstract.…
2. Mechanism — Define task cases and hard gates; score candidates on quality, latency, cost, context needs, tool/schema support, and vendor constraints; keep a fallback model.…
3. Controls — quality threshold, p95 latency budget, cost ceiling, context length need, structured-output support, data residency, fallback model
4. Tradeoff — Larger models often raise quality and cost/latency together. Tiny models need more scaffolding. Switching vendors can win price and lose schema features.…
5. Failures — Leaderboard shopping; Hidden context limits
6. Metrics — field-level F1, schema-valid rate, $/1k docs, p95 latency, truncation rate, fallback success
7. Ship rule — Choose model L with schema constraints and a line-item sum check; keep H as fallback for low-confidence docs only.

Neighboring layers (retrieval, serving, policy, human review) still own what model selection cannot on the invoice-extraction service. Do not ask this chapter’s dial to replace permissions or source truth.

Mastery is transfer: take model selection into a second scenario that is not the invoice-extraction service and rebuild the checklist without copying sentences. If you can only recite this chapter’s examples for model selection, you have memorized a story, not a model.

Also pin one numeric memory from this model selection chapter: If model L hits 96% field-F1 at $0.12/1k docs and model H hits 97% at $0.55/1k, the +1 pt may lose unless errors are extremely costly. That number is not decoration; it is a template for how claims about model selection on the invoice-extraction service should look in design docs. Scoped specifically to model selection / invoice-extraction service / mastery-connect.

People confuse model selection with neighboring buzzwords when debugging the invoice-extraction service. Before changing prompts, ask whether the broken stage was evidence gathering, the model selection judgment itself, validation, or the product action. Fixing the wrong stage creates folklore (“we tried model selection and it failed”) that blocks the next team on the invoice-extraction service. Scoped specifically to model selection / invoice-extraction service / mastery-connect.

Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to choosing a model rather than generic AI advice.

Write a five-line artifact for this page: goal, inputs, check, owner, stop rule. Invent one fluent failure that the check would catch. Keep details specific to choosing a model rather than generic AI advice.