Choosing and using models

Write an operational task statement:

Given a support message in English or Spanish, return one of five queue IDs and needs_review: true when the request is ambiguous.

This is testable. “Use AI for support” is not. Capture:

• representative inputs and expected outputs;
• required languages, modalities, and context length;
• acceptable quality and failure behavior;
• latency targets and traffic volume;
• privacy, residency, and compliance constraints;
• the consequence of a wrong answer;
• which decisions require human review.

Separate must-haves from preferences. JSON output may be mandatory for integration. A conversational style may be irrelevant. A model that fails one hard privacy requirement is not a candidate even if its benchmark score is highest.

Useful selection dimensions include:

Task quality. Measure accuracy, factual coverage, instruction following, code correctness, or another task-specific outcome. General benchmarks are clues, not final evidence.

Latency and throughput. Time to first token matters in chat; total processing time matters in batch jobs. Rate limits and concurrency can dominate model speed.

Cost. Include input and output tokens, retries, retrieval, tool calls, evaluation, and human review. A cheap model with frequent failures may cost more overall.

Context and output limits. Ensure the model can accept the required material, but do not assume filling the entire window yields good attention.

Deployment and privacy. Consider provider data handling, retention, region, encryption, access controls, and whether hosted, dedicated, or local deployment is required.

Features. Structured output, tool calling, vision, streaming, fine-tuning, and batch processing may simplify the system.

Operational fit. Version stability, observability, quotas, support, and fallback options matter after the demo.

No model wins every dimension. A fast small model may classify routine messages, while a larger model handles ambiguous cases.

Create a small but meaningful evaluation set before tuning prompts around favorite examples. Include:

• common cases in realistic proportions;
• difficult edge cases;
• malformed or incomplete inputs;
• adversarial or conflicting instructions;
• examples where abstaining is correct;
• languages and formats seen in production.

Define scoring rules. Exact match works for a queue ID. A summary may require a rubric for factual coverage and unsupported claims. Code can be tested. Human review should use clear criteria and, where possible, hide model identity to reduce bias.

Run every candidate with the same task conditions. Track quality, latency, token use, error rates, and refusal behavior. Inspect failures, not only averages. A 95% score can hide catastrophic mistakes concentrated in one important category.

Keep a held-out set for the final comparison. Repeatedly editing the prompt against all examples can overfit the evaluation just as surely as repeatedly editing conventional code against a visible test suite.

One model does not have to handle every case. A practical system might:

1. Use deterministic rules for obvious, exact cases.
2. Send routine language cases to a smaller model.
3. Escalate low-confidence or policy-sensitive cases to a stronger model.
4. Route consequential or unresolved cases to a human.

Confidence needs careful definition. Self-reported certainty is weak. Better routing signals may include missing fields, disagreement between repeated checks, unsupported citations, out-of-distribution inputs, or a separate calibrated classifier.

Fallbacks should preserve safety. If the model provider is unavailable, the system might queue the job, show a limited manual workflow, or use a tested alternate model. Silently switching to a weaker model for high-impact decisions can violate the original quality boundary.

After selecting a model, design the request as a stable interface. Put durable behavior in system-level instructions, supply only relevant context, separate untrusted content from commands, and define an output contract.

Use examples where they resolve genuine ambiguity. Too many examples consume context and may introduce contradictions. Retrieval should return focused passages with source identifiers rather than an entire document library. Tools should have narrow permissions and validated arguments.

For exact or current facts, prefer authoritative systems:

• calculate totals with code or a calculator;
• query balances from a system of record;
• retrieve current policy text;
• validate generated JSON against a schema;
• run code tests before accepting generated changes.

The LLM is often best used as the language and pattern layer around these components, not as their replacement.

Production data changes. Users discover new ways to phrase requests. Providers update models. Costs and rate limits change. A one-time evaluation is not enough.

Log enough to diagnose failures while minimizing sensitive data. Track model and prompt versions, latency, token counts, tool outcomes, validation failures, and user corrections. Sample outputs according to risk and privacy policy. Build regression cases from confirmed incidents.

Set release criteria before changing a model or prompt. A new version should pass the existing evaluation and targeted tests for its claimed improvement. Roll out gradually when the task matters, monitor key metrics, and preserve a rollback path.

Model choice is reversible only if the surrounding interface is portable. Avoid depending unnecessarily on one provider’s undocumented quirks. At the same time, do not force every model into the lowest common denominator when a valuable feature has strong tests and a fallback.

For each use case, record:

• task and owner;
• selected model and version policy;
• evaluation set and pass thresholds;
• expected cost and latency;
• data-handling constraints;
• known failure modes;
• validation and review steps;
• fallback and rollback behavior.

This short record turns “we tried it and it looked good” into an inspectable decision. It also makes future comparisons faster because the next model is judged against stable requirements.