Quality and risk checks

Generic “accuracy” rarely maps cleanly to a product experience. Decompose the task into observable dimensions. For a policy-grounded support draft, these might include:

• correct eligibility and escalation;
• retrieval of current authoritative sources;
• claim support and citation correctness;
• preservation of customer facts and uncertainty;
• instruction following and format;
• harmful or prohibited content;
• reviewer effort and acceptance;
• latency, availability, and cost;
• accessibility and workflow fit.

Define the unit of analysis. A response-level pass rate may hide one dangerous sentence. Claim-level support may be necessary. For classification, inspect precision and recall by class, especially rare high-consequence routes. For extraction, check each critical field, not only whether the overall record appears usable.

Evaluation examples should reflect intended scope and difficult boundaries. Include common cases, rare but consequential cases, ambiguous input, incomplete information, conflicting sources, multiple languages or segments in scope, adversarial instructions, stale data, service failure, and accessibility paths.

Keep separate sets for development and final release decisions. If teams repeatedly tune on the same “golden” examples, performance on that set stops estimating unseen behavior. Version datasets, rubrics, prompts, models, retrieval collections, and evaluator guidance.

Human reference labels also contain disagreement. Record reviewer qualifications, adjudication, and uncertain cases. When policy itself is ambiguous, route it to the policy owner rather than forcing annotators or the model to invent certainty.

Synthetic examples can expand edge-case coverage but should supplement, not replace, permitted real-world examples. Generated tests often mirror assumptions in the prompt and miss messy production behavior.

Define release thresholds and pause conditions before stakeholders see an attractive demo. Otherwise, teams may reinterpret weak results to protect the schedule.

Use several gate types:

Task quality: required performance by intent and critical field.
Severe harm: often zero tolerance in the release set, plus a production response plan.
Coverage: the share of cases the feature can safely handle; abstention may be correct.
Operational: latency, availability, source freshness, review capacity, and fallback.
Experience: user comprehension, control, accessibility, and correction effort.
Equity: performance across relevant groups and conditions, with justified minimum sample sizes.
Governance: approvals, documentation, logging, retention, and incident ownership.

A threshold is not a guarantee. It is a decision rule based on available evidence. Document residual risk and what production monitoring can and cannot detect.

Risk does not live only in model output. Map the complete path: input collection, permissions, preprocessing, retrieval, generation, interface, human review, integration, external action, logging, retention, and feedback.

For each failure, estimate:

• affected people and groups;
• consequence and scale;
• reversibility;
• how quickly the failure is detectable;
• likelihood under actual use;
• existing prevention;
• containment and recovery;
• accountable owner.

Examples include privacy leakage, prompt injection, outdated sources, automation bias, discriminatory performance, wrong account actions, inaccessible controls, review overload, vendor outage, cost spikes, and feedback loops that treat unreviewed outputs as truth.

Risk scores can aid sorting, but do not average away catastrophic consequences. Some actions need hard constraints, authorization, dual control, or exclusion regardless of expected-value scoring.

An AI product should know what happens when it cannot complete the task safely. Good failure behavior may include abstaining, showing uncertainty, preserving source evidence, requesting missing information, routing to a specialist, using a deterministic fallback, or returning to the existing workflow.

Avoid fake certainty such as “No issue found” when a dependency failed. Distinguish:

• the system checked and found no issue;
• the system could not complete the check;
• the input was outside scope;
• evidence conflicted;
• a human decision is required.

Test the failure path as a first-class experience. A fallback that takes ten minutes to locate or loses the user’s work is not operationally safe.

Human review is not effective merely because an approval button exists. Reviewers need the original input, relevant sources, changed fields, uncertainty, and consequence. They need time, authority, training, and a way to reject or escalate.

Measure reviewer behavior: acceptance without inspection, correction time, disagreement, queue length, and missed errors. Automation bias can increase as output becomes more fluent. If review volume exceeds capacity, the control degrades even though the process diagram still contains a human box.

Bind approval to the exact output and context reviewed. If a source or account state changes, invalidate approval. Consequential actions should be enforced by permissions and deterministic rules, not by prompt wording.

Use a progression proportionate to risk:

1. offline evaluation;
2. shadow mode with no user impact;
3. suggestion mode for trained users;
4. narrow production scope with monitoring;
5. expansion only after evidence and review.

At each stage, define entry, exit, pause, and rollback conditions. Keep a manual fallback and test the kill switch. Monitor not only model quality but source freshness, input drift, subgroup performance, override patterns, queue effects, incidents, cost, and user complaints.

Reevaluate after model, prompt, vendor, policy, data, interface, or scope changes. “The model did not change” does not mean the product stayed the same.

A launch review should answer:

• Is the user problem supported and scope explicit?
• Are data use, retention, access, and vendors approved?
• Does representative evaluation pass predefined gates?
• Are excluded cases detected and routed safely?
• Can users recognize, inspect, correct, and decline AI output?
• Are high-impact actions protected by code and permissions?
• Are monitoring, alerts, owners, incident response, and rollback ready?
• Are claims to users accurate and bounded?
• Is expansion tied to evidence rather than enthusiasm?

Record the decision and dissent. Conditional approval should name the unmet condition and prevent launch automatically where possible.