Mastery checklist and next steps

A company inventories every AI vendor but does not record how departments use each model. Is the inventory complete?

Answer: No. Vendor and model identity are useful, but governance follows the deployed system and use. The inventory should include purpose, users, affected people, data, prompts or adaptations, retrieval sources, tools, decision authority, owner, risk tier, release, status, and linked evidence. One provider model may support a low-risk writing aid and a high-impact screening workflow; those uses need different controls.

A launch requires approval from product, engineering, legal, security, and compliance. All five boxes are checked, but nobody is empowered to pause production. What should change?

Answer: Keep specialist reviews, but name one accountable launch owner and an operational role with explicit pause authority. Define escalation and safe defaults. Participation does not replace accountability, and launch authority does not automatically create incident authority.

A card says, “The assistant is accurate, unbiased, secure, and always reviewed by a human.” What is wrong with this statement?

Answer: Each claim is too broad to verify. Replace it with scoped evidence: task and population, system release, test-set version, measures, subgroup results, security tests, dates, limitations, review step, reviewer information, authority, timeout behavior, and links to artifacts. State unknowns and residual risks. Documentation should support decisions, not advertise certainty.

A candidate meets its launch threshold, but the production unsupported-answer rate rises above its monitoring threshold. The owner argues that the approved evaluation remains valid. What is the governance response?

Answer: The historical evaluation remains a record of tested conditions, not permission to ignore new evidence. Investigate the shift, restrict scope or restore human review, and pause or roll back if stop conditions apply. Add representative failures to the evaluation set, identify whether inputs, retrieval, configuration, or population changed, and record the decision.

A chatbot moves from drafting refund replies to issuing refunds automatically. The model and prompt are unchanged. Is this a minor release?

Answer: No. Tool permission and decision authority materially change impact and reversibility. Reclassify risk, review permissions and fraud threats, add transaction evaluations, introduce approval or limits, define monitoring and reconciliation, establish rollback, update documentation, and obtain the required approval. Governance controls the system, not only model weights.

During an incident, an engineer finds a likely fix and wants to restart immediately because tests pass on the reported example. What else is needed?

Answer: Preserve evidence, assess scope, test related and adversarial cases, review downstream harm, record the changed configuration, define heightened monitoring and rollback triggers, handle notifications or remediation, and obtain restart authorization. One passing example can show the symptom changed; it does not establish safe recovery.

Purpose and inventory

• The system boundary includes model, prompts, data, retrieval, tools, interface, people, and downstream actions.
• Intended users, affected people, approved uses, prohibited uses, and decision authority are explicit.
• The current release and component versions can be reconstructed.
• Data roles, sources, purposes, access, retention, and provider flows are recorded.
• The risk tier reflects impact, scale, sensitivity, reversibility, human control, and uncertainty.

Ownership and evidence

• One accountable role owns each major decision.
• Operators, specialists, reviewers, escalation contacts, and backups are named.
• Approvers have authority and do not treat silence as approval.
• The system card records evidence, limits, controls, findings, status, and review date.
• Claims link to evaluations, approvals, logs, runbooks, or other verifiable records.
• Exceptions have an owner, rationale, compensating control, and expiry.

Evaluation and operation

• Evaluation begins with a decision and plausible failure modes.
• Test cases represent normal, difficult, subgroup, misuse, and prior-incident conditions.
• Measures include severity and slices, not only an overall average.
• Thresholds and automatic blockers are set before the release decision.
• Production monitoring covers inputs, behavior, outputs, and outcomes.
• Every important signal has a threshold, owner, response time, and available action.
• Logging is useful but proportionate, access-controlled, and retained appropriately.

Change and response

• Material-change triggers cover models, prompts, data, tools, permissions, workflow, population, and purpose.
• Rollouts have stages, entry criteria, stop conditions, and a recoverable prior state.
• Emergency changes receive minimum tests, approval, monitoring, and follow-up.
• Incident intake, severity, command, containment, evidence, communication, and restart routes are practiced.
• Incidents and near misses update evaluation sets, controls, documentation, and training.
• Retirement removes access and integrations while handling data and record obligations.

Governance and regulation are related but not interchangeable. Regulation may require risk management, records, transparency, data controls, human oversight, monitoring, incident reporting, or routes for challenge. Governance assigns owners and operating mechanisms that can satisfy applicable duties.

Do not assume a complete internal checklist proves legal compliance. Obligations differ by jurisdiction, sector, role in the supply chain, data, and use. A legal or compliance review should identify what applies. Governance then keeps those decisions connected to actual releases and evidence. If a rule changes, change management updates the system rather than leaving advice in an old memo.

The strongest bridge is traceability: an obligation points to a control, the control points to an operator and evidence, and findings point to remediation. This also reveals gaps where a policy claim has no operating proof.

Human oversight is one governance control, not a universal solution. Place people where judgment can prevent or correct consequential effects. Specify what they decide, what evidence they see, their expertise and workload, available choices, timeout behavior, and escalation authority.

The governance owner asks whether the human gate is appropriate and operating. The workflow designer makes it usable. Monitoring checks overrides, delays, disagreements, missed reviews, and outcomes. Change management reassesses the gate when system authority or evidence changes.

Low-risk drafting may need ordinary editing. High-impact actions may need approval, dual control, or specialist escalation. High-volume reversible classification may use sampling. The control should follow impact and uncertainty rather than the desire to say “a human is involved.”

Choose one real system and create a one-page governance map. Include its approved use, boundary, owner, risk tier, card link, release gate, three monitoring thresholds, pause authority, material-change triggers, incident route, and next review date. Ask a person outside the project to follow the map. Any step that depends on tribal knowledge becomes your next improvement.