Worked case: govern a support assistant

The product owner writes the intended use:

The assistant answers authenticated customers’ English-language questions about published billing policies by retrieving approved help-center sources. It may explain charges and draft next steps. It may not issue refunds, alter accounts, make individualized affordability decisions, invent policy, or answer outside the approved billing scope.

That sentence narrows users, language, subject, sources, and authority. The system includes a hosted language model, system prompt, retrieval service, approved document index, chat interface, authentication, redaction layer, logging service, escalation queue, and human support staff.

The team identifies affected people beyond direct users: account holders whose information may appear in a shared conversation, support staff receiving escalations, and customers who may rely on an incorrect answer.

The inventory record names the provider and model version, data categories, integrations, region, accountable product owner, technical owner, operations owner, approval status, and links to evidence.

The team rates the initial use as moderate rather than minimal. Answers concern money and personal account context, can influence customer choices, and operate at scale. However, the system has no account-changing tools, uses a bounded knowledge source, and offers human escalation. A future refund capability would require a new and higher-risk assessment.

Roles are explicit:

• The billing product director is accountable for intended use and launch.
• The engineering lead owns configuration, access, deployment, and rollback.
• The help-center content owner approves retrieval sources and freshness.
• The evaluation lead runs tests and reports results independently of the release engineer.
• Privacy and security specialists review data flow, provider terms, redaction, logging, and abuse controls.
• Support operations owns the escalation queue, sampled review, and customer remediation.
• An on-call service manager may pause automated replies.
• A cross-functional launch approver accepts residual risk after reviewing evidence.

The safe default is draft-only operation if monitoring is unavailable, the knowledge index is stale, or no incident owner is reachable.

The card separates data roles. The provider’s base training data is documented at the level the vendor can support. Northstar does not fine-tune the model. Retrieval uses a curated set of public billing policies with document owners and expiry dates. Operational inputs include authenticated chat text and limited account facts needed to explain a charge. Logs retain redacted prompts, sources, outputs, routing, and feedback for a defined period.

Provider settings prohibit using Northstar inputs for model training under the contract. Access to raw logs is restricted. The team records an unresolved limitation: automated redaction may miss unusual identifiers. The compensating controls are minimized account context, blocked requests for highly sensitive fields, sampled privacy review, and a kill switch.

The system card states that answers are informational, may be wrong, and must cite the policy source. It gives customers a visible route to a person. Internal guidance tells reviewers not to treat fluent language as evidence.

The evaluation plan supports one decision: whether to begin a 5% customer canary in approved English billing topics.

The test set includes common historical question patterns under approved data controls, expert-authored edge cases, newly changed policies, ambiguous charges, emotional requests, prompt-injection attempts, requests for account changes, unrelated topics, and cases involving sensitive information.

The team measures:

• factual correctness against approved policy;
• whether cited sources support the answer;
• correct escalation and refusal;
• leakage of unrelated customer information;
• prohibited promises or account actions;
• quality by billing topic and major writing patterns;
• latency and availability.

Launch thresholds require all critical privacy cases to pass, no claims of completed account changes, at least 95% supported answers on in-scope questions, and at least 98% correct escalation on prohibited or uncertain cases. Any high-severity failure blocks launch regardless of the average.

The first candidate misses the escalation threshold because it tries to answer debt-hardship requests. Instead of accepting the overall score, the owner keeps those requests human-only, adds routing, updates the card, and reruns the affected suite.

The system begins in shadow mode. It drafts answers beside human agents without sending. Reviewers label correctness, source support, and needed edits. After thresholds hold for two weeks and queue capacity is confirmed, the launch approver authorizes a 5% canary for low-complexity topics.

Monitoring covers input shifts, retrieval failures, unsupported-answer samples, escalation rate, refusal rate, complaints, human corrections, latency, and attempted prompt injection. Results are sliced by billing topic. Operations reviews daily during the canary and weekly after expansion.

Stop conditions include any cross-customer disclosure, claim of an unauthorized account action, critical policy error, unsupported-answer rate above threshold, unavailable monitoring, or escalation backlog beyond capacity. The service manager can switch all traffic to human support without a code release.

Expansion to 25%, then 100% of the approved scope requires a recorded review. It does not authorize new languages, topics, or actions.

Two months later, the model provider offers a cheaper version. Procurement savings do not make the change routine. The technical owner opens a material-change record because model behavior may differ.

The candidate runs against the stable regression set, fresh production-like samples, and adversarial cases. Overall correctness is similar, but source support drops on long policy pages. The team delays migration, adjusts retrieval, and compares again. After thresholds pass, it rolls out through shadow and canary stages, preserves the prior configuration, updates the system card, and monitors heightened signals for one week.

A customer reports receiving an answer that quotes another account’s payment note. Support creates an incident record and pages the on-call manager. Automated replies pause immediately. The incident commander restricts evidence access, preserves the trace and release identifiers, and brings in security, privacy, engineering, legal, support, and the owner.

Investigation finds that a cache key omitted the customer identifier after a recent infrastructure change. The team invalidates the cache, patches the key, checks exposure, supports affected customers, evaluates notification duties, and tests isolation under concurrency. Restart is limited to shadow mode until privacy tests pass and the accountable approver reviews evidence.

The retrospective adds cross-user isolation cases to every release suite, a deployment check for cache configuration, and a monitoring signal for mismatched account identifiers. The incident updates documentation and risk review. Governance has turned failure into stronger controls.