Worked case: scope an AI support feature

The PM gathers four permitted sources:

• observation of twelve agents handling standard cancellation and refund questions;
• six weeks of ticket and search events;
• quality reviews of 160 resolved tickets;
• interviews with eight agents and two team leads.

The evidence shows that agents spend a median of 19 minutes on eligible policy questions. About seven minutes involve searching across three policy locations. Quality reviewers find outdated policy language in 11 of 160 sampled resolutions. Agents report that policy changes are hard to notice. However, the dataset also shows that unusual billing disputes require judgment and escalation, not merely faster search.

The original “resolve tickets end to end” framing is too broad. Search friction is supported; full autonomous resolution is not. The PM writes:

Tier-one agents handling standard cancellation questions spend substantial time locating current approved policy, and some sampled replies use outdated language. This increases response time and correction work.

The team labels the sample limits. The observed agents are English-language specialists on day shifts. The evidence does not establish quality for other languages, overnight operations, legal threats, hardship cases, or disputed charges.

The outcome is not “deploy a copilot.” It is:

Reduce median time from ticket open to a reviewable, policy-grounded draft for eligible cancellation questions, without increasing unsupported policy claims, incorrect escalations, or agent workload.

The team considers four mechanisms:

1. consolidate policy documents and improve ordinary search;
2. add rule-based intent routing and policy shortcuts;
3. retrieve approved passages and generate an internal draft;
4. generate and send a response, update the account, and issue credits.

The first two are cheaper and reduce some friction. The third may reduce composition time while preserving review. The fourth bundles irreversible actions with uncertain interpretation and has no supporting evaluation. The team chooses to test improved search alongside a retrieval-and-draft prototype so the AI option must beat a credible baseline.

The beta accepts English-language cancellation tickets for two plans in one region. It retrieves from a versioned, approved policy collection and creates an internal draft. The interface shows the draft, cited passages, source version, and retrieval time. The agent may edit, reject, or copy the draft. Nothing is sent automatically.

The release explicitly excludes:

• refunds, credits, and account changes;
• legal threats, charge disputes, hardship, safety, and vulnerability cases;
• sources outside the approved policy collection;
• unsupported languages, plans, and regions;
• customer-facing auto-send;
• model training on ticket content outside approved data terms.

Out-of-scope intent routes to the existing queue. Missing or conflicting sources produce a visible abstention. If generation or retrieval is unavailable, agents use the existing search workflow.

This is narrower than the sponsor’s request, but it creates measurable value and evidence without pretending that review eliminates every risk.

The core user story is:

As a tier-one agent assigned an eligible cancellation ticket, I want a draft linked to current approved passages so I can verify policy and respond with less searching.

Acceptance criteria include:

• an eligible ticket produces an unsubmitted draft with at least one inspectable source passage;
• an ineligible ticket produces no draft and displays the correct route;
• no sufficiently supported passage produces an abstention, not a generic answer;
• conflicting current passages block drafting and notify the policy owner;
• the product records source version, prompt version, model version, agent action, and timestamps;
• stale-source detection invalidates a draft before copy;
• the interface is keyboard accessible and identifies generated content;
• service failure leaves ticket state unchanged.

AI helps create candidate criteria and adversarial cases. Product, support, engineering, design, security, privacy, accessibility, and the policy owner review them. The PM removes an invented “two-second response” requirement because no user or system evidence supports it.

The primary product measure is median time to an agent-approved draft for eligible tickets. Guardrails include:

• claim-level support by cited policy;
• correct eligible-versus-escalate routing;
• severe unsupported-claim count;
• agent rejection and material-edit rates;
• time spent verifying;
• policy-owner exception volume;
• performance by ticket intent and agent experience;
• service reliability and fallback use.

The team creates a permitted evaluation set from historical tickets, removes unnecessary identifiers, and includes routine cases, ambiguous wording, conflicting policy, prompt injection, outdated sources, accessibility flows, and known escalations. Domain reviewers produce expected routes and required policy points.

The beta gate requires zero severe unsupported claims in the release set, at least 98% correct routing for covered intents, at least 97% claim support, and no material accessibility blocker. These values are product decisions based on consequence and baseline—not numbers suggested by the model. Passing the offline set permits a limited beta; it does not prove universal safety.

Rollout begins in shadow mode: the system generates drafts that agents do not see, allowing comparison without workflow influence. Next, a small trained group receives suggestions. Every action remains reviewable, and the existing process remains available.

The team monitors threshold performance, subgroup differences, queue effects, latency, policy-source freshness, abstentions, edits, and incidents. A kill switch disables generation while preserving search. The policy owner can withdraw a source immediately. Support leads own training and escalation; engineering owns service incidents; product owns scope and metric review.

Pause conditions include any severe unsupported claim, a privacy incident, stale policy served as current, or sustained routing below the agreed threshold. Expansion requires representative evidence for each new region, plan, language, or intent. Scope does not expand merely because the first cohort likes the tool.

The sponsor receives a clear decision brief: the eight-week target is a bounded internal-draft beta, not end-to-end resolution. The evidence supports policy-search friction. It does not support autonomous account actions. The brief compares improved search with AI drafting, names the gates, and states what would be required for future expansion.

Agents receive workflow guidance and limitations. Legal, privacy, and security partners receive the data flow and controls. Finance receives a cost model that includes review, evaluation, operations, and incidents—not only model calls.

The team records dissent from one stakeholder who still wants auto-send. That proposal remains a future hypothesis requiring stronger evidence, separate risk review, and explicit authorization.