Mastery: product AI checklist

A stakeholder asks for a personalized AI coach because competitors have one. Which response is strongest?

A. Ask AI to copy the competitor’s feature list.
B. Commit to a coach and write the PRD later.
C. Identify the target user and job, investigate observed friction, define an outcome, and compare AI and non-AI mechanisms.
D. Reject AI because competitor requests are always invalid.

Answer: C. Competitor activity may be relevant context, but it is not proof of user value. Product discovery should test the situation and outcome before choosing the mechanism.

An assistant adds “responses appear instantly and are always accurate” to a PRD. What should the PM do?

Answer: Remove the unsupported guarantee. Define observable latency and quality measures from user needs, technical evidence, and consequence. Write behavior for unavailable, uncertain, and out-of-scope conditions. Trace each approved requirement to its source and owner.

The model’s confidence and polished wording do not authorize a commitment.

Ten interview participants report different onboarding problems. The generated summary says, “Users need an AI guide.” What is wrong?

Answer: It generalizes from a bounded sample, collapses distinct causes, hides variation, and jumps from finding to solution. Rebuild evidence units with source IDs, exact quotations or faithful observations, contexts, counterexamples, confidence, and limitations. State findings separately from recommendations and triangulate reach with other methods.

A proposal has the highest RICE score, but its ranking reverses when reach falls by 10%, and it involves an unapproved sensitive-data use. Is it first?

Answer: Not on this evidence. The ranking is fragile, and the data constraint may block the proposal regardless of score. Show ranges, sources, sensitivity, and factors outside the formula. Resolve or exclude the unapproved use, then make an accountable decision aligned with strategy and risk.

Can the executive update say a pilot is “on track” while engineering receives a “release blocked” status?

Answer: No. Audience versions may differ in detail and requested action, but current facts, status, dates, measures, and uncertainty must stay consistent. Start from one approved fact sheet and explain what the failed gate means for each audience.

Offline quality exceeds the average threshold, but one critical class has poor recall and reviewers cannot keep up with the queue. Should the feature launch broadly?

Answer: No. Aggregate quality can hide a consequential slice, and overloaded review is a failed control. Narrow scope, improve routing or abstention, restore review capacity, and require class-specific and operational gates before expansion.

Complete the following checklist for a real proposal. Do not use AI to invent missing evidence. Mark items as verified, assumed, open, not applicable, or blocked, and name an owner and review date.

1. Problem and outcome

• target user, context, job, and observed friction;
• evidence sources, dates, samples, and limitations;
• consequence and baseline;
• user or operational outcome independent of AI;
• non-AI alternatives considered;
• assumptions that would invalidate the proposal.

2. Scope and behavior

• smallest valuable, reversible, inspectable slice;
• eligible inputs and users;
• observable outputs and user controls;
• exclusions and safe routes;
• dependencies and authoritative sources;
• prohibited actions;
• conditions for future expansion.

3. Requirements and traceability

• behavioral functional requirements;
• privacy, security, reliability, accessibility, latency, and supportability requirements;
• positive, negative, boundary, and failure acceptance scenarios;
• requirement-to-evidence and requirement-to-test links;
• approved terminology and definitions;
• open questions with decision owners.

4. Research and synthesis

• participant consent and permitted tool use;
• minimized and protected data;
• stable source identifiers;
• verified evidence extraction;
• themes with inclusion rules and contradictions;
• distinction among sample frequency, severity, and potential reach;
• findings separated from recommendations.

5. Prioritization and decision

• framework matched to the decision;
• normalized input definitions;
• source and confidence for each estimate;
• low, base, and high scenarios;
• sensitivity and ranking stability;
• mandatory work and risk outside the score;
• final decision, rationale, dissent, owner, and revisit trigger.

6. Evaluation

• task-specific measures and rubrics;
• representative common, boundary, rare, and adversarial examples;
• segment and condition slices;
• separate development and release sets;
• versioned prompt, model, data, retrieval, and evaluator guidance;
• predefined release thresholds;
• residual risk and known blind spots.

7. Human control and safe failure

• review evidence, choices, authority, and training;
• expected review volume and capacity;
• approval bound to the exact proposal and current state;
• abstention and escalation behavior;
• deterministic enforcement for permissions and consequential actions;
• accessible fallback that preserves work;
• no-success, no-result, and system-failure states distinguished.

8. Rollout and operations

• offline, shadow, suggestion, narrow-release, and expansion stages as appropriate;
• entry, exit, pause, and rollback criteria;
• dashboards, alerts, logs, source freshness, and audit records;
• privacy, security, policy, product, engineering, and operations owners;
• incident response, kill switch, and tested manual fallback;
• reevaluation triggers for any material system change;
• user communication that accurately states capabilities and limits.

Ask a cross-functional group to challenge the pack from several perspectives: a target user, an excluded user, an operator, an attacker, a policy owner, an accessibility reviewer, a privacy or security partner, and the person responding to incidents.

AI can generate candidate questions:

• What input makes the system appear confident when evidence is absent?
• Which user experiences worse performance?
• What pressure could cause reviewers to rubber-stamp?
• Which metric can improve while real outcomes worsen?
• What stale dependency would silently corrupt results?
• What action is difficult to reverse?
• What claim in stakeholder messaging exceeds the evidence?

Verify and prioritize these questions with domain experts. Red teaming is not a one-time prompt and not a substitute for technical testing.

The final decision is not simply launch or fail:

Ship when the bounded scope passes gates and operations are ready.
Narrow when a smaller segment or behavior has sufficient evidence and safe fallback.
Learn when a targeted prototype, experiment, or research step can resolve a decision-critical assumption without exposing users to unjustified risk.
Stop when the problem lacks support, expected value is weak, necessary data use is unacceptable, harm cannot be controlled, or a simpler solution is better.

Record which option was selected, by whom, based on what evidence, and when it will be revisited. Stopping a weak AI proposal is a successful product decision.