Mastery: run a trustworthy analysis

A leader asks, “Which customers are best?” What should happen first?

A. Upload all customer records and request a ranked list
B. Ask AI to choose a definition based on column names
C. Clarify the decision and define “best” operationally
D. Rank customers by whichever numeric field has the largest values

Answer: C. “Best” could refer to revenue, margin, retention, growth, strategic fit, or service cost. Each definition creates different rankings and consequences. Clarify what action the ranking supports, define the measure and period, and consider whether ranking people or organizations raises fairness, privacy, or policy concerns.

An orders export has 50,000 rows but only 42,000 distinct order IDs. Can you calculate average order value by dividing summed revenue by row count?

Answer: Not yet. Determine what one row represents and why order IDs repeat. The table may contain line items, updates, payments, or accidental duplicates. Aggregate to one row per eligible order using a documented rule, then divide total eligible order value by eligible order count. Check whether a join or status history duplicates revenue.

The formula can be simple while the grain decision is hard.

Satisfaction scores are present only for 12% of support tickets. The observed average improved after a workflow change. Is the result enough to say customer satisfaction improved?

Answer: No. Respondents may differ from nonrespondents, and response patterns may have changed after the workflow. Compare response rates and respondent composition by period, channel, priority, and customer group. Report the observed respondent result narrowly. Seek additional evidence before generalizing to all customers.

Filling missing scores with the average would not solve selection bias.

An AI states that conversion rose from 4% to 5%, “a one-percent increase.” What is the precise description?

Answer: Conversion rose by 1 percentage point and by 25% relative to the 4% baseline. Report the numerator, denominator, eligibility rule, period, and uncertainty as well. If the earlier rate came from 25 visitors and the later rate from 20, the visible difference could represent no additional conversions at all.

Important numbers need formulas and independent reproduction, not only corrected wording.

Sales rose after a pricing change. What can the time order establish?

Answer: It establishes that the increase followed the change in the observed data. It does not isolate the effect of price. Promotions, seasonality, inventory, market conditions, measurement changes, and customer mix may contribute. A causal claim needs a design and assumptions capable of ruling out credible alternatives, such as a randomized test or a justified comparison with appropriate pre-period checks.

AI can propose explanations, but proposals should be labeled as hypotheses.

Use this checklist before sharing a consequential finding.

Frame

• The decision or learning goal is written in one sentence.
• The primary question is specific and answerable with available evidence.
• Population, measure, denominator, comparison, and period are defined.
• Descriptive, diagnostic, predictive, and causal goals are not confused.
• Success and stop conditions are explicit.
• Claims the data cannot support are listed.

Govern

• The data source is authorized, versioned, and owned.
• Only necessary columns and rows enter the AI environment.
• Direct identifiers and sensitive free text are removed or protected.
• Tool retention, access, and sharing rules are understood.
• Small-group reporting and re-identification risks are controlled.
• Raw data remains unchanged and recoverable.

Prepare

• One row’s meaning and expected unique key are documented.
• Types, units, currencies, timezones, and category definitions are confirmed.
• Missingness is profiled overall and by relevant group.
• Exact duplicates and repeated keys are investigated separately.
• Every cleaning rule has a reason and affected-row count.
• Before-and-after row counts reconcile.

Analyze

• Prompts include the question contract, schema, and boundaries.
• Calculation tables come before charts and narrative.
• Numerators, denominators, filters, and exclusions are visible.
• Charts match the analytical task and use honest scales.
• Counts accompany rates, and sample sizes accompany comparisons.
• Observations, interpretations, hypotheses, and recommendations are separated.
• Major subgroups and reasonable sensitivity choices are checked.

Verify

• Decision-critical formulas are written plainly.
• Join relationships, unmatched keys, and row-count changes are inspected.
• Representative raw rows and edge cases are recalculated manually.
• Results are reproduced with an independent method.
• Group totals reconcile to overall and authoritative totals.
• Invariants and range checks pass.
• Partial periods, late-arriving data, and future-information leakage are tested.
• Corrections trigger reruns of all dependent outputs.

Communicate

• The opening states the decision, finding, magnitude, and key limit.
• Every reported number traces to a verified calculation.
• Causal language matches the study design.
• Caveats are specific and prominent in proportion to their importance.
• Recommendations include owner, scope, measure, guardrail, and review point.
• Source version, extraction date, period, definitions, and analyst are visible.
• The evidence package is available for review and correction.

Do not ask one giant prompt to clean, analyze, chart, and recommend. Use checkpoints:

Prompt 1 — clarify: “Review my question contract. List ambiguity, unavailable evidence, and decisions I must make. Do not analyze yet.”

Prompt 2 — profile: “Generate read-only profiling steps for schema, keys, missingness, ranges, categories, and dates. Do not repair values.”

Prompt 3 — prepare: “Given these approved cleaning rules, generate transformations plus before-and-after checks. Preserve raw columns and create an audit log.”

Prompt 4 — calculate: “Produce intermediate tables, formulas, filters, denominators, and code. Do not write an executive conclusion.”

Prompt 5 — challenge: “List plausible failure modes, subgroup reversals, alternative definitions, and sensitivity checks that could change this result.”

Prompt 6 — communicate: “Using only the verified evidence block, draft a decision note. Preserve values and qualifiers. Flag unsupported claims.”

Human approval belongs between stages. A model should not invent a cleaning decision and then use its own invention as evidence.

An analysis is ready to share when the primary question is answered at the appropriate confidence, critical calculations reproduce independently, major data-quality issues are bounded, reasonable alternative definitions do not silently reverse the conclusion, and the communication preserves limitations.

Do not ship when:

• the denominator or row unit remains unknown;
• a many-to-many join is unexplained;
• missingness is concentrated and could reverse the result;
• current and comparison periods are mismatched;
• a causal recommendation rests only on timing or correlation;
• sensitive data was handled outside approved controls;
• the result exists only in AI prose and cannot be reproduced.

Stopping is a valid analytical outcome. “We need a reliable customer identifier before estimating retention” may be more valuable than a precise-looking but invalid retention rate.

For a final practice project, assemble:

1. question contract;
2. source inventory and data dictionary;
3. raw profile;
4. cleaning rules and log;
5. analysis-ready table version;
6. calculation code or formulas;
7. audit and sensitivity tables;
8. chart with accessible labels;
9. verification record;
10. decision note with limitations and next step.

Ask a reviewer to select one number and trace it backward. If they cannot identify its definition, source rows, transformation, and check, improve the packet before sharing.