Recognize common analysis failure modes

AI may infer columns, categories, or records that are not present. If a prompt says “compare regions” but the table contains only store IDs, a model may invent a region mapping or speak as though one exists. It may also describe a sample as if it were the complete dataset.

Prevent this by supplying an explicit schema and requiring the model to list columns used. Tell it to stop when a required field is absent. Compare every reported value with a calculation table generated from actual rows.

Watch for “helpful” repairs: invented dates, silently expanded abbreviations, guessed currency, or fabricated labels for coded values. Plausible completion is useful in writing and dangerous in measurement.

An analysis can use correct arithmetic on the wrong unit. Counting invoice lines as invoices, averaging daily averages instead of transactions, or joining one order to several customer-history records changes the result.

State the expected grain before each transformation. Check key uniqueness and row counts. If AI proposes a join, ask it to predict whether the number of rows should stay the same, increase, or decrease, then compare prediction with reality.

Many-to-many joins deserve an explicit stop. They may be valid, but their weighting effect must be intended and documented.

If only completed projects have a completion_days value, averaging that column excludes all unfinished projects. The result describes survivors, not all projects. An apparent improvement may occur because difficult cases remain open and therefore disappear from the metric.

Missing data can concentrate in one period or group. AI-generated prose may mention the overall missing percentage while overlooking that one segment has 60% missing values.

Profile missingness by the same dimensions used in the analysis. Include records that have not yet reached the outcome where appropriate, or clearly narrow the population. Ask what kinds of records are absent entirely: cancelled accounts, failed imports, offline sales, or people who never consented to tracking.

Selection bias occurs when observed records differ systematically from the population behind the decision. Survey respondents, support contacts, and active users are rarely random samples of all customers.

Denominator mistakes can reverse conclusions. A product may have the most returns because it has the most sales, while its return rate is ordinary. A city may have fewer incidents because reporting coverage is lower.

Require counts and rates together. Define who had the opportunity to produce the outcome. Do not compare rates built from different eligibility rules.

An overall trend can differ from every subgroup trend when the group mix changes. Suppose success rates improve for both new and experienced agents, but the overall rate falls because many more cases are assigned to new agents. The aggregate is mathematically correct yet misleading about performance.

Check major segments before interpreting an overall change. Compare group weights across periods. Do not slice endlessly until a preferred answer appears; select segments based on the question and known process.

Composition changes also affect means. A hospital’s average wait can rise even if each department improves when more emergency cases arrive. AI needs operational context to recognize that possibility.

AI is especially good at turning co-movement into a story. “Sales rose after the redesign” is temporal. “The redesign increased sales” is causal. Other campaigns, seasonality, price, inventory, or customer mix may explain the change.

Use cautious language for observational data: associated with, coincided with, or followed. For causality, identify treatment assignment, comparison group, pre-period trends, confounders, and assumptions. If the design cannot support cause, say so.

Time-series data also creates spurious relationships. Two growing measures can correlate because both trend upward. Seasonality can make unrelated December measures move together. Compare matched periods and inspect underlying trends.

Outliers may be errors, rare valid events, or the finding itself. Deleting them because they “distort the average” hides information. Investigate source records, compare robust measures such as the median, and report results with and without a justified rule.

Truncated data creates false stability. A current month with only ten days should not be compared as a total with complete months. Dashboards may also omit late-arriving events, making recent performance look artificially low.

Leakage happens when analysis uses information unavailable at the relevant decision time. A churn model that includes a cancellation-reason field can look excellent because that field is collected after churn. Even descriptive analysis can leak future status into a historical cohort. Ask when each field becomes known.

If you examine 100 segments, several will look unusual by chance. Asking AI to “find the biggest insight” without a preselected question encourages it to elevate noise.

Separate planned analysis from exploration. For exploratory findings, report how many comparisons were searched and validate the pattern on new data or a later period. Avoid repeatedly changing filters until the desired result appears.

AI can generate many hypotheses quickly; that makes disciplined confirmation more important, not less.

A correct analysis can still be unsafe. Uploading restricted customer data to an unapproved service, retaining exports indefinitely, or publishing a small-group breakdown can expose people. Generated code may overwrite raw files or send data to external libraries.

Use approved tools, least-necessary columns, read-only raw inputs, versioned outputs, and minimum reporting thresholds. Review code before execution. Treat free-text columns as potentially sensitive and as a source of prompt injection if their contents are passed into an AI workflow.

Before accepting a result, imagine it is wrong and ask how. Check:

• wrong question or definition;
• incomplete or stale source;
• type, unit, timezone, or category error;
• duplicate, join, or grain error;
• missingness or selection bias;
• denominator mismatch;
• unstable outlier handling;
• subgroup reversal;
• causal overclaim;
• privacy or governance breach.

Record which checks passed, failed, or remain unknown. Unknown is different from passed.