Test a capability claim

Capability claims need a defined baseline, representative examples, measurable criteria, edge cases, and evidence gathered in the intended setting.

Use this three-part method:

1. Translate marketing language into testable statements. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
2. Build ordinary, difficult, and adversarial examples. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.
3. Compare outcomes with the current process and record failure types. Write down what this means in the scenario, what evidence would show it was done, and who owns the decision.

Walk through A vendor says its assistant understands every company document and reduces errors by ninety percent.. Label three moments where “Test a capability claim” changes what you trust: (1) the first fluent answer, (2) the first missing source or permission, and (3) the decision a human must own. Write the before/after task so the model only does the slice that evidence supports. Keep one sentence that states how this page’s idea differs from a generic “AI is smart/dumb” score.

• Letting the vendor choose only showcase examples. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
• Reporting average quality while hiding severe failures. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.
• Testing without the people who will use or be affected by the system. This shortcut removes useful friction, but it also hides an assumption that should be tested. Replace it with an observable check.

Before you close the tab, capture a reusable habit for Test a capability claim inside What AI can and can't do: name the observable check, the evidence you would open, and the stop condition. Rehearse it once on a low-stakes example, then once on a higher-stakes variant. The habit succeeds when you can explain the check without reopening this lesson. Target outcome: Explain AI capabilities as task-specific and conditional rather than magical.