Reference · How-to · ~10 min

How to triage an AI incident

Last updated

Separate outage from silent quality failure — both are incidents.

Separate outage from silent quality failure — both are incidents.

#Steps

1. **Declare** severity: user impact, blast radius, data exposure risk.

2. **Capture** trace IDs, prompt/version, retrieval hits, model ID.

3. **Mitigate**: canary off, roll back model/prompt/index, or disable tool.

4. **Classify**: infra vs quality vs security (injection / leak).

5. **Verify** fix on golden tasks before full traffic.

6. **Write** a short postmortem with owner and prevention.

#Failure notes

  • Fixing latency while answers silently degraded
  • Redeploying without pinning the known-good index
  • **Try the lesson:** `incident-response-lab` in Lane D · Glossary: [AI monitoring](/reference/ai-monitoring)