Incident response

An AI incident is not limited to an outage. It may involve actual or credible potential harm caused or enabled by the system. Examples include:

• exposure of personal, confidential, or proprietary data;
• discriminatory or systematically unequal outcomes;
• unsafe advice or actions;
• unauthorized tool use or excessive permissions;
• manipulation, prompt injection, fraud, or abuse;
• widespread incorrect or unsupported outputs;
• failure of required human review;
• misleading disclosure or inability to challenge a result;
• monitoring, logging, or rollback failure that removes control;
• breach of an approved-use boundary or legal obligation.

Teams can also track near misses, where a control prevented harm or luck limited impact. Near misses provide valuable examples before a larger event occurs.

Write reporting criteria in language ordinary employees and vendors can use. “Report suspected leakage, harmful recommendations, unauthorized actions, repeated group-specific errors, or loss of control” is more useful than requiring a reporter to prove severity.

Provide several intake routes: monitoring alerts, employee reports, user complaints, security channels, reviewer escalations, and vendor notifications. All should create a traceable record and reach an on-call owner.

Initial triage asks:

1. What happened or may have happened?
2. Is harm continuing?
3. Which system, release, users, data, tools, and locations may be involved?
4. What is the likely severity and spread?
5. Which immediate controls can reduce exposure?
6. Which specialists and decision makers must join?

Severity should combine impact and urgency. Consider sensitivity of data, physical or financial harm, rights and vulnerable people, number affected, duration, detectability, reversibility, malicious activity, public exposure, and regulatory or contractual duties. Use levels with explicit response times and authorities.

Do not wait for perfect certainty before containing a credible high-impact risk. At the same time, distinguish facts, hypotheses, and unknowns in the incident record.

Containment options include pausing the feature, disabling a tool, removing a data source, restoring human review, restricting affected users or languages, rolling back a release, revoking credentials, or switching to a safe fallback.

Choose the least disruptive action that reliably limits harm, but safety comes before availability. The incident commander needs authority to act without searching for the normal launch committee.

Preserve evidence under appropriate access controls:

• system and component version identifiers;
• prompts, retrieved sources, outputs, and tool calls;
• timestamps, trace IDs, user and reviewer actions;
• deployment and configuration changes;
• alerts, complaints, tickets, and communications;
• relevant provider status or release notices.

Do not copy sensitive data into uncontrolled chats or documents. Restrict access, retain only what is necessary, and involve privacy, security, legal, or records specialists where required. Evidence preservation and data minimization must be designed together.

Name an incident commander who coordinates decisions and a scribe who records timeline, evidence, actions, and owners. Bring in technical, product, operations, security, privacy, legal, communications, vendor, and domain expertise according to the event. The accountable system owner remains involved, but the incident commander controls response coordination.

Create a regular update rhythm. Internal updates should state confirmed facts, current impact, containment status, unknowns, next decisions, and owner. External communication should be accurate, understandable, timely, and approved by the appropriate roles.

Avoid premature statements such as “no data was exposed” before scope is known or “the AI made a mistake” in a way that erases organizational responsibility. If affected people need instructions, correction, appeal, or remediation, make those routes concrete.

Notification duties vary by jurisdiction, contract, sector, and harm. A response plan should identify who determines whether customers, partners, regulators, insurers, or law enforcement must be notified and within what deadline. This lesson is an operating framework, not legal advice.

Containment stops immediate exposure; recovery restores a controlled service. Before restart, define:

• the understood cause or safe working hypothesis;
• fixes and compensating controls;
• regression and adversarial tests;
• accepted residual uncertainty;
• monitoring during a heightened observation period;
• rollback triggers;
• restart approver;
• remediation for affected people and records.

A prompt edit alone may not resolve a systemic issue. Look across data, retrieval, permissions, interface, staffing, incentives, and provider behavior. If the cause remains uncertain, restart a narrower mode: draft-only, limited users, no tools, stronger human approval, or reduced data.

Treat restart as a governed change. Record the new configuration and evidence. Do not allow commercial pressure to turn “service available” into the only recovery criterion.

After stabilization, run a review that reconstructs the timeline and asks why controls did or did not work. Separate the initiating failure from contributing conditions: unclear ownership, weak evaluation, missing subgroup tests, alert fatigue, excessive permissions, stale documentation, or an unpracticed rollback.

Produce actions with owners and dates. Feed incident examples into evaluation suites. Update thresholds, model or system cards, role maps, training, vendor requirements, user support, and risk classification. Share lessons at an appropriate level so other teams can recognize similar patterns.

Track whether actions are completed and effective. “Train staff” is weak unless it identifies the behavior, audience, evidence of completion, and follow-up check. “Improve monitoring” needs a signal, threshold, owner, and response.

Practice with tabletop exercises. Present a plausible event, then ask who is paged, who can pause, what evidence exists, how users are supported, and who authorizes restart. A plan first opened during an incident is only a document.