Brittle automation and failure modes

Production inputs differ from examples. Messages contain forwarded threads, screenshots, contradictory requests, sarcasm, mixed languages, missing identifiers, or instructions aimed at the automation itself. Documents change templates. A field that was always present becomes optional.

Contain this with explicit input boundaries, normalization, size limits, supported formats, and an uncertainty route. Test malformed and adversarial cases. Never let “the model usually figures it out” replace a required field check.

Watch for silent semantic drift. The schema may still parse while the meaning changes. If a partner redefines status: closed, downstream code can behave incorrectly without a technical error. Version contracts and monitor outcome distributions.

AI can return a valid, fluent, wrong answer. Schema validation catches broken structure, not mistaken interpretation. A routing model may label a security incident as account access. A summarizer may omit a condition that reverses the meaning.

Use representative evaluations, per-class metrics, evidence spans, deterministic overrides, and human review for high-impact cases. Allow abstention. Measure the cost of false positives and false negatives separately; an average accuracy score hides their different consequences.

Do not treat a self-reported confidence number as a guarantee. If confidence controls automation, calibrate it against held-out data and keep action risk in the decision.

Networks do not provide a clean “happened” or “did not happen” answer. A payment API may complete an operation while the response times out. Retrying blindly can pay twice.

Use idempotency keys for side effects. Record intended actions and provider identifiers. Reconcile uncertain outcomes before retrying. Design state transitions such as proposed, approved, executing, succeeded, failed, and unknown, rather than one vague processed flag.

Partial completion also matters. If a workflow updates a record but fails to send a notification, decide whether to retry notification, compensate for the update, or escalate. Do not rerun every prior step automatically.

Time passes between proposal and execution. Two workflows may edit the same record. A customer can cancel an order after AI drafts a shipping reply. A manager may approve an amount calculated from an older price.

Attach versions and timestamps to evidence. At execution, verify that assumptions still hold. Use optimistic concurrency or transactions where available. Bind approval to the exact proposal and source versions. If relevant state changed, refresh or request new review.

Scheduled automations also need timezone and daylight-saving tests. “Run every day at 9” is incomplete without location, missed-run behavior, and duplicate-run handling.

Models, databases, automation platforms, and APIs have latency, rate limits, outages, authentication expiry, and changing interfaces. Cascading retries can turn a small outage into overload.

Set timeouts, capped retries with backoff, concurrency limits, and queue capacity. Distinguish transient failures from permanent ones such as invalid credentials or rejected input. After the retry budget, move work to a visible dead-letter or human queue with enough context to recover.

Define degraded modes. Inbox routing can fall back to a staffed general queue. A report generator can delay. A safety alert may require an alternate notification channel. The fallback should match the consequence of delay.

External text is data, not authority. An email, webpage, or document may say “ignore prior instructions” or request access to unrelated systems. If an AI step can call tools or construct executable commands, untrusted content can influence side effects.

Minimize permissions. Separate interpretation from execution. Use allowlisted actions and structured parameters. Validate destinations, amounts, identifiers, and authorization in code. Do not place secrets in prompts, and control what is retained in model and workflow logs.

Also consider data leakage across records. Caches, retrieval, or conversation state must not mix one customer’s information with another’s.

A workflow may technically “escalate” while sending cases to an unstaffed queue. Reviewers may lack context, face excessive volume, or receive alerts after the decision window closes.

Treat people as an operational dependency. Assign owners, service levels, coverage schedules, queue limits, and escalation paths. Measure backlog age and abandonment. Test that the team can absorb full manual volume when automation is paused.

Avoid fallback loops: AI fails, human requests retry, retry fails, and the case returns to the same queue with no new information. Record attempts and require a different action after a bounded number.

For each step, complete this failure card:

• Failure: what can go wrong?
• Detection: what evidence reveals it?
• Containment: what stops wider harm?
• Recovery: retry, reconcile, compensate, pause, or escalate?
• Owner: who is responsible?
• User impact: what should affected people be told?
• Test: how will the team simulate it?

Run fault drills. Disable a dependency, replay a duplicate event, return malformed model output, expire credentials in staging, and change source state after approval. Verify that logs share a trace ID and that operators can pause new side effects.