Limits, security, and human control

Give each agent only the tools and data required for its task. A ticket-triage agent may read one ticket, search approved help articles, assign a category, and draft a response. It does not need bulk customer export, billing changes, shell access, or unrestricted outbound messaging.

Apply least privilege at several layers:

• issue short-lived credentials scoped to one task or tenant;
• filter records server-side rather than asking the model to ignore others;
• allowlist tools and destinations;
• separate read tools from side-effect tools;
• constrain paths, domains, amounts, and recipient types;
• run untrusted code or files in isolated environments.

Tool descriptions are guidance, not enforcement. “Only use this tool for approved recipients” is weaker than a server that requires a signed approval record and rejects recipients outside the approved set.

Agents often process adversarial or accidental instructions embedded in web pages, documents, emails, code comments, or tool output. This is an indirect prompt-injection risk. The runtime should preserve authority boundaries: system policy and explicit user instructions outrank untrusted content.

Useful defenses include provenance labels, content isolation, output encoding, and policy checks before actions. Tools should return data in structured fields where practical. The model may summarize a document, but document text cannot expand its tool allowlist or rewrite its budget.

Assume the model can still be influenced. Security should survive a bad action proposal. If it proposes send_email to an unapproved external address, the policy layer rejects the call and records the reason. This approach evaluates observable actions instead of trying to prove what happened inside the model.

An agent can consume resources variably, so define multiple budgets:

• maximum loop steps;
• per-tool and total calls;
• model tokens or monetary spend;
• wall-clock duration;
• retrieved bytes or documents;
• number and value of side effects;
• retry count by error category.

One global step limit is not enough. Ten read-only searches and ten account deletions do not carry equal risk. An action budget might allow unlimited local calculations within time limits, five searches, two draft creations, and zero executions without approval.

Budget exhaustion must have explicit behavior. The system should stop, preserve partial results, state what is missing, and offer a controlled continuation when appropriate. Silently resetting the budget on retry defeats the control.

Use human in the loop when impact, ambiguity, policy, or irreversibility requires judgment beyond deterministic rules. Place the review before the first consequential side effect. A person who reviews a confirmation message after an account was already closed cannot prevent the closure.

The review package should show:

• the exact proposed action and target;
• supporting evidence and source provenance;
• relevant uncertainty, conflicts, and missing fields;
• policy checks already performed;
• alternatives, including reject and edit;
• what happens on timeout.

Approval should be specific and expiring. “Approve this agent” is too broad. “Approve refund of $24.50 for transaction T-82 within ten minutes” can be verified at execution. If the amount or target changes, require new approval.

Human review is not a universal cure. Reviewers can become overloaded, approve by habit, or lack access to evidence. Measure review quality and workload. Automate low-risk checks while reserving human attention for decisions where it changes outcomes.

Assume some runs will behave unexpectedly. Containment controls reduce blast radius:

• per-run sandboxes and tenant isolation;
• rate limits and transaction caps;
• canary release to a small task slice;
• feature flags for tools or autonomy levels;
• anomaly alerts and a kill switch;
• reversible actions and tested rollback;
• append-only, access-controlled audit events.

A kill switch must operate outside the agent loop. If disabling an email tool requires asking the same agent to stop using it, the control is not independent. Operators should be able to revoke credentials, pause queues, or deny execution centrally.

Monitor meaningful indicators: repeated denied calls, unusual destinations, rapid budget consumption, duplicate side effects, rising escalation rates, and behavior changes after model or tool updates. A log that nobody can query during an incident offers little protection.

Consider three failures.

First, a browsing agent follows a page instruction and attempts to reveal a secret. Root cause: untrusted content influenced action selection. Containment: the secret should not be in model context, and no allowed tool should reveal it.

Second, a scheduling agent sends 200 invitations after interpreting a distribution list as one recipient. Root cause: target expansion and scale were not validated. Containment: preview recipient count, cap sends, and require approval above a threshold.

Third, a purchasing agent retries after a timeout and places two orders. Root cause: uncertain side-effect state and no idempotency. Containment: use idempotency keys, reconcile order status, and stop for review when execution outcome is unknown.

In each case, “tell the agent to be careful” is insufficient. The correction belongs in permissions, policy, execution, or recovery.