Change management

Start by identifying the components that determine behavior:

• provider and model version;
• fine-tuning or adaptation version;
• system and developer prompts;
• generation settings and safety configuration;
• retrieval index, source documents, embedding model, and ranking logic;
• tools, permissions, business rules, and downstream integrations;
• user interface, disclosures, and review workflow;
• evaluation suite and scoring rubric;
• monitored population, geography, language, and use;
• data processing, logging, and retention configuration.

Together these form a governed configuration. Give releases a system-level identifier that points to the exact component versions. “We use Model X” is not enough to reproduce or investigate behavior.

Vendor-managed models create a special challenge. A provider may update behavior behind a stable name. Prefer pinned versions when available, monitor provider notices, include notification and evidence terms in procurement, and run recurring canary tests. If a component cannot be pinned, record that uncertainty and strengthen detection and fallback.

Not every change needs a full approval cycle. Create categories based on likely effect, not file size.

Routine changes have low expected behavioral impact and stay within approved scope—for example, correcting a typo in a non-directive interface label. They may need peer review and a normal release record.

Material changes can affect quality, data, users, controls, or known failure modes. Examples include changing a prompt instruction, adding knowledge sources, adopting a new model version, adjusting thresholds, expanding to a new language, or modifying a human-review queue. They need targeted evaluation and owner approval.

Major changes alter purpose, decision authority, affected population, sensitive data, tool permissions, or risk classification. Examples include moving from drafting to automatic sending, using the assistant for employee decisions, or granting refund authority. They require renewed risk assessment, specialist review, broader evaluation, and formal approval.

Define examples and thresholds in advance. A numeric trigger might include a planned volume increase, new data category, new external population, or meaningful regression. Leave room for judgment, but do not make every team negotiate the process from scratch.

A useful change proposal states:

1. what is changing and why;
2. which governed components and users are affected;
3. expected benefits and new or altered failure modes;
4. data, privacy, security, legal, and human-workflow effects;
5. tests to run and acceptance thresholds;
6. rollout, monitoring, and rollback plans;
7. required reviewers and final approver;
8. documentation and training updates;
9. links to evidence and the resulting release identifier.

The record should distinguish an assumption from evidence. “No user impact expected” is an assumption until supported by comparison tests and rollout signals.

Reviewers should see differences, not only the new state. Compare old and new prompts, models, data sources, permissions, test results, and known limitations. Aggregate scores can remain stable while individual behaviors change, so inspect regressions and newly introduced failures.

A staged rollout limits exposure while creating evidence. A common sequence is offline comparison, internal use, shadow mode, small canary, limited population, then broader release.

In shadow mode, the candidate processes real inputs but does not affect users; results are compared with the current process under appropriate privacy controls. In a canary, a small share of eligible traffic receives the candidate. Each stage should have entry criteria, duration, monitored signals, an owner, and stop conditions.

Do not let rollout become automatic expansion. Someone should review evidence and authorize each meaningful step. Define rollback triggers before launch: critical policy violation, sensitive data exposure, severe metric regression, complaint spike, tool misuse, or inability to observe the system.

Rollback must be technically possible and operationally practiced. Preserve the prior known-good configuration, compatible data and interfaces, deployment instructions, and authority to act. If a database or index migration cannot be reversed, design a forward recovery or kill switch.

Incidents sometimes require immediate prompt, model, rule, or access changes. An emergency path can shorten approval, but it should not erase accountability.

Require a named incident owner, reason, exact change, risk check, minimum critical tests, approver, monitoring window, and expiry or follow-up date. After containment, complete retrospective evaluation and documentation. Temporary controls have a habit of becoming permanent; expiration forces a deliberate decision.

Avoid fixing one symptom while widening another risk. A prompt that refuses more aggressively may reduce harmful answers but block legitimate users. A filter that removes personal details may also remove facts needed for a correct decision. Emergency changes still need proportionate testing.

When a release changes approved behavior, update the system card, inventory, risk record, monitoring thresholds, runbooks, user notices, and reviewer guidance. Train affected operators before the release reaches them. A changed interface or escalation category can invalidate a control even if model quality improves.

Audit periodically for configuration drift: unrecorded prompt edits, stale indexes, changed provider aliases, expanded permissions, disabled alerts, expired exceptions, or deployments that do not match the approved release. Technical enforcement helps. Restrict production changes, require reviewed configuration, capture immutable release metadata, and block deployment when required evidence is missing.

Change management is successful when the team can answer: what changed, who approved it, what evidence supported it, who received it, how it is performing, and how to reverse it.