Measure quality, not just speed

Measure the workflow from multiple viewpoints:

1. Reply quality: factuality, grounding, completeness, clarity, tone, and authorization.
2. Case outcome: resolution, reopen rate, repeat contact, transfer, and customer effort.
3. Operational performance: handle time, queue age, agent workload, and cost.
4. Risk and control: privacy, missed escalation, unsupported commitment, policy violation, and auditability.

No single metric represents all four. Customer satisfaction can reflect factors outside the agent’s control. Handle time can reward premature closure. Draft acceptance can reward convenience rather than correctness.

Score representative replies against observable criteria:

• Correct: every factual and policy claim matches current evidence.
• Grounded: important claims trace to applicable sources.
• Complete: all customer questions and required notices are addressed.
• Authorized: remedies, disclosures, and commitments stay within permission.
• Actionable: owner, next step, timing, and fallback are clear.
• Effort-aware: the customer is not asked to repeat known information.
• Appropriate: tone fits the situation without emotional overreach.
• Safe: privacy and escalation rules are followed.

Use a severity scale. A missing greeting is minor. A wrong refund amount is major. Exposing another customer’s data or missing an urgent safety escalation is critical. An average score can conceal rare catastrophic failures, so report severe error rates separately.

Create anchor examples for reviewers. “Complete” should mean the same thing across teams. Have multiple reviewers score a sample and discuss disagreements; otherwise apparent model change may actually be reviewer inconsistency.

Before deployment, build a test set from realistic, minimized, policy-approved cases. Include:

• common intents and languages;
• short, long, messy, and multi-issue tickets;
• missing and conflicting records;
• outdated or irrelevant knowledge;
• policy conditions and exceptions;
• identity, payment, privacy, legal, and safety escalations;
• emotionally charged language and quoted threats;
• cases where the correct result is ask, escalate, or abstain.

Protect the test set from becoming a prompt-writing worksheet. Keep a holdout set and add new cases from incidents and observed failures.

Compare AI-assisted work with the current baseline. Randomize or match cases when possible. Differences in queue difficulty, season, channel, or agent experience can otherwise create misleading gains.

Draft acceptance is only a starting signal. Capture meaningful edits:

• facts corrected or removed;
• policy conditions restored;
• promises weakened;
• escalation added;
• customer questions recovered;
• tone shortened or softened;
• internal information removed;
• next steps clarified.

An accepted draft with one corrected date may be more useful than a rejected draft rewritten for personal style. Track edit type and severity, not just character distance.

Ask agents whether the tool reduces searching, helps them learn policy, or creates review fatigue. If reviewing a plausible draft takes longer than writing from scratch, usage may remain high only because management expects it.

Avoid measuring individual agents punitively from model acceptance. A careful agent may reject more unsafe drafts and appear less “efficient” than someone who trusts them.

Useful outcome measures include first-contact resolution, time to meaningful resolution, reopen rate, repeat contacts for the same issue, transfer count, and customer effort. Segment results by intent and risk level.

Customer satisfaction needs interpretation. A customer denied an ineligible refund may score a correct response poorly. Conversely, an unauthorized concession can receive a high score. Pair feedback with quality review.

Look for displacement. Handle time may fall while supervisor review queues grow. Automated acknowledgments may improve first-response time without changing resolution. Translation may speed replies while increasing correction rates in one language. End-to-end measurement makes these trade-offs visible.

Use sampling based on risk. Review a random baseline plus larger samples for new intents, low-confidence outputs, escalations, high-value remedies, sensitive categories, and changed policies. Maintain a route for agents and customers to report harmful or confusing replies.

Watch for drift:

• policy or product updates;
• changing ticket mix;
• new fraud or abuse patterns;
• model or prompt changes;
• knowledge-index failures;
• seasonal volume and staffing shifts.

Every material change should trigger targeted regression tests. Keep versions of prompts, models, sources, and rules so investigators can reconstruct what produced a reply.

Set stop thresholds before launch. Examples include any confirmed cross-customer disclosure, a critical missed escalation, or a sustained increase in unsupported commitments. The response may be to disable automatic sending, revert a knowledge update, narrow eligible intents, or return to suggestion-only mode.

Estimate value from time saved, reduced repeat contact, lower transfers, improved availability, and avoided errors. Subtract model use, integration, content maintenance, evaluation, review, escalation, and incident costs.

Do not assume every saved draft minute becomes productive capacity. Validate whether queues shrink, service improves, or agents take on higher-value work. Report uncertainty and distribution: a workflow may save three minutes on simple cases but add ten minutes on exceptions.

The strongest business case often combines modest speed gains with better consistency and lower customer effort—not maximum automation.