Capstone: support bot with RAG + tools
Instrument metrics and rollout decisions
Build a support API that retrieves versioned policy, reads live order state through a scoped tool, and returns a cited answer or human escalation as an operable release, not a slide-deck example.
1Try it yourself
Playground
Support bot: RAG + tool
Retrieve policy text, optionally fetch live order status, then answer with citations.
User: Can I refund order #8821 on my annual plan?
- → Retrieve FAQ chunk
- ○ Call order lookup tool
- ○ Generate cited reply
Before you start
Why this matters
Before changing code, write the single production outcome this chapter must prove and the signal that would stop you. For this lab, the service boundary is SupportRequest(user_id, message) produces Answer(text, citations, order_facts, escalation) with every claim attributable to policy or tool output. Record one request identifier you can follow from ingress through the final decision. If you cannot name the owner of the stop decision, the rollout is not yet controlled.
The source architecture is retrieve policy, call a live-data tool, produce a cited reply, and escalate on no-match, anger, or fraud. This chapter turns that compact lesson into implementation evidence. The running scenario is a support API that retrieves versioned policy, reads live order state through a scoped tool, and returns a cited answer or human escalation. You will keep the same scenario across all eight chapters so setup decisions, tests, telemetry, and rollback controls accumulate into one coherent system rather than eight disconnected exercises.
2Learn the idea
Read
Instrument the decisions
Emit one structured event at each control point: request accepted, policy or variant selected, dependency completed, result produced, and rollback or promotion changed. Every event needs timestamp, request ID, release revision, configuration revision, outcome category, and duration. Add only low-cardinality dimensions to metrics; put request IDs in traces and logs, not metric labels.
The primary dashboard reports task resolution, citation correctness, tool authorization failures, escalation precision, p95 latency, and cost per resolved conversation. Show numerator and denominator, cohort or revision splits, sample count, and a comparable control. Add saturation and dependency health so operators can tell a bad release from a shared outage. Quality metrics need dataset or grader revision next to the score.
Read
Set actionable alerts
Page on symptoms that require immediate action: sustained availability loss, rapid error-budget burn, security rejection anomaly, or queue age threatening the stated objective. Create tickets for slow trends. Every alert links to a runbook and includes the affected revision or cohort. Test alert queries against a synthetic event before trusting them.
Avoid averages that erase harm. Compare p50 and p95, inspect tenant or cohort slices with privacy-safe cardinality, and display absolute counts beside rates. Low traffic can make a zero-error canary meaningless. The dashboard must visibly mark insufficient data and stale series; rollout automation must treat both as hold.
Read
Correlate without leaking
Follow this policy: authenticate the user, authorize each order lookup, minimize transcript PII, treat retrieved text as data rather than instructions, and require humans for refunds. Redact at collection, not only in the dashboard. Sample successful high-volume traces, but retain errors and rollout transitions at a higher rate within policy. Restrict access to sensitive operational data and set retention intentionally.
Run a trace exercise using one fixture. Starting from its public request ID, locate the selected policy, external calls, final disposition, and emitted metric within five minutes. Then start from a breached chart and find representative traces. Both directions matter during an incident.
Read
Turn metrics into a release decision
Populate the gate function from the validation chapter using a fixed observation window. Require minimum traffic and complete guardrails. Record the output and reasons as a deployment event. A human release commander reviews quality examples before promotion even when numeric gates pass. The completed observability package includes dashboard definition, alert tests, trace evidence, and a decision record.
Read
Encode the release signal
Define the alert as versioned configuration, including a minimum window and explicit no-data behavior. Labels remain low cardinality; conversation_id belongs in traces, not metric labels.
alert: capstone-support-bot-support_tool_authorization_failures
expr: support_tool_authorization_failures > 0
for: 5m
labels:
severity: page
owner: building-labs
annotations:
action: hold rollout and inspect revision
no_data: hold
A representative event connects the chart to one decision without prompts, secret bytes, or raw customer identifiers. The local synthetic probe should emit this shape:
{
"event": "release_guardrail_evaluated",
"revision": "support-bot-v4",
"config_revision": "local-fixture-1",
"outcome": "hold",
"reason": "support_tool_authorization_failures_threshold",
"sample_count": 1200,
"duration_ms": 37
}
Validate with one below-threshold and one above-threshold sample. Expected behavior is inactive first and firing only after five continuous minutes second; missing samples must hold. If events exist but the graph is empty, compare metric names and labels before changing the query. If no trace can be found, verify identity propagation and ensure it was not made a high-cardinality metric label.
Checking tutor…