Page 6 of 8~128 min topic

Capstone: ship a tiny AI app

Instrument the grounded café FAQ application

Metrics for gold_pass and demo_seconds must distinguish bad input from component failure for café ops lead answering hours/wifi/pets questions from a kiosk.

~16 min this pageTesting and observability

1Learn the idea

Read

Golden signals for this system

Instrument grounded café FAQ app with validation, five-note retrieval, citations, tests, demo so café ops lead answering hours/wifi/pets questions from a kiosk can answer: demand, errors, latency/age, saturation. Emit fields needed by gold_pass and demo_seconds with bounded labels. Sample successful high-volume traces; keep errors and rollout transitions denser within policy.

Read

Alert path worth paging

Define at least one alert that would fire for CAP-CAFE-DEMO-01, with a for/pending window that survives deploy blips. Missing scrape or missing revision labels must not look like health. Include a trace/log example id format you will actually search.

Read

Implementation artifact

print({"event": "faq_ask", "citation": citation, "abstain": abstain})

Read

Tests for telemetry

Add a unit/integration check that metrics increment on the happy path and on the demo relies on personal .env and unreproducible notebook state branch. Store machine-readable output in CI artifacts when practical.

Read

Stage depth

Cardinality discipline: tenant and revision are usually enough; raw question text is not a label. Exemplars or trace links beat screenshots alone when debugging CAP-CAFE-DEMO-01. Define who owns alert fatigue review. If you export to a vendor, record retention and access. Synthetic probes should use non-sensitive fixtures and still exercise CLI faq_ask. Practice the query you will type at 2am once, while calm.

Read

Field notes for `capstone-ship-it` / `observability`

Document the exact PromQL or log query in the runbook stub for this service. Verify histograms have buckets around your SLO target. Add a canary synthetic check that exercises the oracle path every few minutes in staging. Confirm that PII redaction happens before export. Track build/version as a label on the golden signals. Delete noisy debug logs before they become accidental product dependencies. In this chapter the product is grounded café FAQ app with validation, five-note retrieval, citations, tests, demo, the human stakeholder is café ops lead answering hours/wifi/pets questions from a kiosk, and the incident id you design against is CAP-CAFE-DEMO-01. Re-state the oracle in your notes — gold 5/5 cited + 3/3 abstain; demo under 120s from clean checkout — and keep the invariant visible: unsupported questions abstain; every answer cites a note id; two-minute clean-room demo. Track gold_pass and demo_seconds as the scoreboard. Surface under change control: CLI faq_ask. If you only have forty minutes, finish the fixture for demo relies on personal .env and unreproducible notebook state before polishing UI. Promotion language stays ternary: promote, hold, or roll back based on evidence, not hope.

Go deeper

Before you start

Why this matters

Name the dashboard row or log line café ops lead answering hours/wifi/pets questions from a kiosk opens first during CAP-CAFE-DEMO-01. It must include a correlation id and a bounded label from gold_pass and demo_seconds. If telemetry is missing, write whether you promote, hold, or roll back — and why hold is the default.

Capstone metrics (gold_pass and demo_seconds) appear in the final evidence pack.

In the wild

See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.

Check your understanding

Page assessment

Answer from memory. Completion is saved from this evidence, not from opening the next page.

1. Can you jump from alert to CAP-CAFE-DEMO-01-class evidence?
2. Do labels stay low-cardinality?
3. Is missing telemetry treated as hold?

All responses are required.