Page 6 of 8~120 min topic

Fine-tuning lab

Instrument training runs and serve-time adapter IDs

Metrics for tone_pass_rate and unsupported_fact_rate must distinguish bad input from component failure for support ops manager reviewing tone before a seasonal campaign.

~15 min this pageTesting and observability

1Learn the idea

Read

Golden signals for this system

Instrument LoRA adapter that keeps support replies in brand voice while facts stay in RAG so support ops manager reviewing tone before a seasonal campaign can answer: demand, errors, latency/age, saturation. Emit fields needed by tone_pass_rate and unsupported_fact_rate with bounded labels. Sample successful high-volume traces; keep errors and rollout transitions denser within policy.

Read

Alert path worth paging

Define at least one alert that would fire for FT-PROMO-LEAK-12, with a for/pending window that survives deploy blips. Missing scrape or missing revision labels must not look like health. Include a trace/log example id format you will actually search.

Read

Implementation artifact

metrics = {"tone_pass_rate": 0.84, "unsupported_fact_rate": 0.03, "adapter": "lora-support-v3"}
assert metrics["tone_pass_rate"] >= 0.82 and metrics["unsupported_fact_rate"] <= 0.05

Read

Tests for telemetry

Add a unit/integration check that metrics increment on the happy path and on the training set includes tomorrow's promo price; model invents it after promo ends branch. Store machine-readable output in CI artifacts when practical.

Read

Stage depth

Cardinality discipline: tenant and revision are usually enough; raw question text is not a label. Exemplars or trace links beat screenshots alone when debugging FT-PROMO-LEAK-12. Define who owns alert fatigue review. If you export to a vendor, record retention and access. Synthetic probes should use non-sensitive fixtures and still exercise POST /v1/fine_tuning/jobs. Practice the query you will type at 2am once, while calm.

Read

Field notes for `fine-tuning-lab` / `observability`

Document the exact PromQL or log query in the runbook stub for this service. Verify histograms have buckets around your SLO target. Add a canary synthetic check that exercises the oracle path every few minutes in staging. Confirm that PII redaction happens before export. Track build/version as a label on the golden signals. Delete noisy debug logs before they become accidental product dependencies. In this chapter the product is LoRA adapter that keeps support replies in brand voice while facts stay in RAG, the human stakeholder is support ops manager reviewing tone before a seasonal campaign, and the incident id you design against is FT-PROMO-LEAK-12. Re-state the oracle in your notes — held-out tone score ≥ 0.82 and factual hallucination rate ≤ 0.05 on 40 gold pairs — and keep the invariant visible: weights never encode store hours or prices; those stay in retrieval docs. Track tone_pass_rate and unsupported_fact_rate as the scoreboard. Surface under change control: POST /v1/fine_tuning/jobs. If you only have forty minutes, finish the fixture for training set includes tomorrow's promo price; model invents it after promo ends before polishing UI. Promotion language stays ternary: promote, hold, or roll back based on evidence, not hope.

Go deeper

Before you start

Why this matters

Name the dashboard row or log line support ops manager reviewing tone before a seasonal campaign opens first during FT-PROMO-LEAK-12. It must include a correlation id and a bounded label from tone_pass_rate and unsupported_fact_rate. If telemetry is missing, write whether you promote, hold, or roll back — and why hold is the default.

Check your understanding

Page assessment

Answer from memory. Completion is saved from this evidence, not from opening the next page.

1. Can you jump from alert to FT-PROMO-LEAK-12-class evidence?
2. Do labels stay low-cardinality?
3. Is missing telemetry treated as hold?

All responses are required.