Fine-tuning lab
Instrument training runs and serve-time adapter IDs
Metrics for tone_pass_rate and unsupported_fact_rate must distinguish bad input from component failure for support ops manager reviewing tone before a seasonal campaign.
1Learn the idea
Read
Golden signals for this system
Instrument LoRA adapter that keeps support replies in brand voice while facts stay in RAG so support ops manager reviewing tone before a seasonal campaign can answer: demand, errors, latency/age, saturation. Emit fields needed by tone_pass_rate and unsupported_fact_rate with bounded labels. Sample successful high-volume traces; keep errors and rollout transitions denser within policy.
Read
Alert path worth paging
Define at least one alert that would fire for FT-PROMO-LEAK-12, with a for/pending window that survives deploy blips. Missing scrape or missing revision labels must not look like health. Include a trace/log example id format you will actually search.
Read
Implementation artifact
metrics = {"tone_pass_rate": 0.84, "unsupported_fact_rate": 0.03, "adapter": "lora-support-v3"}
assert metrics["tone_pass_rate"] >= 0.82 and metrics["unsupported_fact_rate"] <= 0.05
Read
Tests for telemetry
Add a unit/integration check that metrics increment on the happy path and on the training set includes tomorrow's promo price; model invents it after promo ends branch. Store machine-readable output in CI artifacts when practical.
Read
Stage depth
Cardinality discipline: tenant and revision are usually enough; raw question text is not a label. Exemplars or trace links beat screenshots alone when debugging FT-PROMO-LEAK-12. Define who owns alert fatigue review. If you export to a vendor, record retention and access. Synthetic probes should use non-sensitive fixtures and still exercise POST /v1/fine_tuning/jobs. Practice the query you will type at 2am once, while calm.
Read
Field notes for `fine-tuning-lab` / `observability`
Document the exact PromQL or log query in the runbook stub for this service. Verify histograms have buckets around your SLO target. Add a canary synthetic check that exercises the oracle path every few minutes in staging. Confirm that PII redaction happens before export. Track build/version as a label on the golden signals. Delete noisy debug logs before they become accidental product dependencies. In this chapter the product is LoRA adapter that keeps support replies in brand voice while facts stay in RAG, the human stakeholder is support ops manager reviewing tone before a seasonal campaign, and the incident id you design against is FT-PROMO-LEAK-12. Re-state the oracle in your notes — held-out tone score ≥ 0.82 and factual hallucination rate ≤ 0.05 on 40 gold pairs — and keep the invariant visible: weights never encode store hours or prices; those stay in retrieval docs. Track tone_pass_rate and unsupported_fact_rate as the scoreboard. Surface under change control: POST /v1/fine_tuning/jobs. If you only have forty minutes, finish the fixture for training set includes tomorrow's promo price; model invents it after promo ends before polishing UI. Promotion language stays ternary: promote, hold, or roll back based on evidence, not hope.
Go deeper
Before you start
Why this matters
Name the dashboard row or log line support ops manager reviewing tone before a seasonal campaign opens first during FT-PROMO-LEAK-12. It must include a correlation id and a bounded label from tone_pass_rate and unsupported_fact_rate. If telemetry is missing, write whether you promote, hold, or roll back — and why hold is the default.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.