Measure whether the café FAQ retriever works

Translate the claim into assertions or a tiny eval harness. The metric to protect is: retrieval hit rate, citation correctness, supported-answer rate, abstention accuracy. Always record the denominator (how many cases) beside any rate. A percentage without a denominator is marketing, not measurement.

gold=[('When is the cafe open?','hours'),('How do I get wifi?','wifi')]
docs={'hours':'cafe open weekdays','wifi':'get wifi card'}
def hit(q): return max(docs,key=lambda k:len(set(q.lower().split())&set(docs[k].split())))
assert all(hit(q)==want for q,want in gold); print('hit_rate',1.0)

Expected evidence: retrieval miss or ungrounded answer. A passing assertion proves only the behavior it names; broader usefulness still needs the chapter’s full limits.

Be explicit: beating the baseline (always-answer-from-largest-document heuristic) on this fixture does not prove behavior under retrieval miss or ungrounded fluent answer. Label observations separately from conclusions so the next page inherits honest evidence about the café FAQ retriever.

Compute retrieval hit rate, citation correctness, supported-answer rate, abstention accuracy with the denominator written beside the rate every time. For this chapter, the evaluation set is intentionally tiny; that is allowed only if you say so in the evidence. Compare against always-answer-from-largest-document heuristic before celebrating.

Add one negative case aimed at retrieval miss or ungrounded fluent answer. A suite with only happy cases cannot protect the café FAQ retriever when the characteristic failure appears in review.

If a check is expensive or flaky, shrink it until it is deterministic on café notes: hours, wifi, pets, allergens, parking. Flaky green builds teach the team to ignore gates. Record what this page does not prove so security-ops and mastery-ship inherit honest limits.

At the evaluation stage for build-mini-rag, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about café notes: hours, wifi, pets, allergens, parking that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: always-answer-from-largest-document heuristic.

For this page specifically, success looks like metrics with explicit denominators and a negative case while still centering the user decision to answer a café question only when a relevant local note is retrieved and cite its ID. If you cannot point to a file, command, or assertion that proves that for the café FAQ retriever, stay on this page instead of advancing.

How-to: build a 5-document RAG app · Glossary: RAG

Previous · Next

For build-mini-rag, write a transfer example that differs in one constraint from the chapter scenario. Keep the quality bar fixed. Explain which check still applies.

For build-mini-rag, write a transfer example that differs in one constraint from the chapter scenario. Keep the quality bar fixed. Explain which check still applies.

For build-mini-rag, write a transfer example that differs in one constraint from the chapter scenario. Keep the quality bar fixed. Explain which check still applies.