Frame the café FAQ retriever experiment

Success is not “I followed the tutorial.” Success is producing evidence that: overlap retrieval returns the right note or abstains on zero overlap. The accepted input is narrow on purpose: five short documents with stable IDs plus one normalized user question. That narrowness is what lets you inspect every field and prevents a toy demo from being narrated as a production system.

Record the baseline you must beat: always-answer-from-largest-document heuristic. If the finished artifact cannot beat that baseline on the fixture below, stop and revise the claim before writing more code.

docs={'hours':'Cafe open 8am to 6pm weekdays.','wifi':'Ask staff for the Wi-Fi card.','pets':'Service animals are welcome.'}
print(sorted(docs),len(docs))

Expected evidence: three sorted IDs and count 3. Treat the printout as a claim about this fixture, not as proof that the toolchain merely started.

For the five-document café FAQ retriever, a decorative win often looks like a clean run that never checks retrieval hit rate, citation correctness, supported-answer rate, abstention accuracy. Write the metric down now so later pages cannot redefine success after the fact. Also note the operational threat you will eventually gate on: prompt injection inside a stored FAQ note treated as instructions.

For build-mini-rag, write the claim on a sticky note in this exact shape: “Given five short documents with stable IDs plus one normalized user question, the café FAQ retriever will …”. Fill the ellipsis with the observable part of: overlap retrieval returns the right note or abstains on zero overlap. Tape the baseline beside it: always-answer-from-largest-document heuristic. If someone later replaces your metric with a vibe check, the sticky note is how you push back.

Also sketch the one-sentence user story: a person uses this output to answer a café question only when a relevant local note is retrieved and cite its ID. If that sentence needs a dashboard, a model zoo, or five services, the lab scope is too wide—shrink the fixture (café notes: hours, wifi, pets, allergens, parking) until the story fits on one screen.

Decide now whether live network calls are allowed on page 1. For this lab they usually are not; inventory and contracts should run offline against café notes: hours, wifi, pets, allergens, parking. Note the metric you will eventually require (retrieval hit rate, citation correctness, supported-answer rate, abstention accuracy) so page 4 cannot invent a softer target. The characteristic failure to keep in mind is retrieval miss or ungrounded fluent answer.

At the experiment brief stage for build-mini-rag, the job is narrower than finishing a product demo. You are creating one progressive evidence piece about café notes: hours, wifi, pets, allergens, parking that later pages inherit without redefining success. Keep that fixture small enough to inspect by hand, keep outputs copy-pasteable as text, and refuse to narrate this baseline as if it were a production SLA: always-answer-from-largest-document heuristic.

For this page specifically, success looks like a falsifiable claim and baseline written before coding while still centering the user decision to answer a café question only when a relevant local note is retrieved and cite its ID. If you cannot point to a file, command, or assertion that proves that for the café FAQ retriever, stay on this page instead of advancing.

How-to: build a 5-document RAG app · Glossary: RAG

Next