How to evaluate RAG quality

Measure retrieval and answers separately — a eloquent lie is still a failure.

1. Write **20–50 golden questions** from real users

2. For each, note the **expected source doc/section**

3. Run retrieval only — did the right chunk appear in top-k?

4. Run full RAG — is the answer **supported** by retrieved text?

5. Track metrics over time as you change chunking/models

| Metric | Question |

|--------|----------|

| Retrieval hit@k | Is gold chunk in top k results? |

| Faithfulness | Is every claim in the answer backed by a chunk? |

| Refusal rate | Does it say "I don't know" when nothing matches? |

| Latency / cost | Acceptable for your product? |

Ask questions **outside** your corpus — bot should refuse, not invent.

**Try the lessons:** `eval-and-benchmarks` · `rag-pipeline-steps`