Reference · How-to · ~12 min
How to embed documents in batch
Last updated
Index many chunks efficiently — **chunk once, embed in batches, store with metadata**.
Index many chunks efficiently — **chunk once, embed in batches, store with metadata**.
Steps
1. **Load docs** — markdown, PDF text, HTML; strip boilerplate
2. **Chunk** — ~500 tokens, 50-token overlap; keep source filename
3. **Batch strings** — 50–200 chunks per API call (watch token limits)
4. **Call embedding API** — same model you'll use at query time
5. **Store** — vector + text + `{source, chunk_id, page}`
6. **Verify** — embed one known sentence; nearest neighbor should be its chunk
Batch loop (pseudo)
for batch in chunks_in_batches(all_chunks, size=100):
vectors = embed_api(batch.texts)
db.upsert(zip(batch.ids, vectors, batch.texts, batch.meta))