Practice: choose and evaluate a stack

Write the task before comparing technology. Name the corpus size, growth rate, languages, content length, update frequency, query volume, latency target, access filters, and consequence of a bad result. Define whether the system supports discovery, duplicate detection, question answering, or an automated decision.

Then define relevance. For “E1047 after firmware update,” a general restart page may be partially useful, while the product-specific E1047 procedure is fully relevant. A document for another customer’s private environment is never eligible, regardless of semantic score. Relevance judgments should encode these distinctions.

Collect representative queries from privacy-reviewed logs, support experts, and deliberately difficult scenarios. Avoid evaluating only polished questions written by the development team. Include typos, short queries, multilingual queries, exact identifiers, no-answer cases, and changing policies.

Split development and test sets. Tune chunking, thresholds, and ranking on development examples; reserve held-out queries for the final comparison. Otherwise repeated adjustments can overfit the benchmark.

Filter model candidates by basic constraints:

• language and domain coverage;
• query and document input limits;
• dimensions and storage footprint;
• recommended similarity metric;
• hosted versus self-managed operation;
• throughput, batch support, latency, and price;
• data handling, region, licensing, and version guarantees.

A hosted model reduces infrastructure work but sends content to a provider under its data terms. A local model can improve control and predictable high-volume cost while requiring capacity planning, monitoring, and updates. Neither deployment style is automatically private or secure; assess the actual flow.

Use model cards and public benchmarks to form a shortlist, not to declare a winner. Re-embed the same evaluation corpus with each candidate. Compare under its recommended query mode, normalization, and metric. Do not mix one model’s vectors with another model’s queries.

Estimate storage concretely. One million 768-dimensional vectors stored as 32-bit floats contain about 1,000,000 × 768 × 4 bytes, roughly 3.07 GB before index and metadata overhead. A 1,536-dimensional version doubles the raw vector storage. Actual databases add graph, quantization, replication, and metadata costs.

Exact search compares a query against every eligible vector and returns the true nearest neighbors under the metric. It is simple and can be adequate for small collections, filtered subsets, or offline work. Cost grows with corpus size.

Approximate nearest neighbor indexes reduce latency by searching a promising part of the space. HNSW graph indexes commonly trade memory and build time for fast queries. Inverted-file approaches partition the space and probe selected regions. Product implementations differ, so benchmark the database you will run.

Approximation introduces index recall: whether approximate search returns the neighbors an exact search would. This is not the same as semantic relevance. A model may identify the wrong passage as geometrically nearest even when index recall is perfect. Conversely, a good model can appear worse if aggressive index settings miss its true neighbors.

Measure both. For a sample, compare approximate results with exact results to tune index parameters. Then measure relevance against human judgments. Add realistic filters, concurrency, replicas, and update load to latency tests.

Use several metrics because they answer different questions:

• Recall@k: Did at least one relevant item appear in the top k?
• Precision@k: What fraction of the top k items were relevant?
• MRR: How early did the first relevant item appear?
• nDCG: Were highly relevant items ranked above partially relevant ones?
• No-answer accuracy: Did the system avoid claiming a match when none existed?

Suppose 200 answerable queries are tested. System A places a relevant result in the top five for 184, so Recall@5 is 184 / 200 = 0.92. System B reaches 0.94, but its p95 latency is 700 ms and its Portuguese recall is 0.71. The aggregate improvement does not settle the choice.

Slice results by language, product, query length, exact-code presence, recency, and risk. Review individual failures. A two-point metric gain may come from easy navigational queries while critical troubleshooting becomes worse.

For RAG, also evaluate whether retrieved passages contain the required evidence and whether generated answers cite and use it faithfully. A fluent answer score can hide weak retrieval; retrieval recall can hide unsupported generation.

Model cost includes initial corpus embedding, incremental updates, queries, retries, and future re-embedding during migrations. Index cost includes storage, replicas, backups, compute, and network transfer. Reranking adds latency and inference spend but can allow a smaller first-stage model to perform well.

Run load tests using realistic vector filters and candidate counts. Record p50, p95, and p99 latency rather than an average. Test ingestion while queries run. Verify deletion, permission updates, backup restore, and rebuild time. A stack that scores well but takes a week to remove revoked content may be unacceptable.

Plan versioning. Put a model or pipeline version on each collection. Build a new index alongside the old one, replay the evaluation suite, send limited traffic, compare outcomes, and retain rollback until the new path is stable. In-place mixing makes failures difficult to reverse.

Create minimum gates before seeing results. For example:

• Recall@5 at least 0.90 overall and 0.85 in every supported language;
• no permission-filter failures in security tests;
• no-answer precision at least 0.95;
• p95 retrieval under 250 ms at expected peak load;
• estimated monthly cost below the approved limit;
• exact error-code tests no worse than the lexical baseline.

Then compare viable configurations, including a keyword baseline and hybrid search. The simplest stack meeting requirements may be better than the absolute top scorer because it is easier to operate and debug.

Document why the chosen model, dimension, metric, chunking, index, filters, and reranker fit the workload. Record known weak slices and fallback behavior. A decision memo makes future model upgrades an evidence-based comparison rather than a restart.