Retrieval Strategy Picker

BM25 is a keyword-based ranking algorithm (like TF-IDF, used in Elasticsearch and Lucene). It matches exact terms and handles rare words well but misses synonyms and semantic meaning. Dense vector retrieval uses neural embeddings to encode meaning — 'automobile' and 'car' have similar vectors. Dense search finds semantically relevant results even with different vocabulary, but can miss exact rare terms.

Hybrid retrieval combines BM25 keyword scores with dense vector scores (via reciprocal rank fusion or linear interpolation). It gets the best of both worlds: exact keyword precision + semantic understanding. Use hybrid when your queries include both natural language questions AND specific technical terms, IDs, or domain jargon.

ColBERT (late interaction) encodes each token separately and computes fine-grained token-level similarity at query time. It achieves higher precision than single-vector dense search, especially for long documents. The tradeoff is larger storage (one vector per token vs one per document) and slower retrieval. Worth it for high-stakes domains like legal, medical, or academic research.

For small corpora (<10K documents), you can store embeddings in memory (numpy arrays, SQLite with sqlite-vec). For production scale, a dedicated vector DB (Qdrant, Weaviate, Pinecone, Milvus) provides ANN (approximate nearest neighbor) indexing, filtering, and scalability. Many support hybrid search natively.

For multilingual search, use a multilingual embedding model like 'multilingual-e5-large' or 'paraphrase-multilingual-mpnet-base-v2'. These encode text from 50-100+ languages into the same embedding space. BM25 requires language-specific tokenizers and stop-word lists for each language. Dense retrieval is usually simpler to deploy for multilingual use cases.