RAG Chunk Size Optimizer

There is no universal optimum — it depends on your documents and queries. Short chunks (256-512 tokens) work well for precise Q&A and fact retrieval. Longer chunks (512-1024 tokens) preserve more context for summarization and reasoning tasks. Start at 512 tokens with 10% overlap and evaluate retrieval quality.

Overlap is the number of tokens shared between adjacent chunks. It prevents important context from being split across chunk boundaries. A 10-20% overlap (e.g., 50-100 tokens for 512-token chunks) is typical. Too much overlap increases index size and retrieval noise; too little loses boundary context.

Smaller chunks improve precision (the retrieved chunk is relevant) but can miss surrounding context. Larger chunks improve recall (the answer is likely inside) but increase noise in the retrieved text sent to the LLM. The LLM's context window limits how many chunks you can pass.

Sentence-window retrieval (also called 'small-to-big') stores small chunks for embedding but retrieves surrounding context at query time. You index 128-token child chunks but return their 512-token parent chunks to the LLM. This balances precision in retrieval with context richness in generation.