chunk size optimization

Chunk size optimization balances context completeness with retrieval precision in RAG systems. **Trade-offs**: **Small chunks** (100-200 tokens): Precise retrieval, less noise, but may split context, multiple chunks needed, embedding overhead. **Large chunks** (1000+ tokens): Complete context, fewer chunks, but less precise retrieval, may include irrelevant content. **Factors to consider**: Document type (structured vs narrative), query patterns (specific vs broad), embedding model context limits, LLM context window. **Empirical guidance**: 256-512 tokens often optimal for general use, technical docs may prefer smaller (more precise), narratives may prefer larger (maintain flow). **Dynamic chunking**: Vary size based on content structure (section boundaries, paragraphs). **Evaluation approach**: Test multiple sizes on representative queries, measure retrieval recall and answer quality. **Relationship with overlap**: Overlap mitigates splitting issues for any chunk size. **Semantic chunking**: Use LLM/heuristics to chunk at semantic boundaries rather than fixed sizes. **Best practice**: Start with 400-500 tokens, 50-100 overlap, tune based on evaluation results.