RAG Chunking Strategies 2026: Fixed Size, Recursive and Semantic Splitting Compared
The chunk size is the most-tweaked parameter in RAG and the one most people set by gut feel. What do the chunking strategies actually change, and how do you pick yours with data?
💡 What You Will Learn
The chunk size is the most-tweaked parameter in RAG and the one most people set by gut feel. What do the chunking strategies actually change, and how do you pick yours with data?
📜 Table of Contents
Why Chunking Decides RAG Quality
Retrieval is only as good as the units it retrieves. Chunks too small lose context (a chunk cannot see the paragraph before it); chunks too big dilute relevance (half the chunk is off-topic, the embedding averages it out). Both failures lower retrieval precision.
The Three Main Strategies (2026)
- Fixed-size splitting - split every N characters with an overlap window. Fast, simple, predictable. The overlap (10-20%) prevents cutting sentences in half from losing meaning. Baseline for everything.
- Recursive splitting - split by paragraph first, then sentence, then character, respecting document structure. LangChain's RecursiveCharacterTextSplitter is the default for a reason: it keeps paragraphs and sentences whole.
- Semantic splitting - use an embedding model to detect topic boundaries and cut there (e.g. LlamaIndex SemanticSplitterNodeParser). Best quality, slowest and most expensive - you embed every candidate boundary.
The Evidence on Size
Community benchmark results (Anthropic's chunking research, RAGAS-based studies) consistently show:
- 256-512 tokens per chunk is the practical sweet spot for most QA workloads
- Long-context models reduce but do not eliminate the need for good chunking
- Overlap of 10-20% measurably improves retrieval for sentence-based documents
The Deeper Question: Chunking for What?
Chunking strategy should follow your query type: - Fact lookup (what is the refund policy): small chunks, high precision - Summarization (summarize this contract): whole sections, structure-aware - Comparison (how do plan A and B differ): chunks that each contain one complete point
How to Choose With Data, Not Vibes
- Build 50-100 real questions with the chunk that contains each answer.
- Split the same docs with 3 strategies (fixed 256, recursive, semantic).
- Measure retrieval hit rate - did the correct chunk make it into the top 3?
- Pick the winner; if semantic splitting wins by less than 5%, use recursive (cheaper and faster).
The 2026 Trend
Context engineering is absorbing chunking: instead of blindly tuning sizes, teams design chunks as self-contained units with titles and metadata (see the context engineering guide). The chunk becomes a document, not a slice.
