In Part 1 we saw the eight-step RAG pipeline. This part zooms into the second box — chunking — and it's the one I'd urge you to take most seriously. Chunking is where a lot of "our RAG just isn't accurate" problems are actually born. The good news: the ideas are intuitive once you see them, and getting them roughly right takes you most of the way.
Why we can't just store whole documents
Imagine your knowledge base is a 40-page employee handbook. Why not embed the entire document as one searchable item? Two problems:
- A whole document is about too many things. The handbook covers refunds, leave, security, expenses, dress code… If you squeeze all of that into one "meaning summary," it becomes a vague blur that matches everything weakly and nothing strongly.
- You'd hand the model far too much. Even if you found the right document, pasting 40 pages into the prompt is slow, expensive, and distracting. You want to hand over the one relevant paragraph, not the whole book.
So we slice the document into focused passages. Each chunk should ideally be about one thing, so that when someone asks about refunds, the refund chunk lights up clearly:
Each chunk is about a single topic, so it can match a question crisply.
The two dials: chunk size and overlap
Almost every chunking decision comes down to two numbers. Understand these and you understand chunking.
Dial 1 — chunk size
How much text goes in each chunk (usually measured in tokens — roughly ¾ of a word each, but you can think "words" for intuition). This is a genuine trade-off with no free lunch:
| Chunks too small | Chunks too large | |
|---|---|---|
| What happens | A single idea gets split across pieces; each piece lacks context. | Many unrelated ideas share one chunk; its "meaning" gets muddy. |
| Retrieval effect | The right answer is fragmented — you may retrieve half of it. | Matches are vague; you drag in irrelevant text alongside the answer. |
| Example | "Faulty items may be returned" and "within 30 days" land in different chunks. | The refund rule is buried in a chunk that also covers leave and expenses. |
A common, sensible starting range is roughly 200–500 tokens per chunk (about 150–400 words). It's big enough to hold one complete thought, small enough to stay focused. You tune from there based on your content.
Dial 2 — overlap
If you cut a document into hard, non-overlapping slices, you'll sometimes slice through the middle of a sentence or idea — and the answer to a question ends up split across the boundary. Overlap fixes this by repeating a little text from the end of one chunk at the start of the next, so important sentences are never orphaned at a cut line.
The highlighted sentence appears at the end of chunk 7 and the start of chunk 8, so whichever one is retrieved, the "30 days" rule travels with it.
A typical overlap is about 10–20% of the chunk size — e.g. a 50-token overlap on a 300-token chunk. Too little and you risk severed ideas; too much and you store a lot of duplicate text, inflating cost and index size. Modest overlap is cheap insurance.
The four chunking strategies
How you decide where to cut matters as much as the size. There's a ladder of strategies, from naive to smart.
1. Fixed-size chunking
The simplest approach: cut every N tokens regardless of what the text says. Count to 300, cut, repeat. It's fast and predictable, but blind — it happily slices mid-sentence or mid-table because it doesn't understand structure. With overlap it's a perfectly reasonable baseline, and it's where many people start.
2. Recursive (structure-aware) chunking
The workhorse of real systems. Instead of cutting blindly, it tries to split on natural boundaries in priority order: first paragraphs (double newlines), then single newlines, then sentences, then — only if a piece is still too big — words. It "recurses" down to finer boundaries until each chunk fits the target size. The result respects the document's natural seams, so chunks rarely break mid-thought.
["\n\n", "\n", ". ", " "] is the sensible default for most documents — prose, wikis, policies, manuals. If you're unsure which strategy to pick, start here.
3. Document-structure (layout-aware) chunking
For richly formatted content — Markdown, HTML, PDFs with headings and tables — you can split along the real structure: by heading section, by table, by list. A Markdown splitter might keep each ## section as a chunk and even attach the heading text to the chunk so its topic is explicit. This preserves meaning beautifully, but needs content that actually has clean structure to exploit.
4. Semantic chunking
The most advanced: instead of using size or punctuation, it uses meaning. It embeds sentences and starts a new chunk whenever the topic visibly shifts (when consecutive sentences stop being similar). Chunks then align with ideas rather than arbitrary lengths. It's powerful for messy, unstructured text — but it's slower and costs more, because you're running an embedding model just to decide the cut points.
Fixed-size
Simple & fast. Good baseline with overlap. Risks blind cuts.
Recursive
Splits on natural boundaries. The pragmatic default for most text.
Structure-aware
By heading, table, or list. Best for Markdown, HTML, structured PDFs.
Semantic
Cuts where the topic shifts. Most accurate, slowest, costs more.
Don't forget metadata
A chunk shouldn't travel alone. Alongside its text, store useful metadata: the source file name, page number, heading, author, date, and security/permission tags. This pays off later in two big ways. First, you can filter retrieval ("only chunks from current HR policies") — we'll use this in Part 5. Second, you can show citations, so the model's answer links back to "Handbook, p.12." Metadata is what turns a black-box answer into a trustworthy, traceable one.
A worked before-and-after
To make the payoff concrete, here's the same question against badly and well chunked content.
Practical defaults to start with
- Strategy: recursive splitting on paragraphs → sentences. Upgrade to structure-aware if your content has clean headings.
- Size: ~300 tokens (try 256–512 depending on how dense your text is).
- Overlap: ~10–20% (e.g. 50 tokens).
- Metadata: source, page/section, date, and permissions on every chunk.
- Then test: run real user questions and read what got retrieved. If answers miss context, increase size or overlap; if they're vague, decrease size. Chunking is iterative, not one-and-done.
What's next
Now that our document is sliced into clean chunks, we need to make them searchable by meaning. That's the job of Part 3: Embeddings & Vectorization — how a piece of text becomes a list of numbers, why similar meanings end up with similar numbers, and how that quietly powers the whole "search without keywords" trick. We'll visualise it, promise.