How you split documents determines how good your RAG system is
Start With a Simple Question
You have a 100-page PDF. You want to build a RAG system on it.
You already know you can't dump the whole PDF into the LLM prompt — too many tokens, too expensive, context window too small.
So you need to split it into smaller pieces first. These smaller pieces are called chunks.
But here's the question nobody thinks about carefully enough:
How do you split it?
This seems like a boring technical detail. It's not. It's one of the most important decisions in your entire RAG system.
The wrong chunking strategy = poor search results = wrong answers = broken RAG system.
Let's understand why.
Why Chunking Matters So Much
Think about how vector search works.
You embed a chunk → store it → later search for similar chunks using a query vector.
The chunk needs to be:
Small enough → focused on one topic → embeds well → found precisely
Large enough → contains enough context → LLM can understand it
Two problems with bad chunking:
Problem 1 — Chunks too small:
Original text:
"Aspirin reduces fever and pain. However patients with
stomach ulcers should avoid it as it can cause bleeding.
Always consult a doctor before use."
Split into tiny chunks:
Chunk A: "Aspirin reduces fever and pain."
Chunk B: "However patients with stomach ulcers should avoid it"
Chunk C: "as it can cause bleeding."
Chunk D: "Always consult a doctor before use."
User asks: "Is aspirin safe for someone with stomach ulcers?"
Vector search finds: Chunk B (best match)
→ "However patients with stomach ulcers should avoid it"
LLM gets: Only Chunk B
LLM says: "Patients with stomach ulcers should avoid it"
→ Missing the reason (bleeding) ❌
→ Missing the "consult doctor" advice ❌
→ Incomplete answer
Too small = loses context = incomplete answers.
Problem 2 — Chunks too large:
Original text: entire chapter about medications
→ 2000 words about aspirin, ibuprofen, paracetamol,
codeine, morphine, and 20 other drugs
Stored as ONE chunk
User asks: "What are the side effects of aspirin?"
Vector search finds: this chunk (it mentions aspirin)
But also mentions 20 other drugs
LLM gets: 2000 words — most of it irrelevant
→ LLM gets confused by irrelevant information
→ Answer quality drops
→ More tokens = more expensive
→ Might miss the specific aspirin info buried in text
Too large = noisy context = confused LLM = worse answers.
The Goldilocks Zone
Too small Just right Too large
──────────── ────────────── ──────────────
loses context focused + complete noisy + expensive
bad answers good answers confused LLM
~50 words ~200-500 words ~2000+ words
For most use cases — 200 to 500 words per chunk is the sweet spot.
Chunking Strategy 1 — Fixed Size Chunking
What it is
Split the document every N characters or N words — no matter what.
Document:
"The quick brown fox jumps over the lazy dog.
The dog did not move. The fox ran away quickly.
Later that day, the fox returned to find..."
Fixed size chunking at 50 characters:
Chunk 1: "The quick brown fox jumps over the lazy dog. The"
Chunk 2: " dog did not move. The fox ran away quickly.\n La"
Chunk 3: "ter that day, the fox returned to find..."
Simple. Fast. But has one obvious problem — it cuts mid-sentence.
Chunk 1 ends with "The" — incomplete sentence. This loses meaning.
The Solution — Chunk Overlap
To fix mid-sentence cuts — you add overlap.
Overlap means the end of one chunk is repeated at the start of the next chunk.
Document: "A B C D E F G H I J K L M N O"
Chunk size: 6 words
Overlap: 2 words
Chunk 1: A B C D E F
Chunk 2: E F G H I J ← E and F repeated from chunk 1
Chunk 3: I J K L M N
Chunk 4: M N O
Now even if a sentence gets split — the overlapping words give context to both chunks.
Real example:
Chunk size: 500 characters
Overlap: 100 characters
Chunk 1: "...Aspirin reduces fever and pain. It works by blocking
prostaglandins which are chemicals that cause inflammation.
The recommended dose for adults is 325mg to 650mg every"
Chunk 2: "The recommended dose for adults is 325mg to 650mg every
4 to 6 hours. Do not exceed 4000mg in 24 hours. Patients
with stomach ulcers should avoid aspirin because..."
"The recommended dose..." appears in both chunks — giving context to each.
Fixed Chunking in Code
function fixedChunking(text, chunkSize = 500, overlap = 100) { // text = full document text as one string // chunkSize = how many characters per chunk (default 500) // overlap = how many characters to repeat between chunks (default 100) // example: text = "Aspirin reduces fever..."
const chunks = []; // chunks = array that will hold all our text pieces // example final value: ["Aspirin reduces...", "...reduces fever and pain..."]
let startIndex = 0; // startIndex = where current chunk starts in the full text // begins at 0 (start of document)
while (startIndex < text.length) { // keep going until we've covered the whole document
let endIndex = startIndex + chunkSize; // endIndex = where current chunk ends // example: startIndex=0, chunkSize=500 → endIndex=500
if (endIndex > text.length) { endIndex = text.length; // don't go past the end of the document }
// Try to end at a sentence boundary — not mid-word // Look for the last period, question mark, or newline before endIndex if (endIndex < text.length) { const lastPeriod = text.lastIndexOf(".", endIndex); const lastNewline = text.lastIndexOf("\n", endIndex); const lastBoundary = Math.max(lastPeriod, lastNewline); // Math.max = take whichever boundary is further in the text
if (lastBoundary > startIndex + (chunkSize * 0.5)) { endIndex = lastBoundary + 1; // only use this boundary if it's at least halfway through the chunk // prevents very tiny chunks // +1 to include the period itself } }
const chunk = text.slice(startIndex, endIndex).trim(); // slice = extract substring from startIndex to endIndex // trim = remove leading/trailing whitespace // example: " Aspirin reduces fever... " → "Aspirin reduces fever..."
if (chunk.length > 0) { chunks.push(chunk); // add this chunk to our array }
startIndex = endIndex - overlap; // move startIndex forward — but go back by overlap amount // example: endIndex=500, overlap=100 → next startIndex=400 // so next chunk starts 100 characters before this one ended // this creates the overlap between chunks }
return chunks; // returns array of text strings // example: ["Aspirin reduces fever...", "...fever and pain. Ibuprofen..."] }
// ── USAGE EXAMPLE ────────────────────────────────────────
const sampleDocument = ` Aspirin is commonly used to reduce fever, pain, and inflammation. It works by blocking prostaglandins — chemicals that cause pain signals. The recommended adult dose is 325mg to 650mg every 4-6 hours. Do not exceed 4000mg in 24 hours without medical supervision.
Ibuprofen is a nonsteroidal anti-inflammatory drug (NSAID). It is effective for headaches, dental pain, menstrual cramps, and arthritis. Common side effects include stomach upset and kidney stress with long term use. Always take ibuprofen with food to reduce stomach irritation.
Paracetamol is used for mild to moderate pain and fever. Unlike aspirin and ibuprofen, it does not reduce inflammation. It is generally safe for most people including pregnant women when used correctly. Overdose is dangerous and can cause permanent liver damage. `;
const chunks = fixedChunking(sampleDocument, 300, 50); // chunkSize = 300 characters // overlap = 50 characters
console.log(`Total chunks created: ${chunks.length}\n`);
chunks.forEach((chunk, index) => { console.log(`Chunk ${index + 1} (${chunk.length} chars):`); console.log(chunk); console.log("─".repeat(50)); });
Output will look like:
Total chunks created: 4
Chunk 1 (298 chars):
Aspirin is commonly used to reduce fever, pain, and inflammation.
It works by blocking prostaglandins — chemicals that cause pain signals.
The recommended adult dose is 325mg to 650mg every 4-6 hours.
──────────────────────────────────────────────────
Chunk 2 (285 chars):
every 4-6 hours.
Do not exceed 4000mg in 24 hours without medical supervision.
Ibuprofen is a nonsteroidal anti-inflammatory drug (NSAID).
It is effective for headaches, dental pain, menstrual cramps...
──────────────────────────────────────────────────
Notice — "every 4-6 hours" appears at the end of chunk 1 and start of chunk 2. That's the overlap working.
Chunking Strategy 2 — Semantic Chunking
What it is
Instead of splitting by character count — split by meaning.
Keep sentences together that belong together. Split when the topic changes.
Document about medications:
Paragraph 1: About aspirin — fever, pain, dosage
Paragraph 2: About ibuprofen — inflammation, kidneys
Paragraph 3: About paracetamol — liver, overdose
Semantic chunking:
Chunk 1 = everything about aspirin (complete idea)
Chunk 2 = everything about ibuprofen (complete idea)
Chunk 3 = everything about paracetamol (complete idea)
Each chunk = one complete topic = embeds perfectly
This is better than fixed chunking because:
Fixed chunking might cut:
"...aspirin is good for fever. Ibuprofen is better for inf..."
→ Half aspirin, half ibuprofen in one chunk
→ Confusing embedding — what is this chunk even about?
Semantic chunking keeps:
"Aspirin is good for fever and pain. Dosage is 325mg..."
→ One complete idea
→ Clean embedding — clearly about aspirin
→ Found precisely when asked about aspirin
Simple Semantic Chunking — Split by Paragraph
The simplest form of semantic chunking — split on double newlines (paragraph breaks):
function paragraphChunking(text, maxChunkSize = 1000) { // text = full document text // maxChunkSize = if a paragraph is too long, split it further
const paragraphs = text.split(/\n\s*\n/); // split on double newlines (paragraph breaks) // \n\s*\n = newline, optional whitespace, newline // example: "Para 1\n\nPara 2" → ["Para 1", "Para 2"]
const chunks = []; let currentChunk = ""; // currentChunk = text we're building up
paragraphs.forEach((paragraph) => { const cleaned = paragraph.trim(); // remove extra whitespace from paragraph edges
if (cleaned.length === 0) return; // skip empty paragraphs
if ((currentChunk + cleaned).length > maxChunkSize) { // adding this paragraph would make chunk too large
if (currentChunk.length > 0) { chunks.push(currentChunk.trim()); // save current chunk currentChunk = ""; // start fresh } }
currentChunk += cleaned + "\n\n"; // add paragraph to current chunk });
if (currentChunk.trim().length > 0) { chunks.push(currentChunk.trim()); // save the last chunk }
return chunks; }
Advanced Semantic Chunking — Using Embeddings
The most powerful form — actually measure semantic similarity between sentences and split when meaning changes significantly:
async function semanticChunking(sentences, openai, threshold = 0.7) { // sentences = array of individual sentences // threshold = if similarity drops below this — start new chunk // example: threshold 0.7 means "if next sentence is less than // 70% similar to current chunk — it's a new topic"
console.log(`Embedding ${sentences.length} sentences...`);
// Embed all sentences const embeddings = await Promise.all( sentences.map(async (sentence) => { const response = await openai.embeddings.create({ model: "text-embedding-3-small", input: sentence, }); return response.data[0].embedding; // each sentence → array of 1536 numbers }) );
const chunks = []; let currentChunkSentences = [sentences[0]]; // start first chunk with first sentence
for (let i = 1; i < sentences.length; i++) { // loop from second sentence to end
// Calculate similarity between current sentence // and the previous sentence const similarity = cosineSimilarity(embeddings[i], embeddings[i - 1]); // similarity = how related this sentence is to the previous one // example: 0.85 = very related (same topic) // example: 0.45 = not very related (topic changed)
if (similarity < threshold) { // similarity dropped below threshold // → topic has changed → save current chunk → start new one
chunks.push(currentChunkSentences.join(" ")); // join all sentences in current chunk into one string currentChunkSentences = [sentences[i]]; // start new chunk with this sentence
console.log(`Topic change detected at sentence ${i} (similarity: ${similarity.toFixed(3)})`); } else { // topic is still the same → add to current chunk currentChunkSentences.push(sentences[i]); } }
// Save the last chunk if (currentChunkSentences.length > 0) { chunks.push(currentChunkSentences.join(" ")); }
return chunks; }
// Helper — cosine similarity (same as Module 3.4) function cosineSimilarity(vecA, vecB) { const dot = vecA.reduce((sum, val, i) => sum + val * vecB[i], 0); const magA = Math.sqrt(vecA.reduce((sum, val) => sum + val * val, 0)); const magB = Math.sqrt(vecB.reduce((sum, val) => sum + val * val, 0)); return dot / (magA * magB); }
Fixed vs Semantic — When to Use Which
┌─────────────────────┬──────────────────────┬───────────────────────┐
│ │ FIXED CHUNKING │ SEMANTIC CHUNKING │
├─────────────────────┼──────────────────────┼───────────────────────┤
│ How it splits │ Every N characters │ When topic changes │
│ Speed │ Very fast │ Slower (needs embeds) │
│ Cost │ Free │ Costs API calls │
│ Quality │ Good │ Better │
│ Best for │ Most use cases │ High quality RAG │
│ Structured docs │ Great │ Overkill │
│ Narrative text │ OK │ Much better │
│ Code files │ Split by function │ Not needed │
└─────────────────────┴──────────────────────┴───────────────────────┘
For most applications — start with fixed chunking + overlap. It works well and is simple to implement. Switch to semantic chunking when you need higher quality and can afford the extra API calls.
Chunking Rules — Practical Guide
Here are the rules you'll follow in real projects:
Rule 1 — Match chunk size to your content type:
Short FAQs / Q&A docs → 100-200 words per chunk
General documents → 200-400 words per chunk
Technical documentation → 300-500 words per chunk
Legal / medical docs → 400-600 words per chunk
Rule 2 — Always use overlap:
Overlap = 10-20% of chunk size is a good default
Chunk size 500 chars → overlap 50-100 chars
Chunk size 1000 chars → overlap 100-200 chars
Rule 3 — Store metadata with every chunk:
// Every chunk should know where it came from { text: "Aspirin reduces fever and pain...", metadata: { source: "medical_handbook.pdf", // which file page: 12, // which page chunkIndex: 3, // which chunk on that page totalChunks: 47, // total chunks in document } }
Rule 4 — Don't split mid-sentence if possible:
Bad: "Aspirin reduces fever and pa | in and inflammation"
Good: "Aspirin reduces fever and pain." | "It also reduces inflammation."
Rule 5 — Test your chunking before building the full system:
// Print your chunks before embedding them // Make sure they make sense as standalone pieces chunks.forEach((chunk, i) => { console.log(`Chunk ${i}: ${chunk.substring(0, 100)}...`); console.log(`Length: ${chunk.length} chars`); }); // If chunks look weird — fix chunking before wasting API money
A Real Life Analogy — Library Book Index
Think of chunking like creating an index for a textbook.
Bad index (too granular):
"the" → pages 1,2,3,4,5,6... (every page)
→ useless — too specific
Bad index (too broad):
"medicine" → pages 1-500 (entire book)
→ useless — too vague
Good index (just right):
"aspirin side effects" → pages 23-24
"ibuprofen dosage" → pages 31-32
"paracetamol overdose" → pages 47-48
→ useful — specific enough to find, broad enough to be complete
Your chunks are like index entries. Each one should be specific enough to be found accurately, and complete enough to be useful when found.
3-Line Summary
- Chunks too small lose context and give incomplete answers — chunks too large add noise and confuse the LLM — the sweet spot is 200 to 500 words with 10-20% overlap between chunks.
- Fixed chunking splits every N characters (fast, simple, works well for most cases) — semantic chunking splits when topic changes by measuring embedding similarity between sentences (slower, more expensive, better quality).
- Always store metadata with every chunk (source file, page number, chunk index) — this lets you filter searches, cite sources in answers, and debug when results are wrong.
Module 5.2 — Complete ✅
Coming up — Module 5.3 — Retrieval & Context Injection
You know how to store chunks. Now we cover what happens when a user asks a question — how retrieval actually works, how you inject retrieved chunks into a prompt correctly, and the exact prompt structure that makes LLMs give accurate sourced answers.