Chunking Strategy

What it is

Chunking is splitting a document into smaller pieces before embedding them. Each piece, or chunk, becomes one row in your vector store and is the unit you retrieve. The chunk is what the LLM eventually reads, so its boundaries decide whether the model sees a complete fact or half of one.

There is no single correct chunk size. A legal contract, a Slack export and a resume need different strategies. The decision that matters is: when a chunk is pulled out on its own, with no neighbours, does it still make sense and still answer a question?

For this site I chose paragraph chunks. Every paragraph in my knowledge base is one complete idea: one job, one project, one skill group. That made the self-contained rule easy to satisfy and gave me 50-odd clean chunks instead of hundreds of fragments.

Why it's needed

If the whole document is one chunk, retrieval returns everything every time, the prompt fills with irrelevant text, cost goes up and the model gets distracted. If chunks are too small, a fact gets cut in half and the retrieved piece says 'he worked there for two years' with no mention of where.

Chunking also sets your retrieval precision ceiling. The embedding of a chunk is an average of everything in it, so a chunk that mixes three topics has a blurry vector that matches none of them well. Clean chunks produce sharp vectors.

How it works

• Paragraph splitting: split on blank lines. Works when the author already wrote one idea per paragraph, which is true for resumes, FAQs and most documentation.
• Fixed-size splitting: cut every N characters or tokens, typically 300 to 800 tokens. Simple and predictable, but it ignores structure and cuts mid-sentence.
• Recursive splitting: try to split on headings, then paragraphs, then sentences, then characters, until every piece is under the size limit. This is what LangChain's RecursiveCharacterTextSplitter does and it is the sensible default for mixed documents.
• Overlap: repeat the last 10 to 20 percent of one chunk at the start of the next, so a sentence that straddles a boundary appears whole in at least one chunk.
• Make chunks self-contained: replace pronouns with names ('Abdur led the migration' instead of 'he led the migration') and keep the entity being described inside each chunk.
• Add metadata: store the source file, heading and position with each chunk so you can filter, cite and debug retrieval later.

Code example

import re

def paragraph_chunks(text: str) -> list[str]:
    """One chunk per paragraph. Used for the portfolio knowledge base."""
    return [p.strip() for p in text.split("\n\n") if p.strip()]

def fixed_chunks(text: str, size: int = 500, overlap: int = 80) -> list[str]:
    """Fixed-size character windows with overlap, snapped to sentence ends."""
    sentences = re.split(r"(?<=[.!?])\s+", text)
    chunks, current = [], ""
    for s in sentences:
        if len(current) + len(s) > size and current:
            chunks.append(current.strip())
            current = current[-overlap:]  # carry the tail forward
        current += " " + s
    if current.strip():
        chunks.append(current.strip())
    return chunks

doc = open("knowledge_base.txt", encoding="utf-8").read()
paras = paragraph_chunks(doc)
fixed = fixed_chunks(doc)
print(len(paras), "paragraph chunks;", len(fixed), "fixed chunks")
print(paras[0][:120])

Where I used this

Use cases

• Preparing any long document (resume, manual, policy, contract) for RAG
• Splitting PDFs or web articles into searchable units
• Building a support knowledge base from existing help-centre articles
• Indexing code repositories by function or class rather than by file

Common mistakes

• Cutting mid-sentence or mid-table so the retrieved chunk is grammatically broken and the LLM has to guess the missing half.
• Pronouns everywhere: a chunk that says 'it reduced costs by 30%' is useless on its own because 'it' is in the previous chunk.
• Chunks that are too big: a 2000-token chunk has a vague embedding and wastes the context window on text that is mostly irrelevant.
• Forgetting that the embedding model truncates: a 600-word chunk fed to MiniLM only has its first ~200 words embedded.
• Never re-checking after the source changes: a new document format can silently break the splitter and produce one giant chunk.

Interview questions