Chunking: Why It Exists

1. The Physics of the Context Window

You cannot fit a library into a shoebox. LLMs have a fixed Context Window (e.g., 8k, 32k, 128k tokens). Vector embeddings models have an even smaller limit (typically 512 or 8192 tokens).

If you try to embed a 50-page contract in one go, the embedding model will truncate it. You lose 49 pages. Chunking is the process of breaking documents into pieces that fit these limits.

2. Naive Chunking (Fixed Size)

The simplest approach is to split every $N$ characters.

text = "The quick brown fox jumps over the lazy dog."
chunks = [text[i:i+5] for i in range(0, len(text), 5)]
# Output: ["The q", "uick ", "brown", ...]

The Problem: This breaks meaning. If the split happens in the middle of a sentence, the first half ("The user is not") and the second half ("allowed to login") end up in different vectors. When you retrieve one, you miss the other.

3. The Trade-offs

• Small Chunks: High precision. You find the exact sentence. But you lack context (who said it?).
• Large Chunks: High context. You get the whole paragraph. But the embedding might be "diluted" (averaging out 5 different topics).

4. Summary

Chunking is not just "cutting." It is prioritizing what stays together. We need strategies that respect the semantic boundaries of the text.

Key Intuition: "Embeddings average meaning. Chunking defines what gets averaged."