Token-Aware & Syntax-Aware Splitting

1. The Character Count Trap

You split your text at 1000 characters. You think you are safe. But 1000 characters could be 200 tokens (simple English) or 800 tokens (code/unicode). If your embedding model has a hard limit of 512 tokens, your "safe" character chunks might crash the API.

2. TokenTextSplitter

To be safe, we split by tokens, not characters.

from langchain_text_splitters import TokenTextSplitter

splitter = TokenTextSplitter(
  chunk_size=500, # Actual tokens
  chunk_overlap=50
)

# This guarantees compatibility with the model
docs = splitter.split_text(long_text)

Trade-off: It is slower (needs to run a tokenizer) and it might split words in half (meaning visual cut), but it guarantees mathematical safety.

3. Syntax-Aware (Code) Splitting

Splitting code is harder. If you split a Python function in the middle: Chunk 1: def calculate_tax(income): Chunk 2: return income * 0.2

Chunk 2 has lost the context of what it is calculating. It is orphaned code. We need Language-Aware splitting.

from langchain_text_splitters import RecursiveCharacterTextSplitter, Language

py_splitter = RecursiveCharacterTextSplitter.from_language(
  language=Language.PYTHON,
  chunk_size=500
)

# It knows to split at "def", "class", and indentations.

4. Summary

Don't assume characters map to tokens. Don't split code like prose. Use the splitter designed for your data type.

Key Intuition: "Split code by logic, text by tokens."