Text splitters
Chunk text smartly, by headings, paragraphs or code functions, with a chosen size and overlap.
Last updated
Analogy
A formatter for chunking: it knows where the natural line breaks are so you don't cut a sentence in half.
Code example
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters"; const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 100,});const chunks = await splitter.splitText(markdownDoc);In an agent
Good splits mean the agent retrieves whole ideas, not half-sentences.
Practice
Flashcards
- Interview: how do you pick chunk size?Start at 500 to 800 tokens with overlap, split on natural boundaries, then tune with retrieval evals.
Interview questions
- How do you choose chunk size?
- Start around 500 to 800 tokens with some overlap.
- Split on natural boundaries like headings, paragraphs or functions.
- Tune with retrieval evals instead of guessing.
Questions people ask
How do LangChain text splitters work?
A splitter such as RecursiveCharacterTextSplitter cuts text into chunks of a chosen size with a chosen overlap. It tries natural break points first, like headings, paragraphs and lines, before cutting mid-sentence. That keeps each chunk close to one complete idea, which makes retrieval more accurate.
What chunk size and overlap should I give a text splitter?
A common starting point is a chunk size around 1,000 characters with about 100 characters of overlap, which is roughly in line with 300 to 800 tokens. The overlap keeps sentences that cross a boundary from being lost. Tune both values with retrieval evals on your own questions.