Part 05 · Embeddings + RAG · Card 10 of 14
Top K
How many chunks you retrieve per question. Start with 3 to 5.
Last updated
Analogy
The LIMIT on your query. More rows isn't better if nobody reads them.
In an agent
More chunks means more cost and more noise for the model to ignore.
Practice
Flashcards
- Interview: first question when RAG answers wrong?Were the right chunks retrieved? If not, fix retrieval. If yes, fix the prompt or model. Then add an eval case.
Interview questions
- Your RAG gives wrong answers. How do you debug it?
- Check the trace: were the right chunks retrieved?
- If not, fix retrieval: chunking, hybrid search, re-ranking, metadata filters.
- If they were, fix the prompt or the model.
- Then add the failing case to the eval set so it stays fixed.
Questions people ask
How many chunks should RAG retrieve?
Start with 3 to 5 chunks per question. More chunks cost more tokens and add noise the model has to ignore, which can make answers worse. If answers miss information, check retrieval quality first, with re-ranking or better chunking, before simply raising the number.
Is retrieving more chunks always better?
No. Every extra chunk adds cost and gives the model more irrelevant text to wade through. Past a small number, quality often drops. A better pattern is to fetch more candidates, for example 20, re-rank them by relevance, and pass only the best few to the model.