Evals
A test suite for the agent: 20 to 50 real questions with expected results, run on every prompt or model change.
Last updated
Analogy
Unit tests for behaviour that's a little bit random. You track a score instead of all-green.
Code example
const cases = [ { input: "Which PRs are blocked?", expectTool: "listOpenPRs" }, { input: "How do we roll back?", expectTool: "searchDocs", mustContain: "helm rollback" },];// Run the agent per case; check tools called and answer content.In an agent
This is the answer to "how do you know your agent works?", the question most candidates can't answer.
Practice
Flashcards
- What is an eval?A test suite that scores an agent's answers.
- Interview: first question when RAG answers wrong?Were the right chunks retrieved? If not, fix retrieval. If yes, fix the prompt or model. Then add an eval case.
- Interview: how do you know your agent works?An eval set of 20 to 50 real cases run in CI on every change, with exact checks plus LLM-as-judge.
- Interview: how do you reduce hallucinations?Ground with RAG and tools, allow "I don't know", ask for citations, use low temperature, and test with evals.
Interview questions
- How do you choose chunk size?
- Start around 500 to 800 tokens with some overlap.
- Split on natural boundaries like headings, paragraphs or functions.
- Tune with retrieval evals instead of guessing.
- Your RAG gives wrong answers. How do you debug it?
- Check the trace: were the right chunks retrieved?
- If not, fix retrieval: chunking, hybrid search, re-ranking, metadata filters.
- If they were, fix the prompt or the model.
- Then add the failing case to the eval set so it stays fixed.
- How do you evaluate an agent?
- An eval set of 20 to 50 real cases with expected tools and key facts.
- Run it in CI on every prompt or model change.
- Exact checks where possible, LLM-as-judge for open answers.
- Mine production traces for new failure cases.
Questions people ask
How do I test an AI agent?
Build an eval set: 20 to 50 real questions with expected results, such as which tool should be called or what the answer must contain. Run it on every prompt, tool or model change, ideally in CI. Track the score over time, because some randomness means it will rarely be perfect.
What is the difference between evals and unit tests?
Unit tests check exact behaviour and should all pass. Evals check behaviour that is partly random, so you track a score instead of expecting all green. Use exact checks where you can, such as the right tool being called, and a judge model only for open-ended answers.