LLM-as-judge
Use a second model to score open-ended answers, for example from 1 to 5.
Last updated
Analogy
Snapshot tests can't grade an essay, so you hire a (fast, slightly opinionated) grader.
In an agent
Use exact checks where you can and a judge model only for fuzzy answers.
Practice
Flashcards
- What is LLM-as-judge?Using a model to grade another model's answer.
- Interview: how do you know your agent works?An eval set of 20 to 50 real cases run in CI on every change, with exact checks plus LLM-as-judge.
Interview questions
- How do you evaluate an agent?
- An eval set of 20 to 50 real cases with expected tools and key facts.
- Run it in CI on every prompt or model change.
- Exact checks where possible, LLM-as-judge for open answers.
- Mine production traces for new failure cases.
Questions people ask
What is LLM-as-judge?
LLM-as-judge means using a second model to grade an answer, for example scoring it from 1 to 5 against a reference answer or a rubric. It handles open-ended answers that exact string checks cannot grade, like whether an explanation is correct and complete.
Can I trust an LLM judge's scores?
Partly. Judges can be inconsistent or favour long answers. Give the judge a clear rubric and a reference answer, keep its temperature low, and spot-check its scores against your own. Use exact checks wherever possible, and the judge only where answers are truly open-ended.