# LLM-as-judge

Use a second model to score open-ended answers, for example from 1 to 5.

AI developers · Part 09: Production · Card 9 of 13 · Last updated 2026-10-13

## Analogy

Snapshot tests can't grade an essay, so you hire a (fast, slightly opinionated) grader.

## In an agent

Use exact checks where you can and a judge model only for fuzzy answers.

## Practice

### Flashcards

- **What is LLM-as-judge?** Using a model to grade another model's answer.
- **Interview: how do you know your agent works?** An eval set of 20 to 50 real cases run in CI on every change, with exact checks plus LLM-as-judge.

### Interview questions

**How do you evaluate an agent?**

- An eval set of 20 to 50 real cases with expected tools and key facts.
- Run it in CI on every prompt or model change.
- Exact checks where possible, LLM-as-judge for open answers.
- Mine production traces for new failure cases.

## Questions people ask

### What is LLM-as-judge?

LLM-as-judge means using a second model to grade an answer, for example scoring it from 1 to 5 against a reference answer or a rubric. It handles open-ended answers that exact string checks cannot grade, like whether an explanation is correct and complete.

### Can I trust an LLM judge's scores?

Partly. Judges can be inconsistent or favour long answers. Give the judge a clear rubric and a reference answer, keep its temperature low, and spot-check its scores against your own. Use exact checks wherever possible, and the judge only where answers are truly open-ended.

## Related concepts

- [Evals](https://studyvisually.com/learn/tech/ai-developers/production/evals)
- [Guardrails](https://studyvisually.com/learn/tech/ai-developers/production/guardrails)
- [Validate tool arguments](https://studyvisually.com/learn/tech/ai-developers/production/validate-tool-arguments)
- [Least privilege](https://studyvisually.com/learn/tech/ai-developers/production/least-privilege)

Web page: https://studyvisually.com/learn/tech/ai-developers/production/llm-as-judge
Study this card in the app: https://studyvisually.com/app/learn?category=ai-developers&card=llm-as-judge