Key takeaways
- Position bias is reported at up to 75% preference for whichever response appears first. Swap the order and a share of your grades flip.
- A 2026 RAND Corporation study reported that no judge was uniformly reliable across benchmarks, with frontier models exceeding 50% error rates on challenging bias tests.
- The often-quoted figure that a strong model agrees with human raters over 80% of the time is an average. Failures are not evenly distributed, and they concentrate where decisions are hard.
- High agreement between judges is not evidence of correctness. A panel can be consistently and confidently wrong together, which is what reliability without validity means.
- Test your grader before you trust it: swap positions, pad one answer, and grade a set where you already know the right answer.
Does ChatGPT recommend your competitor instead of you?
Check how AI assistants describe your company, and whose name they give when someone asks for a recommendation in your category. Run an audit today, from $19.
Present two candidate answers to an automated grader and it will show a preference for the one you listed first. Reported at up to 75% in some evaluations, which means the ordering of your prompt is doing more work than the quality of the answers.
That is not a bug in one model. It is a property of the method, and it survives into every pipeline built on top of it.
What did the 2026 evaluations actually find?
That judges are reliable and that reliability was the wrong thing to measure.
A large-scale 2026 evaluation published under the title Reliability without Validity separates the two properties directly. Reliability is whether a judge gives the same verdict twice. Validity is whether the verdict is correct. A judge can score perfectly on the first and poorly on the second, and most evaluation pipelines only ever test the first.
Alongside it, a RAND Corporation study reported that no judge was uniformly reliable across benchmarks, and that frontier models exceeded 50% error rates on the harder bias tests. Morgan Sandler and colleagues published an open-source harness in March 2026 for stress-testing judges on exactly these axes.
| Failure mode | Reported figure | Detectable in | What it does to your numbers |
|---|---|---|---|
| Position bias | Up to 75% | 1 hour | Fixed candidate order produces a fixed skew |
| Frontier error on hard bias tests | Over 50% | 1 day | Half the verdicts on difficult cases are wrong |
| Aggregate agreement with humans | Over 80% | 1 day | Looks reassuring, averages over easy cases |
| Verbosity bias | Longer answers preferred | 1 hour | Rewards padding, penalises concision |
| Self-enhancement bias | Own-family outputs preferred | 2 hours | Grading a model with its relatives inflates it |
Why is 80% agreement not as reassuring as it sounds?
Because it is an average over cases, and the cases are not equally difficult.
The often-quoted result is that a strong model agrees with human raters more than 80% of the time, which is about the rate at which humans agree with each other. That comparison is fair and it is also where people stop reading.
Agreement is highest on the cases where the answer is obvious, which are the cases you did not need a grader for. It falls on the ambiguous ones, which are the only cases where the grade changes a decision. An 80% average built mostly from easy cases can coexist with near-random performance on the hard ones.
Why two agreeing judges do not help
The instinct when a grader looks unreliable is to add a second and take the consensus. This is where the reliability-versus-validity distinction earns its keep.
Judge models are trained on overlapping data and share systematic preferences. They agree because they are similar, not because they are right. A panel converging on the same wrong verdict produces a confident, stable, incorrect number, and its stability is what persuades you to trust it.
What should you do before trusting a grader?
Three tests. Each takes about an hour and any one of them can disqualify a pipeline.
| Test | How to run it | Failure means |
|---|---|---|
| Position swap | Grade the same pairs twice with the order reversed | Grades that flip are measuring layout, not quality |
| Length padding | Add harmless filler to one candidate | A score that moves rewards verbosity |
| Known-answer set | Grade fifty cases where the answer is already established | Disagreements here are the validity ceiling |
| Family separation | Never grade a model with a judge from the same family | Self-enhancement bias inflates your own results |
The known-answer set is the one that matters most and the one teams skip, because assembling fifty cases with established correct answers is slow work with nothing to demo at the end of it.
Where do automated judges genuinely work?
On narrow, well-specified checks where correctness is nearly mechanical.
Did the output parse as valid JSON against this schema. Does it contain the required fields. Does it cite a document that exists. Is it in the requested language. These are questions with a defensible right answer, and a model asked to check them is doing classification rather than judgement.
The trouble starts when the same apparatus is pointed at whether an answer was good. That is a judgement call, and the biases above are largest exactly where judgement is required.
The question to ask before your next eval run
Not “which judge model should we use”. Ask: if we reversed the order of our candidates, how many of last week’s grades would change? If nobody has run that experiment, the pipeline is producing numbers rather than evidence.
Frequently asked questions
How accurate is LLM-as-a-judge evaluation?
Reported aggregate agreement with human raters exceeds 80% for strong models, which is roughly the rate at which humans agree with each other. That average conceals uneven failure: a 2026 RAND study reported no judge was uniformly reliable across benchmarks and that frontier models exceeded 50% error rates on challenging bias tests.
What is position bias in LLM judges?
The tendency to prefer whichever candidate response is presented first, reported at up to 75% in some evaluations. It is a property of the prompt layout rather than of the answers, which means a grading pipeline that always presents candidates in the same order will produce a consistent and consistently skewed result.
Can you trust two LLM judges that agree with each other?
Not on that basis alone. Agreement between judges measures consistency, not validity. Models trained on overlapping data share systematic preferences, so a panel can be reliably wrong in the same direction. Correctness has to be established against known-good answers, not against another model.
How do you test whether your LLM grader works?
Three checks. Swap the order of candidates and measure how many grades flip. Pad one candidate with harmless extra length and see whether the score moves. Then grade a set of fifty cases where you already know the answer and count disagreements. Each takes an hour.
Method and sources
- Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias, arXiv:2606.19544. Primary source for the distinction between reliability and validity in judge models.
- RAND Corporation study reported in 2026, finding that no judge was uniformly reliable across benchmarks and that frontier models exceeded 50% error rates on challenging bias benchmarks. Morgan Sandler and colleagues released the Judge Reliability Harness, an open-source stress-testing library, in March 2026.
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge, arXiv:2412.12509. Consulted for consistency methodology.
- Reported position bias of up to 75%, verbosity bias and self-enhancement bias, from 2026 evaluations of judge models aggregated in practitioner analyses.
- The figure of over 80% agreement between a strong model and human raters comes from the foundational LLM-as-a-judge literature and is quoted here as reported, not measured by us.