acceptodds
Under review as a conference paper at ICLR 2027

Why LLM-Judge Agreement Is Overstated, and How to Fix It

Abstract

To show that automatic grading can be trusted, papers report how often two language-model judges gave the same verdict. Between a judge's reply and that number sits code. A parser that cannot read a reply writes in a default. A tie is broken to a fixed side. An answer that refuses, or is too short, is marked wrong before any judge sees it. The same code runs for every judge, so wherever it fires the judges agree by construction rather than by judgement, and the reported agreement is partly its own work. We read 35 public judging pipelines and find such code in 28 of them (80%), a parser default most often; none reports agreement on the rows it left alone. Applying each kind of rule to 38,601 recorded verdicts, from nine judges on six human-labelled benchmarks, we find that length rules raise Cohen's by a median of +0.071 and significantly in 78% of judge pairs, while a parse-failure default moves it either way by up to 0.08 according to which verdict the parser writes. Those same length rules lower agreement with the human labels. Agreement on the untouched rows exposes 87% of the inflated cases and costs nothing to report, so we ask that papers report it, together with the share of rows their code decided.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, evaluation methodology, inter-annotator agreement, meta-evaluation, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gtTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that deterministic rules in LLM-as-a-Judge evaluation pipelines can artificially inflate agreement between judges and proposes simple reporting practices to make these effects transparent.

Critical strengths
  1. The paper highlights an overlooked source of bias in LLM-as-a-Judge evaluation, showing that deterministic rules in evaluation pipelines can affect reported inter-judge agreement. This raises an important concern about the reliability of existing evaluation results.

  2. The paper proposes simple reporting practices, such as reporting agreement on unaffected examples and the proportion of overridden judgments, which could improve the transparency and reliability of LLM-as-a-Judge evaluations.

Critical weaknesses
  1. The paper primarily identifies implementation-level issues in existing LLM-as-a-Judge pipelines. It is unclear whether these findings provide sufficient research contributions beyond highlighting evaluation code pitfalls.

  2. The strongest agreement inflation comes from length-based rules, found in only 1 of 35 audited pipelines, while the most common override has a negligible median effect. Thus, it remains unclear whether substantial agreement inflation is widespread in practice.

  3. The proposed κU measures agreement only on examples unaffected by deterministic overrides. Since these examples may systematically differ from the full benchmark, the authors should discuss the potential selection bias and limitations of κU as a measure of overall judge agreement.

  4. The distinction between row-wise and column-wise overrides could be explained more clearly with concrete examples.

Questions for authors
  1. Could the authors evaluate how frequently deterministic overrides are triggered in the audited pipelines and how much they affect reported agreement under the original evaluation settings, rather than relying primarily on counterfactual experiments?

  2. Since κU is computed on a nonrandom subset of examples, how should it be interpreted as a measure of judge reliability? Could the gap between κ and κU also reflect differences in example composition rather than only override-induced inflation?

Reviewer_hWYvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

LLM-based judging pipelines often apply deterministic overrides to the candidate or judge output. Such overrides are common in many judging pipelines. As a result, comparing two LLM-based judges in the same pipeline may distort agreement. The authors recommend reporting Cohen’s kappa on the rows decided by the judges and not deterministic rules.

Critical strengths
  1. The paper is honest in many of its shortcomings.
  2. It offers actionable guidance for how to better measure agreement between judges run in the same pipeline.
  3. The experimental design cleanly isolates the impact of the rules.
Critical weaknesses
  1. The writing is very low-quality and difficult to follow in places.
  2. Typically agreement is computed between an LLM-based judge and human graders or gold labels, in which deterministic overrides on the candidate or judge outputs are not problematic. I don’t see many papers computing Cohen's kappa across judge models.
  3. The audit of judge pipelines appears sloppy. For instance, the tie-break in JudgeBench acts only on float logits from reward models, which are unlikely to be equal.
  4. There is a disconnect between the audits and the mechanisms applied during the experiments. For instance, the length rule is the least common in the audited pipelines but has the largest effect on agreement, while parsing failures is the most common in the audited pipelines but have little impact on agreement.
Questions for authors

See weaknesses.

Reviewer_2BBWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Researchers often report whether LLM grading is trustworthy by demonstrating how often two LLM judges agree, usually as Cohen’s κ. The authors argue that part of that agreement is produced by the evaluation code, not the judges. The authors audit 35 judging pipelines, and analyze 38k verdicts from nine judges across six benchmarks, and find deterministic overrides due to evaluation code in 80%,

Critical strengths
  1. Conducted an in-depth review of the judging pipeline to identify key limitations in the evaluation protocol. These findings will help other researchers when they use these pipelines.
Critical weaknesses
  1. I think the central finding is quite expected. For instance, it is expected that researchers should only measure the agreement between judges when they are not overridden by evaluation code. Therefore, the novelty and broader impact of this work is very limited.
  2. Although the repository audit could provide useful empirical evidence, its contribution appears closer to documenting an evaluation pitfall than establishing a substantially novel finding.
Questions for authors

Please address the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.