- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that deterministic rules in LLM-as-a-Judge evaluation pipelines can artificially inflate agreement between judges and proposes simple reporting practices to make these effects transparent.
Critical strengths
-
The paper highlights an overlooked source of bias in LLM-as-a-Judge evaluation, showing that deterministic rules in evaluation pipelines can affect reported inter-judge agreement. This raises an important concern about the reliability of existing evaluation results.
-
The paper proposes simple reporting practices, such as reporting agreement on unaffected examples and the proportion of overridden judgments, which could improve the transparency and reliability of LLM-as-a-Judge evaluations.
Critical weaknesses
-
The paper primarily identifies implementation-level issues in existing LLM-as-a-Judge pipelines. It is unclear whether these findings provide sufficient research contributions beyond highlighting evaluation code pitfalls.
-
The strongest agreement inflation comes from length-based rules, found in only 1 of 35 audited pipelines, while the most common override has a negligible median effect. Thus, it remains unclear whether substantial agreement inflation is widespread in practice.
-
The proposed κU measures agreement only on examples unaffected by deterministic overrides. Since these examples may systematically differ from the full benchmark, the authors should discuss the potential selection bias and limitations of κU as a measure of overall judge agreement.
-
The distinction between row-wise and column-wise overrides could be explained more clearly with concrete examples.
Questions for authors
-
Could the authors evaluate how frequently deterministic overrides are triggered in the audited pipelines and how much they affect reported agreement under the original evaluation settings, rather than relying primarily on counterfactual experiments?
-
Since κU is computed on a nonrandom subset of examples, how should it be interpreted as a measure of judge reliability? Could the gap between κ and κU also reflect differences in example composition rather than only override-induced inflation?