- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how explicit rules, paraphrases, and missing requirements affect LLM-judge decisions. Through several controlled experiments, the authors examine verdict changes, panel agreement, abstention behavior, and the effects of additional verification procedures.
Critical strengths
The paper includes several controlled comparisons and reports detailed experimental results.
Critical weaknesses
- Very limited novelty and significance.
The central finding that changing an explicit evaluation rule changes the judge's verdict, while paraphrasing the same rule has little effect, is largely expected from instruction-following LLMs. The paper does not establish why this observation constitutes a meaningful new insight into LLM-judge reliability.
- Weak experimental implications.
The reported increase in abstention after explicitly mentioning missing rules does not necessarily indicate better evaluation quality. In fact, Table 2 shows that merely mentioning conventions also eliminates binary unanimity, while one notice causes 23/128 determinate controls to abstain, raising concerns about the practical value of these findings.
- Fragmented experimental design and unclear contributions.
The paper combines numerous experiments with different datasets, judge panels, metrics, and objectives, making it difficult to identify a coherent central contribution. Several additional experiments report null or negative results, and the matched-call comparison shows no accuracy improvement (94.34% for both methods).
- Limited generalizability.
The main controlled experiments rely on only eight manually authored mechanisms and four judges from two providers. The omission and wording experiments also reuse the same cases, leaving the generalizability of the findings to realistic evaluation tasks largely untested.
Questions for authors
see weaknesses
Thank you. We take the four points in turn.
Novelty. We agree that rule-following alone is expected. It is our control, not our claim. The finding is the omission result: with no rule stated and UNDETERMINED available, four judges from two providers return the same binary verdict on 41/128 ambiguous cases, (post-hoc) always on the same default convention within a mechanism, and on 37 of those cases the same judges follow either explicit rule correctly. Agreement there reflects a shared default. Details are in our reply to Reviewer a4UK.
Implications. We agree that more abstention is not better evaluation, and the paper does not propose notices as a remedy. The two facts you cite are our own controls and we read them as you do: unanimity disappears when conventions are merely mentioned, and one wording over-abstains on determinate cases. The implication is a caution. Panel unanimity and abstention rates both move with cues that carry no rule, so neither should be taken as evidence that a task was well specified. The dependable intervention in our data is stating the rule: 98.5% of verdicts follow a reversed rule, and 0.9% change under paraphrase.
Fragmentation. This is fair, and it is the main cause of the presentation problem. The paper carries earlier studies (eleven judges; SQL, regex and FOLIO cohorts) beside the four-judge experiments, because the project moved through them on the way to the matched design (we describe that path in our reply to the Area Chair). The core is three linked experiments on one case set: the explicit rule, the omitted rule and the wording controls. The matched-call tie is reported as a negative result. We kept the nulls and the tie because dropping them would overstate the evidence.
Generality. We agree this is a limitation. There are eight authored mechanisms and four judges from two providers, and the omission and wording studies reuse the same cases. The claims are about this fixed benchmark and we state them that way. Natural tasks, more judge families and new mechanisms are needed before any claim about how often this happens in practice.