- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether LLM-judge scores change when the same rubric criteria are only grouped into blocks differently. It introduces a permutation-calibrated ratio (CDR) and finds that partition-induced dispersion modestly exceeds repeat noise for all three judges.
Critical strengths
- The studied problem is interesting and common in rubric-based LLM evaluation.
- The proposed metric (CDR) is a meaningful diagnostic that reveals partition sensitivity in current rubric-based judges.
- The statistical analysis is careful, with controls, robustness checks, and replication.
Critical weaknesses
- The paper provides a diagnostic metric but no remedy, and does not test which scoring interface works better.
- The effect is small (0.03–0.08 points), and the paper does not show that it changes rankings or downstream decisions.
- CDR only reflects a judge's consistency across equivalent interfaces and is not validated against human ratings, so it says nothing about judge accuracy.
- Only three judge models are evaluated, which is too few to show how CDR varies with model family and size.
Questions for authors
See Weaknesses.
2 comments
Thanks for the authors' response. Considering the rebuttal and the other reviewers' comments, I will maintain my original score.
Response to Reviewer ACCb
hank you for your positive comments and your clear list of weaknesses. We respond to each one below.
W1. No remedy, and no test of which interface works better.
We now give both (Sec. 7, App. A.8).
A remedy. First, run the calibration to see whether a judge is sensitive to grouping, and by how many points. If it is, average its scores over several boundaries. This removes the grouping effect, and on our human-labeled datasets it does not hurt agreement with human ratings. It matches or beats the typical single boundary in 13 of 15 cases (App. Table 15). Which interface is better. We compared every boundary, the boundary average, scoring each criterion separately, and one holistic score against human labels. No single option is best for every judge. Scoring each criterion separately helps Selene on GSM8K but hurts DeepSeek on HelpSteer2. For this reason we recommend calibrating each judge rather than prescribing one interface. W2. The effect is small and may not change decisions.
The 0.03 to 0.08 values are a lower bound. After removing repeat noise, the spread across partitions is 0.14, 0.17 and 0.19 points (Sec. 5.1, Table 5). Changing the boundary moves 22%, 25% and 19% of items by half a point or more. The shuffled baseline gives 15%, 18% and 2%. Pairwise order flips are above baseline for all three judges (App. Table 11). Overall rankings stay mostly intact, and we now say so. The effect matters most where scores meet a cutoff, as in filtering, top-k selection or rewards.
W3. CDR measures consistency, not accuracy.
We agree, and the paper now says this directly (Sec. 7). We think this is exactly why it should be measured on its own. A judge can match human ratings well on average and still give the same answer a different score depending on how the rubric is grouped. Agreement studies do not reveal this. Our new human-label analysis shows that the choice of boundary rarely changes agreement with humans, and that averaging over boundaries removes the grouping effect at no cost (App. A.8).
W4. Only three judges.
We agree that this is a limitation and state it in Sec. 6. The three judges are quite different: a large reasoning model, an open-weight 27B model and a small dedicated evaluator, with both deterministic and sampled decoding. The new experiments show that they differ in kind, not only in degree. Two are sensitive mainly to grouping, and one is sensitive to the whole layout. This is why the contribution is a method that can be rerun on any judge, rather than a single correction factor. The full pipeline is released for this purpose.
Thank you again. Your comments led to the effect-size, decision and human-agreement analyses now in the paper.