- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper reanalyzes the three benchmark (RewardBench, PPE, and RewardBench 2) to evaluate gaps between judge-judge agreement and judge-human agreement, as well as the cross-domain transfer of calibration. Although the agreement gap and ECE are statistically distinguishable as zero, the sample shows a Gray outcome because it does not reach the preregistered materiality margin. Selective risk means that there are not enough domains that meet the evaluation criteria. The authors limit the scope of interpretation, stating that construct validity cannot be identified through such analysis.
Critical strengths
None
Critical weaknesses
-
Overall, explanation flow is unclear. Lack of introduction to key terminology makes difficult to follow the motivation and procedures. More emphasis is placed on details related to execution than on explanations of what and why it is being analyzed.
-
The distinction between an agreement and validity is already introduced as a known issue in related works (Sec 2, Also Table 1 is not mentioned anywhere.) Since this analysis also fails to identify construct validity. Therefore, it is unclear what new insights the proposed agreement gap and calibration analyses provide beyond existing work.
-
Experimental setup is scattered across the manuscript. Therefore, make difficult to understand overall experimental flow (selected model, configuration, data and methods).
Questions for authors
Refer to weakness
Response to Reviewer gKjD
Thank you for identifying the missing motivation, definitions and scattered setup.
W1: Explanation and motivation
The opening now gives the question and motivation before the method:
"When automatic judges agree with one another, does that agreement also hold against the human reference? We report a preregistered audit of this question on three public preference datasets, using released judge scores."
"Agreement among automatic judges is often reported as evidence that an evaluation is reliable, especially when human labels are scarce. It is a weaker statement than agreement with a human reference, and the difference between the two can be measured wherever both are available."
"A judge chooses a response from its scores; the reference records the released human choice."
"Raw agreement is the fraction of matching choices; Cohen's kappa corrects it for chance under the observed label frequencies."
"Calibration asks whether a predicted probability matches how often the judge's chosen response matches the reference."
Section 1 also defines the agreement gap, task slice, ECE and selective risk before results. Section 1 gives the opening; the abstract states the outcome, and "Applying the submitted preregistered rule" now starts Section 4.1.
Calibration average: With five bins instead of the registered ten it is 0.053 (Section 4.2). Whether the calibration average lies below its margin therefore depends on the bin count. The registered settings define the preregistered outcome, which is unchanged; no outcome is assigned to any other setting (Appendix I.4).
W2: What is new
We do not claim the agreement/validity distinction as new or identify construct validity. The introduction now states:
"Relative to the two audits compared in Section 2, we add joint reporting of same-item panel-minus-reference agreement by task slice and calibrated acceptance-threshold transfer between slices. The decision rule preceded the full run, but two amendments followed partial-data inspection (Section 3.4). We did not test whether acting on these measurements improves downstream decisions."
The restored Section 2 paragraph compares Norman et al.'s repeatability and position-bias checks and Chen et al.'s validity interventions with our measurements (Appendix E, Table 7): "Their exact-match/kappa comparison measures chance correction; our gap subtracts reference from panel agreement on the same items." Section 4.2 states: "The added contribution is joint reporting of agreement gaps, calibration and acceptance by task slice, together with conditional interpretations of the gap."
Table 1 is now cited in the Introduction and at the start of Section 4.1.
W3: Setup
Section 3.1 names the submitted models (all fourteen PPE traces in its footnote), the six-model common panel and its two secondary additions, and HelpSteer2 with Prometheus/Qwen. It states that RewardBench is one model under two prompts; Table 2 gives counts and Appendix A.1 exact submitted configurations.
Section 3.2 defines labels, same-item kappas, pair-gap medians and equal slice weights. Section 3.3 gives calibration fits, held-out-item exclusion, settings and support floors. Section 3.4 gives uncertainty, decision rules and chronology. Sections 4.3 and 5 report the common-panel and extension analyses introduced in Section 3.1.