acceptodds
Under review as a conference paper at ICLR 2027

Agreement Is Not Validity: A Preregistered Audit of Human-Anchored LLM-Judge Evaluation

Abstract

Agreement among automatic judges is frequently reported as evidence that an LLM-based evaluator is reliable. Agreement, however, is a reproducibility property of a panel; validity concerns whether its verdicts track an independent target. We preregistered and executed an artifact reanalysis of this distinction using three public, item-level, human-referenced artifacts spanning twelve author-defined artifact–domain cells. It contrasts judge–judge Cohen's κ with the same judges' human-anchored κ and tests transport of a learned mapping from transformed reward scores to agreement with the released reference. The design uses artifact-macro aggregation, equal domain weight, and a single 10,000-draw BCa bootstrap stream. The registered frozen computation returned GRAY. The H2 macro gap was 0.129636 (95% BCa CI [0.109357,0.151594]): distinguishable from zero but below the registered 0.15 materiality margin. Calibration transport showed the same intermediate pattern, with macro ΔECE 0.044192 (95% BCa CI [0.037323,0.070487], Holm-adjusted p=0.0002), below its 0.05 margin; ΔRisk was GRAY-null because only two domains had valid support. Excess agreement relative to the released human reference is measurable and artifact-dependent; it does not identify construct validity. Exploratory exclusion of PPE human ties reduces the H2 macro to 0.073393. Reference noise and label construction limit the interpretation of the gap.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-judge evaluation, reward models, agreement metrics, calibration transport, human-referenced evaluation, preregistered audit

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gKjDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper reanalyzes the three benchmark (RewardBench, PPE, and RewardBench 2) to evaluate gaps between judge-judge agreement and judge-human agreement, as well as the cross-domain transfer of calibration. Although the agreement gap and ECE are statistically distinguishable as zero, the sample shows a Gray outcome because it does not reach the preregistered materiality margin. Selective risk means that there are not enough domains that meet the evaluation criteria. The authors limit the scope of interpretation, stating that construct validity cannot be identified through such analysis.

Critical strengths

None

Critical weaknesses
  • Overall, explanation flow is unclear. Lack of introduction to key terminology makes difficult to follow the motivation and procedures. More emphasis is placed on details related to execution than on explanations of what and why it is being analyzed.

  • The distinction between an agreement and validity is already introduced as a known issue in related works (Sec 2, Also Table 1 is not mentioned anywhere.) Since this analysis also fails to identify construct validity. Therefore, it is unclear what new insights the proposed agreement gap and calibration analyses provide beyond existing work.

  • Experimental setup is scattered across the manuscript. Therefore, make difficult to understand overall experimental flow (selected model, configuration, data and methods).

Questions for authors

Refer to weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer gKjD

Thank you for identifying the missing motivation, definitions and scattered setup.

W1: Explanation and motivation
The opening now gives the question and motivation before the method:
"When automatic judges agree with one another, does that agreement also hold against the human reference? We report a preregistered audit of this question on three public preference datasets, using released judge scores."
"Agreement among automatic judges is often reported as evidence that an evaluation is reliable, especially when human labels are scarce. It is a weaker statement than agreement with a human reference, and the difference between the two can be measured wherever both are available."
"A judge chooses a response from its scores; the reference records the released human choice."
"Raw agreement is the fraction of matching choices; Cohen's kappa corrects it for chance under the observed label frequencies."
"Calibration asks whether a predicted probability matches how often the judge's chosen response matches the reference."
Section 1 also defines the agreement gap, task slice, ECE and selective risk before results. Section 1 gives the opening; the abstract states the outcome, and "Applying the submitted preregistered rule" now starts Section 4.1.
Calibration average: With five bins instead of the registered ten it is 0.053 (Section 4.2). Whether the calibration average lies below its margin therefore depends on the bin count. The registered settings define the preregistered outcome, which is unchanged; no outcome is assigned to any other setting (Appendix I.4).

W2: What is new
We do not claim the agreement/validity distinction as new or identify construct validity. The introduction now states:
"Relative to the two audits compared in Section 2, we add joint reporting of same-item panel-minus-reference agreement by task slice and calibrated acceptance-threshold transfer between slices. The decision rule preceded the full run, but two amendments followed partial-data inspection (Section 3.4). We did not test whether acting on these measurements improves downstream decisions."
The restored Section 2 paragraph compares Norman et al.'s repeatability and position-bias checks and Chen et al.'s validity interventions with our measurements (Appendix E, Table 7): "Their exact-match/kappa comparison measures chance correction; our gap subtracts reference from panel agreement on the same items." Section 4.2 states: "The added contribution is joint reporting of agreement gaps, calibration and acceptance by task slice, together with conditional interpretations of the gap."
Table 1 is now cited in the Introduction and at the start of Section 4.1.

W3: Setup
Section 3.1 names the submitted models (all fourteen PPE traces in its footnote), the six-model common panel and its two secondary additions, and HelpSteer2 with Prometheus/Qwen. It states that RewardBench is one model under two prompts; Table 2 gives counts and Appendix A.1 exact submitted configurations.
Section 3.2 defines labels, same-item kappas, pair-gap medians and equal slice weights. Section 3.3 gives calibration fits, held-out-item exclusion, settings and support floors. Section 3.4 gives uncertainty, decision rules and chronology. Sections 4.3 and 5 report the common-panel and extension analyses introduced in Section 3.1.

Reviewer_8sWNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits the difference between inter-judge agreement and agreement with released human references across three public artifacts. It also evaluates transfer of score-to-reference-correctness calibration across domains. The registered analysis yields intermediate, artifact-dependent results rather than a material mismatch under the authors' thresholds.

Critical strengths

S1. The estimands and their limitations are stated unusually clearly. This paper distinguish reference agreement from construct validity and do not interpret the proposed gap as an identified shared-error rate.

S2. The paper reports important heterogeneity and sensitivity rather than presenting only a positive macro result. The negative cell estimates, artifact deletions, tie exclusion, and Gray outcome make the audit more informative.

Critical weaknesses

W1. The uncertainty calculation omits known cross-cell dependence. Section 7, lines 414–425, states that covariance induced by overlapping PPE domains is not represented in either the bootstrap or the BCa jackknife. Since macro intervals feed the registered gates, the central inferential outputs require a joint item-level resampling analysis that preserves domain memberships.

W2. The operational interpretation of the gap remains underdeveloped. In Section 5.3, removing PPE reduces the macro gap to 0.018122, and excluding PPE human ties reduces it to 0.073393. These are useful diagnostics, but they also show substantial dependence on reference construction and the chosen target mixture. A protocol sensitivity analysis or a controlled reference-noise analysis would help explain what a given gap warrants in practice.

W3. The incremental decision value over the audits discussed in Section 2 is not fully demonstrated. The narrower estimands are distinct, but an example where they change an evaluation decision beyond existing reference-agreement and calibration summaries would strengthen the contribution.

Questions for authors

Q1. How do the macro intervals and registered gate outcomes change under a bootstrap over unique PPE items that preserves all domain memberships?

Q2. Can the tie-handling sensitivity be evaluated without changing the substantive item population, for example through explicitly compared labeling protocols?

Q3. What concrete action should an evaluator take after observing a Gray result of the magnitude reported here, and what additional data would resolve the selective-risk analysis?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8sWN

W1/Q1: Shared-item uncertainty
Yes, retaining the registered analysis. Shared-item agreement resampling was preregistered in the extension and run before the reviews; calibration and the joint jackknife were added after the reviews.
Items are drawn once across memberships and deleted everywhere for the joint jackknife; calibration maps are refitted.
95% BCa intervals | Submitted | Shared-item
Gap | [0.109,0.152] | [0.110,0.151]
ECE | [0.037,0.070] | [0.037,0.067]
ECE Holm-adjusted p | <0.001 | <0.001
Point estimates and gates are unchanged; new intervals do not amend registration.
Matched sampling/deletion units do not prove nominal BCa coverage for kappa medians or refitted isotonic estimators.
Section 4.1 and Appendix E.6 give methods; Tables 8-9: intervals/decisions.
Calibration average: With five bins instead of the registered ten it is 0.053 (Section 4.2). Whether the calibration average lies below its margin therefore depends on the bin count. The registered settings define the preregistered outcome, which is unchanged; no outcome is assigned to any other setting (Appendix I.4).

W2: Meaning of the gap
A positive gap alone cannot diagnose shared judge error or reference noise. Reference construction remains confounded with task. With a hypothetical balanced binary target and conditionally independent symmetric errors, 90%-accurate judges and an 80%-accurate reference give a 0.16 gap; this does not estimate PPE reliability.
Submitted reaverages: 0.018 without PPE; 0.148 without RewardBench; 0.186 without RewardBench 2.
Opposite slice signs cancel in 0.018; this does not establish negligibility.
Appendix F.7, Table 16 gives reaverages without intervals/decisions; Appendix F.3 gives the conditional reference-noise construction, Appendix I hypothetical label perturbations/readings, and Q2 same-item comparisons.
On RewardBench, median judge-reference kappa is 0.653 to 0.951 by slice, exceeding the corresponding judge-judge medians; these reward models are publicly benchmarked on RewardBench, and we cannot quantify its influence on their development.

W3: Decision value
Partly. A post-review exploratory check, specified before it was computed, asks this on PPE, where fourteen judges are available. Ranking the judges by their mean agreement with the other judges and by their agreement with the reference gives similar orders (Spearman correlation 0.70 to 0.93 across the six slices; mean 0.83). The top judge differs in all six slices: panel agreement selects Athene-RM-8B and reference agreement selects Athene-RM-70B, whose reference kappa is higher by 0.014 to 0.042 (mean 0.021). Panel agreement therefore selects a different judge in every PPE slice; no equivalence or deployment benefit was tested. These are point descriptions on released labels, without intervals; excluding the verbosity baseline gives the same selections.
Incremental downstream utility over existing audits was not tested. Section 4.1 gives this result; Appendix I.1, Table 29 gives both variants.

Q2: Same-item tie scoring
Yes. The submitted nominal-tie mean is 0.130 kappa points, versus 0.113 under same-item half-credit scoring and 0.073 with human ties excluded; the second same-item variant, fixed tie-aware PPE judge mapping, gives 0.086. These are point estimates without intervals; larger gaps mean more excess agreement.
Exclusion also changes the item population and label frequencies. Half-credit scoring has no validated materiality margin or interval, and none of these alternatives independently validates tie semantics (Section 4.1; Appendix F.5).

Q3: Submitted Gray and selective-risk support: action
For agreement, Gray means neither material nor negligible under the rule: report judge-reference beside judge-judge agreement by slice; panel agreement cannot substitute for agreement with human labels.
At the 0.80 threshold, no PPE configuration reached 20% coverage under either map: all fourteen configurations failed that floor in each of the six slices (Appendix Table 14). A future risk analysis needs a prospectively fixed threshold with at least 25 accepted items and 20% coverage under both maps; repeated independent annotations could assess reference reliability, but would not guarantee that support.
Support requires 25 reference matches and 25 mismatches per fit, 25 accepted items and 20% coverage under both maps for a risk contrast, and three supported slices per average, checking ECE and selective risk separately.
Submitted selective risk is supported only in Chat (-0.019) and RewardBench 2 Safety (-0.039); these negative contrasts mean lower reference-mismatch rates under transfer among accepted items.
Actions: Sections 1 and 6; floors: Section 3.3; count-only, coverage-only and combined slice shortfalls: Appendix F.6, Table 14.

Reviewer_z31VOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper reanalyzes three public artifacts to compare judge-judge agreement with judge-reference agreement and assess calibration transport across domains. Both primary macro effects are distinguishable from zero under the reported analysis but below the registered materiality thresholds. The paper appropriately frames these as reference-agreement measurements rather than direct evidence of construct validity.

Critical strengths

I appreciate the willingness to report an intermediate outcome and the unusually clear disclosure of non-blind amendments and implementation checks. The tie, reference-noise, weighting, and artifact-deletion analyses are informative: they show why the same aggregate gap can have several explanations. The prevalence-only calibration comparison and matched-coverage risk diagnostic also make the interpretation more useful than ECE alone.

Critical weaknesses

The main statistical concern is that the bootstrap independently resamples overlapping PPE cells. This omits cross-cell covariance from the macro uncertainty and can make intervals too narrow; disclosure does not establish nominal coverage. Interpretation is also sensitive to the measurement design: excluding PPE human ties reduces the macro gap from 0.130 to 0.073, and removing PPE reduces it to 0.018. These sensitivities, limited judge coverage outside PPE, and the non-blind addition of RewardBench 2 make the strongest supported contribution a descriptive audit of this roster. The near-zero matched-coverage risk contrast further limits practical conclusions from the ECE transport gap.

Questions for authors
  1. Can you recompute macro intervals and H3 inference using shared item-level resampling across all PPE memberships, with a corresponding joint jackknife, while retaining the original analysis as the registered result?
  2. How stable are the findings under a common judge panel across artifacts and tie handling validated independently of the reported outcomes?
1 comment
AuthorsOpenReview ↗

Response to Reviewer z31V

Thanks.

Q1: Shared-item uncertainty
Yes, retaining the registered analysis. Shared-item agreement resampling was preregistered in the extension and run before the reviews; calibration and the joint jackknife were added after the reviews.
Each item is drawn once across its memberships and deleted everywhere for the matching jackknife, with calibration maps refitted each time.
95% BCa intervals | Submitted | Shared-item
Gap | [0.109,0.152] | [0.110,0.151]
ECE | [0.037,0.070] | [0.037,0.067]
ECE Holm-adjusted p | <0.001 | <0.001
Point estimates and gate outcomes are unchanged; new intervals do not amend the registration.
The shared-item ECE percentile interval is [0.023,0.052]; nominal BCa coverage is unproven. One agreement endpoint missed its Monte Carlo check; calibration extreme-tail precision remains unresolved.
Section 4.1 and Appendix E.6 describe the method; Appendix Tables 8-9 give intervals and every decision component.
Calibration average: With five bins instead of the registered ten it is 0.053 (Section 4.2). Whether the calibration average lies below its margin therefore depends on the bin count. The registered settings define the preregistered outcome, which is unchanged; no outcome is assigned to any other setting (Appendix I.4).

Measurement design and amendments
A descriptive audit of evaluated judges/references. Submitted reaverages: 0.018 without PPE, 0.148 without RewardBench; cancellation does not establish negligibility (Appendix F.7, Table 16; no separate intervals/decisions).
Answer-position variation and RewardBench 2 addition followed partial-data inspection. Adding RewardBench 2 moved the average from 0.186 to 0.130, below the material margin; no interval/decision was computed for the nine-slice average. Section 3.4 and Appendix D give chronology; Appendix F gives reference-noise, weighting and prevalence-only diagnostics.

Q2: Stability under a common judge panel
Partly yes. A post-review sensitivity analysis uses the same six repository model names across all three datasets. Mean gaps are 0.278 on PPE, -0.026 on RewardBench and 0.058 on RewardBench 2; the twelve-slice mean is 0.147 (95% item-bootstrap interval [0.133, 0.159]).
All six PPE gaps remain positive under both analyses. The submitted and common-panel gaps are 0.267 and -0.026 for RewardBench Safety, and -0.176 and -0.010 for RewardBench 2 Safety; RewardBench Chat is negative under both. Calibration companions do not establish generalization beyond the submitted judges: RewardBench Chat is unsupported, and Safety's descriptive ECE contrast is 0.012 versus submitted 0.173. The common-panel ECE mean is 0.021373 over eleven supported slices versus submitted 0.044 over twelve. Its point exceeds its percentile interval [0.001649, 0.020798], whose supported-slice set varies across draws; BCa is unavailable and companion bootstraps were not independently reproduced (Appendix K). These are descriptive companions, not a substantive calibration comparison. We do not extend the calibration-transfer result beyond the submitted judges. RewardBench 2 Precise IF changes from 0.105 to 0.172.
PPE reuses submitted traces. Scoring configurations are unattested, reference construction remains confounded with task, and intervals resample items, not judges.
On RewardBench, median judge-reference kappa is 0.653 to 0.951 by slice, exceeding the corresponding judge-judge medians; these reward models are publicly benchmarked on RewardBench, and we cannot quantify its influence on their development.

Run 1 could not estimate RewardBench because of singleton-list score parsing; nine PPE/RewardBench 2 gaps and companions had been seen before the independently reviewed correction and complete rerun. Run 2 reproduced the 63 pre-specified unaffected targets and frame/RNG records exactly; the defect did not affect the submitted analysis. Here frame/RNG records means the sampling frame and saved random-number-generator states. Both runs and the independent recount are reported in Section 4.3 and Appendix K.

Q2: Independent tie validation
We have not independently validated tie labels. Submitted same-item alternatives give 0.113 (half-credit) and 0.086 (fixed tie-aware mapping), versus 0.130 nominal and 0.073 after exclusion; exclusion changes the population. None validates tie semantics (Section 4.1; Appendix F.5).

Matched-coverage risk
In the submitted analysis, at matched top-80% coverage, the mean risk contrast is -0.003 (slice range -0.014 to +0.003; Figure 6).
The Chat example illustrates threshold behavior; incremental downstream utility over existing audits was not tested (Section 4.2; Appendix I).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

Submitted twelve-slice judge-judge agreement exceeds judge-reference agreement by 0.130 Cohen's kappa points (95% interval [0.109, 0.152]).
The interval excludes zero; seven slices reach 0.150, but the average misses this margin; registered calibration also fails its rule: inconclusive (Gray).
Six qualifying slices overlap on PPE.
Submitted calibration transfer increases expected calibration error by 0.044 (95% interval [0.037, 0.070]), below its 0.050 margin under the registered binning.
The submitted selective-risk contrast was not estimable; two amendments followed partial-data inspection.

What we changed because of your reviews

  • [8sWN W1/Q1; z31V Q1] Shared-item agreement resampling was preregistered in the extension and run before the reviews; calibration and the joint jackknife were added after the reviews.
  • [8sWN Q2/Q3] Same-item tie comparisons and reachable-risk requirements (Sections 4.1-4.2).
  • [8sWN W3] Panel agreement therefore selects a different judge in every PPE slice; no equivalence or deployment benefit was tested. (Section 4.1).
  • [z31V Q2] The calibration companions do not establish generalization beyond the submitted judges. (Section 4.3).
  • [gKjD] Rewrote motivation, related work and setup (Sections 1-3).
  • Calibration average: With five bins instead of the registered ten it is 0.053 (Section 4.2). Whether the calibration average lies below its margin therefore depends on the bin count. The registered settings define the preregistered outcome, which is unchanged; no outcome is assigned to any other setting (Appendix I.4).

What we did not do
We did not independently validate tie labels, separate reference construction from task, evaluate incremental downstream utility, or resample judges.

Sections 2 and 4.2 distinguish our same-item panel-minus-reference gap from Norman et al.'s chance-correction comparison.

Retitled from "Agreement Is Not Validity: A Preregistered Audit of Human-Anchored LLM-Judge Evaluation" to "Judge Agreement Versus Human-Reference Agreement: A Preregistered Audit of Reward-Model and LLM Judges".

For 93 RewardBench items, judge scores came from an earlier response-text revision than documented. Correcting the documented revision left ordered calculation inputs unchanged. Recomputed points, support and decisions matched; intervals were retained because inputs and random streams were identical, without rerunning the full bootstrap (Appendix E.2). These were 93 primary rows among 430 changed rows.

Independent code reproduced registered and shared-item point estimates and decisions, common-panel deterministic targets within tolerance, and judge-selection values exactly (Appendices E.6, K, I.1). Common-panel primary interval checks passed; companion bootstraps were not independently reproduced. One shared-item agreement endpoint missed its Monte Carlo check; calibration extreme-tail precision remains unresolved.

Supplement: released at https://anonymous.4open.science/r/agreement-audit-2027.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.