acceptodds
Under review as a conference paper at ICLR 2027

What Can Be Corrected in LLM Judge Ratings? Identifiability, Measurement, and Targeted Adjustment

Abstract

Correcting ratings from LLM judges requires knowing whether associations with features such as verbosity, lexical overlap, or displayed position reflect bias or item quality. Ratings alone can identify differences in these associations across judges, but cannot separate an association shared by the panel from item quality. We formalize this limit with a hierarchical Bayesian measurement model that represents each ordinal rating in terms of latent item quality, judge leniency, and feature associations. We use this model as a diagnostic audit: human labels, controlled counterfactuals, or another external anchor are required to identify a panel-wide association. We therefore vary unidentified panel-wide slopes in sensitivity analyses. Targeted adjustment is a diagnostic control that removes only identifiable judge-specific deviations. On TREC web relevance and SummEval summarization, targeted adjustment has higher Spearman rank-correlation point estimates with human labels than full adjustment, although panel averaging matches or outperforms it. As a practical implementation layer, a 250,000-parameter amortized network matches sampling-based masked-rating reconstruction error (0.302 versus 0.304) while reducing per-dataset inference time from about 240 seconds to 1.3 ms. We recommend selecting a scoring rule through human-label validation and removing a panel-shared association only when an external anchor identifies the corresponding effect.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM-as-a-Judge, bias measurement, identifiability, item response theory, Bayesian hierarchical modeling, ordinal rating aggregation, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cYjqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how feature associations in LLM judge ratings (e.g. verbosity, lexical overlap, positional bias, etc.) can be corrected. Its main argument is that differences between judge LLMs can be estimated from ratings alone. The paper also argues that preferences shared by all the judges cannot be decoupled from the true item quality without external evidence.

The authors introduce a Bayesian measurement model to examine these effects, and propose a lightweight network for faster inference. They run tests on TREC and SummEval, showing that while targeted adjustment generally preserves human-label agreement better than full adjustment, they do not outperform simple panel averaging.

Critical strengths
  1. The paper makes a timely and important distinction between observable correlations and actual biases. It provides useful evidence through many experiments why this distinction is important.
  2. The authors run extensive tests. They also openly acknowledge negative results which I think is very important for transparent scientific progress.
Critical weaknesses
  1. Limited novelty and practical improvement The central identifiability problem has already been studied in psychometric research [1, 2], and related measurement models [3] and IRT-based approaches to LLM judge evaluation [4] already exist. More importantly, targeted adjustment does not outperform simple panel averaging.

  2. Uncertainty about the judge-specific effects The proposed correction depends on reliably estimating how individual judges respond to features. However, the paper shows that accounting for judge discrimination substantially changes the estimated lexical-overlap effects. This is concerning because lexical overlap explains much of the reported difference between adjustment methods.

[1] Looking at DIF From a New Perspective: A Structure-Based Approach Acknowledging Inherent Indefinability (Applied Psychological Measurement, 2019).

[2] DIF Statistical Inference Without Knowing Anchoring Items (Psychometrika, 2023).

[3] Additive Multilevel Item Structure Models with Random Residuals: Item Modeling for Explanation and Item Generation (Psychometrika, 2014).

[4] Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory (ICML, 2026).

Questions for authors
  1. What practical benefit does the proposed audit provide beyond panel averaging or simple judge-leniency correction? Is there a concrete decision that requires the estimated judge-specific feature effects?
  2. Given the instability of lexical-overlap effects in Appendix Q, what evidence supports treating these estimates as reliable judge-specific responses rather than differences in discrimination?
  3. Does Proposition 1 require stronger conditions on the rating graph? Connectivity and full-rank item features alone seem insufficient to identify judge-specific slopes when some judges rate very few overlapping items.
3 comments

Thank you for the helpful suggestions.

Novelty and practical benefit. We agree, and the paper credits the works you cite: Section 3.1 frames the result as the familiar DIF identification problem (Doebler, 2019), Related Work cites anchor-free DIF inference (Chen et al., 2023), Section 3.3 identifies the item model as AMIS-R (Cho et al., 2014), and the revision sharpens our distinction from Choi et al. (2026). To your question: no scoring decision in our experiments needs the judge-specific feature estimates. The panels we use for human-label alignment are fully crossed, so centered deviations cancel across each item's judges, and leniency-only and targeted scores agree within 0.001 (TREC 0.6805 vs. 0.6809; SummEval 0.5718 vs. 0.5726). The estimates would affect scores only when different judges rate different items. Their value is diagnostic: they show whether judge disagreements vary with features, whether the design identifies them, and whether they survive allowing discrimination. That guides where to add co-ratings, which interventions to run, and when to withhold a correction; averaging remains our default scorer.

Reliability of the overlap effects. We agree they are unresolved, and the revision explains why: with judge discrimination, the shared-slope transformation shifts each judge's centered slope by (aj−aˉ)t(a_j-\bar a)t, so judge contrasts inherit the shared-slope ambiguity. On your concern that overlap drives the gap between adjustment methods: that gap comes from the shared overlap association, not the judge-specific estimates. Removing only shared components matches full adjustment, and holding everything else at full adjustment, retaining only the shared overlap association raises the in-sample correlation from 0.673 to 0.685. The instability you identify therefore does not affect the scoring comparison.

Rating-graph conditions. You are right. We replaced connectivity and global feature rank with full rank of the co-rated contrast design for (1,xi)(1,x_i), which Appendix I proves necessary and sufficient. The TREC panel satisfies it (481 fully crossed items; contrast rank 54/54), as do the calibration and SummEval panels.

Thank you for following up. You are right that our answers needed to appear in the manuscript, and we have now uploaded the revised PDF. Each earlier point is there: the distinction from Choi et al. (§2), the qualified overlap class and discrimination limit (Table 1, §3.1, §6.1, Appendix Q), and the article-held-out SummEval result replacing the in-sample row (§5, Table 2, Appendix O). The analyses below are new (Appendices O and T).

GRM relation. Choi et al. fit a logistic GRM with a discrimination and thresholds per prompt variant and a latent quality per sample. It has no item features, and they leave position and verbosity effects to future DIF analysis. We fit that GRM to our panels, with judges in place of prompt variants, then added shared features (GRM-X) and judge-specific feature terms (GRM-DIF, our audit with per-judge thresholds).

(1) Panel-wide associations. Without labels, the GRM ranks items as averaging does (Spearman with human labels 0.689 vs. 0.690 on TREC; 0.598 vs. 0.597 on SummEval), so its alignment diagnostics raise no flag. Its θ\theta, however, contains every panel-wide feature association, and the likelihood, hence every GRM diagnostic, is identical however θ\theta is split into quality and shared response. The split changes alignment: removing the shared associations lowers it to 0.602 on TREC and 0.524 on SummEval, and removing only shared lexical overlap lowers SummEval to 0.354. The audit flags this association, shows that ratings cannot settle it (Proposition 1), and measures what is at stake; on SummEval the human labels show that overlap largely tracks quality. A GRM user who corrected for overlap would get no warning.

(2) Judge-specific departures. On SummEval the GRM marks llama-3.2-3b as the least discriminating judge (a=1.02a=1.02 vs. 2.2 to 2.9). GRM-DIF adds that its ratings rise with summary and source length relative to the panel, and this predicts its withheld ratings on held-out articles (+0.051+0.051 nats per rating, 95% CI [0.005,0.103][0.005,0.103]). A GRM can only report this as low discrimination.

(3) Where it adds little. For predicting withheld ratings on held-out TREC queries, judge-specific terms barely improve on the GRM (+0.001+0.001 nats per rating, CI [−0.001,0.003][-0.001,0.003]; SummEval overall +0.009+0.009, [−0.001,0.020][-0.001,0.020]). Once each judge has its own thresholds, the common-scale overlap heterogeneity is largely absorbed. This qualifies our earlier statement that the slopes predict ratings: they do relative to a common-scale model, not beyond a GRM. It reinforces Table 1's "lexical sensitivity unresolved", and §6.1 now says so.

SummEval interpretation. We read the result as support for averaging, and the decomposition (Appendix O) shows why each correction falls short. The gap between targeted adjustment and averaging is not caused by correction: leniency-only correction, which removes no feature response, falls equally short (0.572). With five ratings on a 0 to 3 scale, the panel mean takes only 10 distinct values over 180 summaries (47 tie at 2.0). Model-based scores break those ties according to which judge gave which rating, and within ties that ordering does not track human scores (ρ=−0.05\rho=-0.05, CI [−0.21,0.09][-0.21,0.09]). Keeping averaging's order and breaking its ties by the targeted score gives 0.577, close to 0.573, so the gap lies within the ties. Full adjustment's further loss comes from one shared association: removing only the shared summary-source overlap slope, the largest, gives 0.534, matching full adjustment (0.536), and the loss is between articles (article-level ρ\rho falls from 0.41 to 0.15). Ratings cannot tell whether this association is quality or bias; the labels indicate quality, so it should be retained. This is the workflow in §7: identify what a correction would remove, anchor it externally, and keep averaging unless held-out labels support another rule.

SummEval stability. Six articles (36 summaries) form one fold, but every summary is scored once, so the correlations pool all 180 summaries from 30 articles. Single folds are unstable (targeted minus full ranges from −0.09-0.09 to +0.29+0.29 across folds), so we do not interpret them alone. We repeated the five-fold split for nine more random article partitions and ran leave-one-article-out validation (75 more fits, same sampler settings, no divergences). Targeted adjustment falls below averaging in all ten partitions (mean −0.024-0.024; an article bootstrap averaged over partitions, which includes partition variability, gives [−0.046,−0.008][-0.046,-0.008]) and under leave-one-article-out (−0.034-0.034, [−0.061,−0.013][-0.061,-0.013]). Targeted exceeds full adjustment in all ten partitions (+0.035+0.035), but that difference remains unresolved across the 30 articles ([−0.024,0.096][-0.024,0.096]). The shortfall from averaging is therefore stable; the targeted-versus-full ordering is consistent in direction but limited by the number of articles, as the paper states.

I thank the authors for their response and the revised manuscript. In particular, I appreciate you fixing the rating-graph condition and for explicitly crediting previous literature.

That said, these clarifications also reinforce my main concern regarding novelty and practical utility. The underlying measurement model and identifiability limits are still closely tied to exiting DIF and IRT formulations. Moreover, the updated SummEval validation confirms that the targeted adjustment still doesn't outperform simple panel averaging. I am not convinced that the practical benefit beyond panel averaging or standard leniency correction is strong enough, and thus will be retaining my score.

Reviewer_zE5LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper deals with a difficulty in correcting LLM evaluation scores for things like verbosity and lexical overlap: it is hard to separate an evaluator's bias from real differences in quality. To address this, it proposes a hierarchical Bayesian model that separates the tendencies shared across evaluators from the differences specific to each evaluator. The authors compare how well each correction method agrees with human judgments on TREC and SummEval. They also run experiments that manipulate position and lexical overlap to check how observational analyses should be interpreted, and they examine an approximate model that reduces inference cost.

Critical strengths

This paper has the following strengths:

  • On the first, the question of whether removing the relationship between input features and scores is always the right correction is a useful one. Length or lexical overlap can reflect real quality rather than bias. A correction that removes every such correlation may also remove signal that should be there, so the paper’s framing of the problem matters.
  • On the second, the controlled experiments are the strongest part of the approach. Changing a document’s display position or adding query words while holding the content fixed gives a direct test of whether the observed effect is bias, which observational data alone cannot show.
Critical weaknesses

This paper has the following weaknesses:

  • The paper's Related Work cites studies that use GRM, but it should more explicitly distinguish what each study measures and what diagnostic purpose it serves. Prior work diagnoses how consistent an individual LLM's evaluations are across prompt variations and how well they align with human judgments. This study instead analyzes the tendencies that multiple LLM evaluators share with respect to document characteristics, along with the differences between evaluators. Adding this distinction to the description would make the paper's position clearer.
  • It is unclear how the evaluator-specific differences in lexical overlap should be interpreted. Because the proposed model is meant to diagnose evaluator characteristics, further validation is needed to show how well these estimates explain actual evaluation behavior.
  • For SummEval, the correction model is estimated from human ratings, and agreement with human ratings is then measured on the same items. From the current results alone, it is hard to judge whether the same correction effect would hold for new summaries. As was done for TREC, the model estimation and evaluation items should be separated in a validation.
Questions for authors

See the weaknesses above.

2 comments

Thank you for the helpful suggestions.

Relation to GRM work. Choi et al. (2026) use a graded-response model to diagnose each judge's consistency under prompt variations and its alignment with human assessments. We instead ask which item-feature associations differ across a panel of judges, which differences depend on response-scale assumptions, and which shared associations cannot be separated from quality without an external anchor. Related Work now draws this distinction. We claim novelty for neither the measurement model nor the identification limit.

Interpreting judge differences in overlap. We agree the evidence does not establish differential lexical sensitivity. The estimated slopes do predict real ratings: removing the overlap and formality slopes raises masked-reconstruction error from 0.304 to 0.319, and judge parameters fitted on half the queries keep 87% of the reconstruction gain on the other half. Whether that behavior is lexical, however, depends on how much of the overlap association is shared, which ratings cannot identify once judges differ in discrimination; counterfactual judge contrasts also do not reproduce the observational ordering. Table 1 and the main text now present these as model-dependent observational heterogeneity that can guide follow-up interventions, not as validated judge traits.

SummEval validation. You were right that the original result reused training labels. We added five-fold validation that holds out whole source articles: each fold fits on 24 articles (144 summaries) and scores the six held-out articles (36 summaries) without their labels. Out-of-fold Spearman correlations are 0.597 for averaging, 0.573 for targeted adjustment, and 0.536 for full adjustment (targeted minus full +0.035+0.035, 95% CI [−0.028,0.095][-0.028,0.095]; targeted minus averaging −0.027-0.027, CI [−0.057,−0.002][-0.057,-0.002]). All fits converged (R^<1.01\hat R < 1.01, no divergences). These results replace the main SummEval row; the in-sample result remains only as a labeled mechanism check.

Thanks for the authors' response. Some of my concerns are addressed, but some are not. It might be helpful if the authors revised the manuscript, but the authors didn't. Thus, I cannot decide whether the current response will be mirrored in the main manuscript or not; and two questions become remain unanswered. Thus, I retain my score.

  1. GRM relation: Could the authors show results where this analysis gives information that the existing GRM-based diagnostics do not? Such result might strengthen the argument.

  2. SummEval validation: How does the paper interpret the result that the correction methods score below averaging (0.573 and 0.536 vs. 0.597), and can this difference be considered stable given only 6 held-out articles and 36 summaries?

Reviewer_DN5AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates which feature associations (e.g., verbosity, lexical overlap, displayed position, formality) in LLM-as-a-judge ratings can be identified and corrected without external anchors. The authors formulate ratings using an ordered-probit hierarchical Bayesian measurement model (AMIS-R). They prove that ratings alone can identify centered judge-specific deviations δj\delta_j, but cannot separate a panel-wide shared slope βˉ\bar{\beta} from latent item quality θi\theta_i (Proposition 1), as the ratings likelihood is entirely flat along this ridge. Point estimates produced by hierarchical models are shown to be selected by identifying assumptions on item residuals rather than data.

To demonstrate the consequences of this limit, the paper introduces "targeted adjustment" as a diagnostic control that removes only judge-specific deviations while retaining shared slopes. On TREC web relevance and SummEval, targeted adjustment maintains higher rank correlation with human labels than full adjustment (which subtracts all feature slopes, including signal correlated with true quality), though it does not outperform simple panel averaging. The paper also conducts paired counterfactual experiments and introduces a 250k-parameter amortized network (SBC-validated) that speeds up inference from 240 seconds to 1.3 ms.

Critical strengths
  1. Clear Conceptual Separation of Evaluation Goals (§1, §7): The paper provides a valuable taxonomy separating three distinct evaluation tasks: predicting withheld judge ratings (where graded-response IRT is best), scoring/ranking items against human labels (where simple panel averaging is best), and auditing which feature responses can be statistically disentangled from latent quality. Clarifying that rating reconstruction does not justify removing feature components is an important contribution to the LLM evaluation literature.
  2. Methodological Transparency and Rigorous Sensitivity Audits (§3.3, §6, Figure 1, Appendix J): The authors are commendably candid that point estimates of shared slopes are chosen by identifying assumptions (E[ei∣xi]=0E[e_i \mid x_i] = 0) rather than ratings data. The profile log-likelihood flatness check (varying by <4.5×10−13< 4.5 \times 10^{-13} nats), the breakdown analysis (robustness values), and the 11-point MCMC prior sensitivity sweep provide a textbook demonstration of partial identification in ML.
  3. Intellectual Honesty on Negative / Null Results (§6.4, Table 2, Table 7, Appendix P): Unlike many papers that attempt to claim SOTA scoring superiority, the authors transparently emphasize that targeted adjustment is a diagnostic control that does not beat panel averaging. They also appropriately apply the Nadeau-Bengio (2003) corrected resampled tt-test to account for training set overlap in their 50-fold evaluation, acknowledging that the difference is not statistically significant (p=0.078p = 0.078).
  4. Controlled Interventions vs. Observational Associations (§6.2, §6.3, Appendix B–D): The paired counterfactual experiments demonstrate that observational associations do not imply causal bias: changing position from 1 to 5 has a negligible effect (+0.035), and appending missing query terms yields a contrast near zero (+0.005), showing that anchors identify specific interventions rather than abstract feature names.
Critical weaknesses
  1. Flawed Sufficient Conditions in Proposition 1 (Active Counterexample): Proposition 1 claims that fixed location/scale, a connected judge–item observation graph, full-rank covariate variation, and J≥2J \ge 2 judges (Appendix I, C1–C5) are sufficient to identify centered judge-specific deviations δj\delta_j. This is strictly insufficient for sparse rating graphs.

    • Counterexample: Consider two judges J1J_1 and J2J_2 who jointly rate only a single item i=0i=0 with x0=0x_0 = 0 (hence the bipartite graph is connected). J1J_1 and J2J_2 then rate disjoint sets of items whose covariate matrices individually and jointly have full rank. For any non-zero vector Δ\Delta, one can shift δ1→δ1+Δ\delta_1 \to \delta_1 + \Delta and absorb it into J1J_1's exclusive item parameters (θi→θi−Δ⊤xi\theta_i \to \theta_i - \Delta^\top x_i), while shifting δ2→δ2−Δ\delta_2 \to \delta_2 - \Delta and absorbing it into J2J_2's exclusive item parameters, preserving ∑jδj=0\sum_j \delta_j = 0. Because x0=0x_0 = 0, the shared item’s linear predictor is unaffected (δj⊤x0=0\delta_j^\top x_0 = 0). Thus, all observed rating probabilities remain identical, and δj\delta_j is unidentifiable.
    • Consequence: The proposition needs to either constrain the design to fully crossed panels or require that the sub-graph of overlapping ratings has full rank on the interaction design matrix. While the empirical TREC panel is nearly fully crossed (missing only 4/4,850 entries), a core mathematical theorem must be sound as stated.
  2. Information-Routing Omission in the Amortized Item Decoder (Appendix R): Appendix R states: "A shared item-encoder MLP embeds each item's one-hot ratings... A masked mean pools the item embeddings into one global code hh. Three decoders map hh to Gaussian approximate posteriors... an item decoder predicts item qualities {θi}\{\theta_i\}." Because masked mean pooling is permutation-invariant, an item decoder that takes only the pooled global vector hh cannot preserve the one-to-one mapping to individual items. In standard neural process / set architectures, an item-level decoder must condition on both hh and the local item encoding eie_i. As written, the architectural description contains a structural omission that renders the reported item recovery (ρ≈0.52–0.54\rho \approx 0.52\text{--}0.54) and amortization reproducibility logically incomplete.

  3. Empirical Instability of the Headline Lexical Overlap Finding (Appendix Q): Table 1 highlights lexical overlap as the premier "partly judge-specific" feature (σov=0.141,P(σ>0.05)=1.00\sigma_{\text{ov}} = 0.141, P(\sigma > 0.05) = 1.00). However, when judge discrimination is incorporated in model M2 (Appendix Q), the scale and residual response components have a high posterior correlation of −0.58-0.58 (classified as non-separable), and the judge deviation rank agreement between M0 and M2 collapses to Kendall’s τ=0.16\tau = 0.16. This indicates that apparent overlap heterogeneity may largely be an artifact of differing judge response scales rather than genuine differential sensitivity to lexical overlap, weakening the empirical certainty of the taxonomy presented in §6.1.

  4. Limited Incremental Novelty Relative to Recent Literature: The core insight that bias and quality cannot be separated without external anchors, that profile likelihoods over bias slopes are flat (0.0000 nats), and that regularization priors select the point estimate was already established for pairwise LLM evaluations by Xu et al. (July 2026, arXiv:2607.02104). Furthermore, Choi et al. (ICML 2026) concurrently applied IRT to LLM judges, and AMIS-R is the standard model from Cho et al. (2014). Since targeted adjustment does not outperform panel averaging and offers no definitive scoring replacement, the paper needs to more concretely articulate what actionable decisions the multi-judge audit enables for practitioners.

  5. Confounded Lexical Overlap Counterfactual Manipulation (§6.3 vs. Appendix D): In §6.3, the counterfactual manipulation for lexical overlap appends "Additional query terms: {missing query terms}." at the end of the text. This artificial keyword-stuffing format likely triggers coherence and formatting penalties in modern LLMs, neutralizing any positive overlap response (+0.005). Appendix D confirms that a natural "templated query-focused preamble" increases ratings by +0.128. Characterizing the near-zero contrast in §6.3 as evidence that models lack causal sensitivity to lexical overlap is misleading; it reflects a penalty for unnatural formatting.

Questions for authors
  1. Proposition 1 Conditions: How will you revise the sufficient conditions in Proposition 1 to rule out the sparse-graph counterexample detailed above? Does the empirical TREC data graph satisfy your revised formal conditions?
  2. Amortized Architecture: Does the item decoder in Appendix R receive local item encodings in addition to the global pooled vector hh? Please specify the exact tensor dimensions, conditioning mechanism, and whether permutation equivariance was empirically verified.
  3. Overlap Scale Artifact: Given that Kendall’s τ\tau between M0 and M2 drops to 0.160.16 in Appendix Q, how do you justify classifying lexical overlap as a firmly established "partly judge-specific" feature in Table 1?
  4. Actionable Practitioner Guidance: Since targeted adjustment does not beat panel averaging and shared slopes cannot be identified without anchors, what concrete workflow or decision tree should an evaluation practitioner follow after running this HBM audit?
  5. Pairwise Generalization: How does this measurement framework formally translate to pairwise preference evaluation (e.g., Bradley–Terry models), where most modern LLM benchmark debiasing efforts take place?
1 comment

Thank you for the careful review. Your counterexample was correct, and we have fixed Proposition 1.

1. Proposition 1. We replaced connectivity and global feature rank with full rank of the co-rated contrast design: with zi=(1,xi⊤)⊤z_i=(1,x_i^\top)^\top, stack rows (ej−ej′)⊤⊗zi⊤(e_j-e_{j'})^\top\otimes z_i^\top for judges co-rating item ii; on the centered judge-coefficient space this design must have rank (J−1)(p+1)(J-1)(p+1). Appendix I proves this is necessary and sufficient; a fully crossed subset with rank⁡[1,X]=p+1\operatorname{rank}[1,X]=p+1 is a simple sufficient case. The TREC panel satisfies it (481 of 485 items fully crossed; contrast rank 54/54), as do the calibration (24/24) and SummEval (20/20) panels.

2. Amortized item decoder. Yes; our description omitted the local code. Ratings (B,N,5,4)(B,N,5,4) and features (B,N,6)(B,N,6) give 26 inputs per item; a shared encoder (26→128→128→12826\to128\to128\to128) yields local codes hih_i, and masked mean pooling yields hh. The item decoder applies a shared 256→128→64256\to128\to64 MLP to [hi,h][h_i,h], with mean and log-variance heads per item; the judge decoder uses hh plus identity and type embeddings. We verified item-permutation equivariance on both checkpoints (100 permutations per case, including padding and missing ratings; maximum float32 deviation below 3×10−63\times10^{-6}). Appendix R gives the full specification; §6.5 states the SBC scope.

3. Lexical overlap and discrimination. We agree; Table 1 now reads "Heterogeneous under fixed scale; lexical sensitivity unresolved." Your question also shows why. With discrimination, ηij=αj+aj(θi+μ⊤xi)+dj⊤xi\eta_{ij}=\alpha_j+a_j(\theta_i+\mu^\top x_i)+d_j^\top x_i, the map μ↦μ+t\mu\mapsto\mu+t, θi↦θi−t⊤xi\theta_i\mapsto\theta_i-t^\top x_i leaves the likelihood unchanged but shifts each centered total slope by (aj−aˉ)t(a_j-\bar a)t, so judge contrasts inherit the shared-slope ambiguity and only the residual djd_j is fixed by the ratings (§3.1, Appendix Q). M0 agrees weakly with these residuals (τ=0.16\tau=0.16); agreement with M2's total slopes is 0.16, 0.73, or 0.78 for μov=0\mu_{ov}=0, 0.25, or 0.5. The −0.58-0.58 you cite is the across-judge correlation of discrimination with residual overlap response. Separating lexical sensitivity from discrimination thus needs the same external anchor as the shared slope. Scoring is unaffected: our alignment panels are fully crossed, so centered deviations cancel within each item, and leniency-only and targeted scores agree within 0.001 (TREC 0.6805 vs. 0.6809; SummEval 0.5718 vs. 0.5726); the full-versus-targeted gap comes from shared associations.

4. Overlap counterfactual. We agree; §6.3 now says the +0.005+0.005 contrast does not show absent lexical sensitivity. A placebo tests your formatting explanation: the same sentence with neutral words ("Additional reference terms: …"), matched in placement, term count, and length, with overlap unchanged. On the five-judge panel, marked append gives +0.020  [−0.024,0.064]+0.020{\mskip5mu} [-0.024,0.064], placebo +0.044  [0.012,0.080]+0.044{\mskip5mu} [0.012,0.080], and marked minus placebo −0.024  [−0.060,0.012]-0.024{\mskip5mu} [-0.060,0.012]. A format penalty would hit both arms, yet the placebo raises ratings and query terms add nothing detectable (unless judges react to the label word itself). A 100-item marked append gives −0.032  [−0.065,−0.002]-0.032{\mskip5mu} [-0.065,-0.002]; the templated preamble (+0.128  [0.088,0.168]+0.128{\mskip5mu} [0.088,0.168]) also changes framing and length. Each anchor identifies one intervention; none licenses subtracting the shared overlap slope.

5. Contribution and practical use. We agree the identification limit and model family are established; for pairwise judges, Xu et al. already show a flat profile likelihood, prior-selected debiased scores, and anchor designs, which the revision now credits. Our contribution is the multi-judge ordinal case: an exact design condition for identified judge contrasts, the discrimination limit above, and what retaining or removing each component does to human-label alignment. We do not claim a better scorer. §7 gives four steps: (i) check overlap rank, adding co-ratings if deficient; (ii) check stability under discrimination and priors, deferring unstable corrections; (iii) anchor any shared correction with human labels or matched interventions; (iv) choose the scoring rule on grouped held-out labels, keeping averaging unless beaten.

6. Pairwise evaluation. The ambiguity carries over: with logit⁡Pj(a≻b)=(θa−θb)+(βˉ+δj)⊤(xa−xb)\operatorname{logit}P_j(a\succ b)=(\theta_a-\theta_b)+(\bar\beta+\delta_j)^\top(x_a-x_b), the map βˉ↦βˉ+t\bar\beta\mapsto\bar\beta+t, θa↦θa−t⊤xa\theta_a\mapsto\theta_a-t^\top x_a leaves every comparison unchanged. Judge contrasts are identified if all judges rate common comparisons whose feature differences span Rp\mathbb{R}^p (general condition in Appendix I.1). Leniency cancels within a comparison; display order is a pair-level term that balanced order swaps can anchor.

Comments on the paper

2 comments
Area_Chair_f7NKOpenReview ↗

Discussion Period - No Rebuttal Yet

Dear Authors,

We have not yet received a response from you with respect to Submission 32401. To take advantage of the discussion period, please submit a response as soon as possible.

Best Regards,

AC

Area_Chair_f7NKOpenReview ↗

Rebuttal Received

Dear Reviewers,

The authors have responded. Please read their response and reply as soon as possible so that any unresolved issues can be clarified.

Best Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.