- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how feature associations in LLM judge ratings (e.g. verbosity, lexical overlap, positional bias, etc.) can be corrected. Its main argument is that differences between judge LLMs can be estimated from ratings alone. The paper also argues that preferences shared by all the judges cannot be decoupled from the true item quality without external evidence.
The authors introduce a Bayesian measurement model to examine these effects, and propose a lightweight network for faster inference. They run tests on TREC and SummEval, showing that while targeted adjustment generally preserves human-label agreement better than full adjustment, they do not outperform simple panel averaging.
Critical strengths
- The paper makes a timely and important distinction between observable correlations and actual biases. It provides useful evidence through many experiments why this distinction is important.
- The authors run extensive tests. They also openly acknowledge negative results which I think is very important for transparent scientific progress.
Critical weaknesses
-
Limited novelty and practical improvement The central identifiability problem has already been studied in psychometric research [1, 2], and related measurement models [3] and IRT-based approaches to LLM judge evaluation [4] already exist. More importantly, targeted adjustment does not outperform simple panel averaging.
-
Uncertainty about the judge-specific effects The proposed correction depends on reliably estimating how individual judges respond to features. However, the paper shows that accounting for judge discrimination substantially changes the estimated lexical-overlap effects. This is concerning because lexical overlap explains much of the reported difference between adjustment methods.
[1] Looking at DIF From a New Perspective: A Structure-Based Approach Acknowledging Inherent Indefinability (Applied Psychological Measurement, 2019).
[2] DIF Statistical Inference Without Knowing Anchoring Items (Psychometrika, 2023).
[3] Additive Multilevel Item Structure Models with Random Residuals: Item Modeling for Explanation and Item Generation (Psychometrika, 2014).
[4] Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory (ICML, 2026).
Questions for authors
- What practical benefit does the proposed audit provide beyond panel averaging or simple judge-leniency correction? Is there a concrete decision that requires the estimated judge-specific feature effects?
- Given the instability of lexical-overlap effects in Appendix Q, what evidence supports treating these estimates as reliable judge-specific responses rather than differences in discrimination?
- Does Proposition 1 require stronger conditions on the rating graph? Connectivity and full-rank item features alone seem insufficient to identify judge-specific slopes when some judges rate very few overlapping items.
3 comments
Thank you for following up. You are right that our answers needed to appear in the manuscript, and we have now uploaded the revised PDF. Each earlier point is there: the distinction from Choi et al. (§2), the qualified overlap class and discrimination limit (Table 1, §3.1, §6.1, Appendix Q), and the article-held-out SummEval result replacing the in-sample row (§5, Table 2, Appendix O). The analyses below are new (Appendices O and T).
GRM relation. Choi et al. fit a logistic GRM with a discrimination and thresholds per prompt variant and a latent quality per sample. It has no item features, and they leave position and verbosity effects to future DIF analysis. We fit that GRM to our panels, with judges in place of prompt variants, then added shared features (GRM-X) and judge-specific feature terms (GRM-DIF, our audit with per-judge thresholds).
(1) Panel-wide associations. Without labels, the GRM ranks items as averaging does (Spearman with human labels 0.689 vs. 0.690 on TREC; 0.598 vs. 0.597 on SummEval), so its alignment diagnostics raise no flag. Its , however, contains every panel-wide feature association, and the likelihood, hence every GRM diagnostic, is identical however is split into quality and shared response. The split changes alignment: removing the shared associations lowers it to 0.602 on TREC and 0.524 on SummEval, and removing only shared lexical overlap lowers SummEval to 0.354. The audit flags this association, shows that ratings cannot settle it (Proposition 1), and measures what is at stake; on SummEval the human labels show that overlap largely tracks quality. A GRM user who corrected for overlap would get no warning.
(2) Judge-specific departures. On SummEval the GRM marks llama-3.2-3b as the least discriminating judge ( vs. 2.2 to 2.9). GRM-DIF adds that its ratings rise with summary and source length relative to the panel, and this predicts its withheld ratings on held-out articles ( nats per rating, 95% CI ). A GRM can only report this as low discrimination.
(3) Where it adds little. For predicting withheld ratings on held-out TREC queries, judge-specific terms barely improve on the GRM ( nats per rating, CI ; SummEval overall , ). Once each judge has its own thresholds, the common-scale overlap heterogeneity is largely absorbed. This qualifies our earlier statement that the slopes predict ratings: they do relative to a common-scale model, not beyond a GRM. It reinforces Table 1's "lexical sensitivity unresolved", and §6.1 now says so.
SummEval interpretation. We read the result as support for averaging, and the decomposition (Appendix O) shows why each correction falls short. The gap between targeted adjustment and averaging is not caused by correction: leniency-only correction, which removes no feature response, falls equally short (0.572). With five ratings on a 0 to 3 scale, the panel mean takes only 10 distinct values over 180 summaries (47 tie at 2.0). Model-based scores break those ties according to which judge gave which rating, and within ties that ordering does not track human scores (, CI ). Keeping averaging's order and breaking its ties by the targeted score gives 0.577, close to 0.573, so the gap lies within the ties. Full adjustment's further loss comes from one shared association: removing only the shared summary-source overlap slope, the largest, gives 0.534, matching full adjustment (0.536), and the loss is between articles (article-level falls from 0.41 to 0.15). Ratings cannot tell whether this association is quality or bias; the labels indicate quality, so it should be retained. This is the workflow in §7: identify what a correction would remove, anchor it externally, and keep averaging unless held-out labels support another rule.
SummEval stability. Six articles (36 summaries) form one fold, but every summary is scored once, so the correlations pool all 180 summaries from 30 articles. Single folds are unstable (targeted minus full ranges from to across folds), so we do not interpret them alone. We repeated the five-fold split for nine more random article partitions and ran leave-one-article-out validation (75 more fits, same sampler settings, no divergences). Targeted adjustment falls below averaging in all ten partitions (mean ; an article bootstrap averaged over partitions, which includes partition variability, gives ) and under leave-one-article-out (, ). Targeted exceeds full adjustment in all ten partitions (), but that difference remains unresolved across the 30 articles (). The shortfall from averaging is therefore stable; the targeted-versus-full ordering is consistent in direction but limited by the number of articles, as the paper states.
I thank the authors for their response and the revised manuscript. In particular, I appreciate you fixing the rating-graph condition and for explicitly crediting previous literature.
That said, these clarifications also reinforce my main concern regarding novelty and practical utility. The underlying measurement model and identifiability limits are still closely tied to exiting DIF and IRT formulations. Moreover, the updated SummEval validation confirms that the targeted adjustment still doesn't outperform simple panel averaging. I am not convinced that the practical benefit beyond panel averaging or standard leniency correction is strong enough, and thus will be retaining my score.
Thank you for the helpful suggestions.
Novelty and practical benefit. We agree, and the paper credits the works you cite: Section 3.1 frames the result as the familiar DIF identification problem (Doebler, 2019), Related Work cites anchor-free DIF inference (Chen et al., 2023), Section 3.3 identifies the item model as AMIS-R (Cho et al., 2014), and the revision sharpens our distinction from Choi et al. (2026). To your question: no scoring decision in our experiments needs the judge-specific feature estimates. The panels we use for human-label alignment are fully crossed, so centered deviations cancel across each item's judges, and leniency-only and targeted scores agree within 0.001 (TREC 0.6805 vs. 0.6809; SummEval 0.5718 vs. 0.5726). The estimates would affect scores only when different judges rate different items. Their value is diagnostic: they show whether judge disagreements vary with features, whether the design identifies them, and whether they survive allowing discrimination. That guides where to add co-ratings, which interventions to run, and when to withhold a correction; averaging remains our default scorer.
Reliability of the overlap effects. We agree they are unresolved, and the revision explains why: with judge discrimination, the shared-slope transformation shifts each judge's centered slope by (aj−aˉ)t, so judge contrasts inherit the shared-slope ambiguity. On your concern that overlap drives the gap between adjustment methods: that gap comes from the shared overlap association, not the judge-specific estimates. Removing only shared components matches full adjustment, and holding everything else at full adjustment, retaining only the shared overlap association raises the in-sample correlation from 0.673 to 0.685. The instability you identify therefore does not affect the scoring comparison.
Rating-graph conditions. You are right. We replaced connectivity and global feature rank with full rank of the co-rated contrast design for (1,xi), which Appendix I proves necessary and sufficient. The TREC panel satisfies it (481 fully crossed items; contrast rank 54/54), as do the calibration and SummEval panels.