- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how the choice LLM judges can change the reliability of comparative evaluations. The authors show two types of errors:
- Errors that result in a difference in the size of the performance gap
- Errors that can change the ranking of the LLMs
The authors found that different LLMs often produce the first type of error, even when evaluating the same models on the same data. To solve this problem, the authors propose using a small number of human-evaluated "gold" reference models. These "anchors" can then be used to calibrate the LLM judge scores. The paper then shows a theoretical analysis on when these anchors are useful, and demonstrate that human evaluations on a few anchors are label efficient. Finally, the authors investigate errors that remain after this calibration and argue that they are inherent model/judge errors.
Critical strengths
- The paper presents strong empirical evidence of judge-dependent evaluation bias. The authors discuss an important practical problem -> different LLM judges can report performance gaps (how much better/worse each model is from each other), even when evaluating the same models on the same data.
- The experiments are extensive and span three evaluation settings and include several robustness checks. Altogether, they provide convincing evidence that the composition of the judge panel affects the reliability of reported effect sizes.
Critical weaknesses
W1: There is insufficient justification of the proposed calibration method.
- The theoretical results rely on restrictive assumptions, particularly that the selected anchor systems have no system-specific residual bias. In practice, it is very difficult for human evaluation to guarantee this.
- The theoretical guarantee concerns squared error with a known calibration factor, whereas the experiments use estimated factors and primarily evaluate median absolute error. This mismatch weakens the theoretical claim, making it hard to justify the proposed method or its claimed practical benefits.
W2: There is incomplete experimental validation of the claimed label efficiency.
- The central claim that a small number of human labels concentrated on anchor systems can outperform substantially larger uniformly allocated budgets is not sufficiently established against well-matched alternatives.
- In particular, the experiments lack a strong global calibration baseline that also exploits shared structure across systems, while aspects of the existing baselines introduce potentially avoidable estimation errors.
- These limitations make it difficult to determine whether the reported improvements arise from the proposed allocation strategy or weaknesses in the comparison methods.
Nits on clarity and presentation
- The manuscript is very difficult to follow despite addressing a relatively intuitive and practically important problem. The writing is highly compressed, introduces substantial mathematical notation before clearly motivating the underlying research question, and frequently combines multiple claims, technical qualifications, and experimental findings within individual paragraphs.
- The repeated use of rhetorical formulations (e.g. "buy the anchor") further obscures the precise meaning and scope of the results.
- More importantly, the distinction between what is theoretically established, what is empirically observed, and what is proposed as a practical recommendation is not always sufficiently clear. This is particularly problematic for the anchor-based calibration claims, where important assumptions and limitations are dispersed across the main text and appendices.
I think the manuscript would benefit from restructuring around a smaller number of clearly articulated research questions, with the motivating problem, main finding, supporting evidence, and limitations presented in a more legible manner.
Questions for authors
- Proposition 1 assumes that the selected anchor systems have zero residual bias, but the proposed anchor-selection strategies do not appear to guarantee this condition. How do the authors justify using these anchors to estimate the global calibration factor, and how sensitive are the reported improvements to violations of this assumption?
- Proposition 2 establishes when rescaling improves expected squared error using the true calibration factor, whereas the experiments primarily evaluate median absolute error using an estimated factor. What evidence supports extending the theoretical conclusions to the practical setting, particularly the claim that the proposed decision rule introduces negligible or no regret?
- The reported label-efficiency gains are central to the paper's contribution, but the uniform affine baseline may be disadvantaged by regression attenuation, and the PPI++ tuning may not match the finite-population evaluation setup. Would the claimed advantages of anchor-based allocation persist against a global calibration baseline that regresses human scores on panel scores and a PPI++ implementation tuned for the experimental sampling design?
3 comments
Dear Reviewer H2ob,
Thank you again for your review; your two technical suggestions changed the paper. (1) Proposition 3 now covers rescaling by any estimated slope, exactly and in squared error, and Table 3 reports how often four anchors land in the safe interval. (2) With the global-calibration and design-tuned PPI++ baselines you proposed, we now attribute the label saving to pooling the scale rather than to concentrating labels, and report it for the four-anchor estimator: about 5x on Arena 2024 and 8x on Arena 2025, smaller on MATH and none on MT-Bench. The paper is also restructured around three research questions, with a claims table separating theorem, measurement and recommendation.
We would welcome any remaining questions before the discussion closes. If your concerns are resolved, we would appreciate it if you would consider updating your score.
Thank you to the authors for the thorough response. I went through the revised manuscript, and I appreciated the authors adding Prop. 3 and including additional experiments. These updates address my earlier concerns regarding theoretical alignment.
That said, I do not think the paper is currently fit for ICLR publication as it is currently. Although I appreciate the restructuring around the three research questions, I still found the main paper hard to read. Even after reading the paper several times, I still found myself going back and forth between sections and the appendix to understand the paper's contributions and main points. For example, in the revised manuscript line 89, without context, it's very difficult to understand what's the type of measurement you did and why the decision rule is significant alongside the budget cost; the abstract and introduction skips over various technical points that are understood in context through the rest of the paper.
Although I'm happy that the technical side of this paper is much more solid after the revision, I also don't think this paper is ready for publication in its current state due to the remaining issues in presentation. I will be keeping my score, but I do encourage the authors to keep polishing their writing and developing this research direction.
Response to Reviewer H2ob
Thank you for the constructive review; your two technical points led to the two most substantive changes in the revision.
W1 / Q1 (anchors with nonzero residual). Proposition 1(ii) now states the exact identity beta_uv = beta + (c_u - c_v)/(theta_u - theta_v). With Proposition 3 (below) the question becomes whether this lands in the safe interval, which we measure directly (App. B.2): using every pair of systems as anchors with their true gold, 63-100% of the pairs the panel separates land in the interval on every panel where rescaling helps (92% commercial, 95% frontier on Arena), except the deep small-open-leaning panel (43%), whose compressed scale leaves a narrow interval. The median over four anchors reduces, but does not remove, the selection bias of the widest pair. The budget experiment also reports pre-specified pairs selected from panel scores without observing gold; they save less (2.2-3.4x on Arena), and a pair drawn at random is worse than no gold, so anchors must be well separated on the panel.
W1 / Q2 (known factor and squared error vs estimated factor and median error). New Proposition 3 covers any factor exactly: rescaling by b > 0 gives squared gap error kappa[(beta/b - 1)^2 s_theta^2 + s_c^2/b^2] over all pairs (exact over the M systems, not an expectation, since the residual is orthogonal to gold by construction), so rescaling helps iff b lies strictly between 1 and b* = (beta^2 + rho)/(1 - (beta - 1)^2 - rho), and for every b > 1 when that denominator is not positive. The interval depends on the unknown beta and rho, so the two-anchor rule "rescale iff the estimate is at least 1" remains a heuristic: on MT-Bench's commercial panel the interval is (1, 1.63), and an estimate above 1.63 would hurt. Its practical content is how often an estimate lands inside: with four anchors and 64 labels, 78-100% of draws on 11 of the 12 panels where rescaling helps (58% on MT-Bench's mixed panel, whose slope is near one; Table 3). All budget results are now reported in squared error, the theorem's currency.
W2 / Q3 (global calibration baseline; PPI++ tuned for the design). We ran both, in a design-matched protocol (labels without replacement, finite-pool target): (i) a global calibration that regresses the labelled systems' gold on their panel scores (unattenuated, since the noise is in the response), (ii) an errors-in-variables-corrected forward fit, (iii) PPI++ with the finite-population-optimal coefficient and no shrinkage, and (iv) GREG. Every estimator choice uses only the labels in the budget. Your concern was partly right, and we revised the paper: on Arena the global calibration on uniformly allocated labels performs like four anchors, so the saving comes from pooling the two scale parameters across systems, not from concentrating the labels. Against the best per-system rectifier, four anchors need about 5x fewer labels on Arena 2024 (four anchors at n = 64 vs PPI++ at n = 256: squared error 5.4 vs 6.2 x 10^-3, difference CI [-0.0012, -0.0003]), 8x on Arena 2025, 1.5-2x on three of four MATH rosters, and about the same on MT-Bench. Without shrinkage a system with one to three labels keeps a nearly raw mean, so the design-tuned rectifiers are poor at small n; every saving is against the best of the three per-system estimators at each budget, including our original shrinkage version. At n = 512 (60% of the pool) PPI++ wins on effect size, and per-system rectifiers are needed for the ranking. Anchors keep one practical advantage: gold is needed on two to four systems and is reused across rosters.
Clarity. Restructured as described in the general response; theorem, measurement and recommendation are separated in the new claims table, and we replaced the rhetorical phrasing in section titles and body text with plain statements. The title is unchanged during the discussion period; §1 now states that its "sample" means per-system corrections.