acceptodds
Under review as a conference paper at ICLR 2027

The Effect Size Is a Free Parameter of the Judge Panel: Buy the Scale, Not the Sample

Abstract

A panel of LLM judges differs from its gold standard in two ways that behave differently. Writing the panel score of system as , the slope is a pure scale error that multiplies every reported effect size and by itself reorders nothing, while the residual can reorder a leaderboard and, by a classical aliasing argument, is beyond the reach of gold-free aggregation. We measure both across three settings and 99,487 judge verdicts from up to 26 judges in 14 model families. The central finding is that is a free parameter of the panel: on identical items under an identical protocol, it runs from under an open-weight roster to under a frontier one on Chatbot Arena, tracking the panel's mean member accuracy, not the task. Judged effect sizes therefore have no unit until the roster is reported. The fix is cheap: two gold-anchored systems with no residual of their own identify , and a closed-form rule says when rescaling helps; two anchors decide its branch, eight buy its full form. Fed an estimated , the rule's mean regret was never positive, and on a miscalibrated panel 32 anchor labels (64 with the debiased four-anchor estimator) beat 512 spent uniformly under prediction-powered inference on effect-size error (CIs exclude a tie). Anchors cannot buy the ranking: the rank-relevant residual clears a conservative matched null on a deepened open-weight-leaning panel (flip risk ) but not on a commercial one at the same depth, where it is of a median gold gap; and it is neither predictable from surface features nor transferable across panels. Code and every judge verdict are released in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
llm-as-a-judge, evaluation, effect size, judge panels, calibration, prediction-powered inference, gold labels

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_H2obOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how the choice LLM judges can change the reliability of comparative evaluations. The authors show two types of errors:

  1. Errors that result in a difference in the size of the performance gap
  2. Errors that can change the ranking of the LLMs

The authors found that different LLMs often produce the first type of error, even when evaluating the same models on the same data. To solve this problem, the authors propose using a small number of human-evaluated "gold" reference models. These "anchors" can then be used to calibrate the LLM judge scores. The paper then shows a theoretical analysis on when these anchors are useful, and demonstrate that human evaluations on a few anchors are label efficient. Finally, the authors investigate errors that remain after this calibration and argue that they are inherent model/judge errors.

Critical strengths
  • The paper presents strong empirical evidence of judge-dependent evaluation bias. The authors discuss an important practical problem -> different LLM judges can report performance gaps (how much better/worse each model is from each other), even when evaluating the same models on the same data.
  • The experiments are extensive and span three evaluation settings and include several robustness checks. Altogether, they provide convincing evidence that the composition of the judge panel affects the reliability of reported effect sizes.
Critical weaknesses

W1: There is insufficient justification of the proposed calibration method.

  • The theoretical results rely on restrictive assumptions, particularly that the selected anchor systems have no system-specific residual bias. In practice, it is very difficult for human evaluation to guarantee this.
  • The theoretical guarantee concerns squared error with a known calibration factor, whereas the experiments use estimated factors and primarily evaluate median absolute error. This mismatch weakens the theoretical claim, making it hard to justify the proposed method or its claimed practical benefits.

W2: There is incomplete experimental validation of the claimed label efficiency.

  • The central claim that a small number of human labels concentrated on anchor systems can outperform substantially larger uniformly allocated budgets is not sufficiently established against well-matched alternatives.
  • In particular, the experiments lack a strong global calibration baseline that also exploits shared structure across systems, while aspects of the existing baselines introduce potentially avoidable estimation errors.
  • These limitations make it difficult to determine whether the reported improvements arise from the proposed allocation strategy or weaknesses in the comparison methods.

Nits on clarity and presentation

  • The manuscript is very difficult to follow despite addressing a relatively intuitive and practically important problem. The writing is highly compressed, introduces substantial mathematical notation before clearly motivating the underlying research question, and frequently combines multiple claims, technical qualifications, and experimental findings within individual paragraphs.
  • The repeated use of rhetorical formulations (e.g. "buy the anchor") further obscures the precise meaning and scope of the results.
  • More importantly, the distinction between what is theoretically established, what is empirically observed, and what is proposed as a practical recommendation is not always sufficiently clear. This is particularly problematic for the anchor-based calibration claims, where important assumptions and limitations are dispersed across the main text and appendices.

I think the manuscript would benefit from restructuring around a smaller number of clearly articulated research questions, with the motivating problem, main finding, supporting evidence, and limitations presented in a more legible manner.

Questions for authors
  1. Proposition 1 assumes that the selected anchor systems have zero residual bias, but the proposed anchor-selection strategies do not appear to guarantee this condition. How do the authors justify using these anchors to estimate the global calibration factor, and how sensitive are the reported improvements to violations of this assumption?
  2. Proposition 2 establishes when rescaling improves expected squared error using the true calibration factor, whereas the experiments primarily evaluate median absolute error using an estimated factor. What evidence supports extending the theoretical conclusions to the practical setting, particularly the claim that the proposed decision rule introduces negligible or no regret?
  3. The reported label-efficiency gains are central to the paper's contribution, but the uniform affine baseline may be disadvantaged by regression attenuation, and the PPI++ tuning may not match the finite-population evaluation setup. Would the claimed advantages of anchor-based allocation persist against a global calibration baseline that regresses human scores on panel scores and a PPI++ implementation tuned for the experimental sampling design?
3 comments
AuthorsOpenReview ↗

Response to Reviewer H2ob

Thank you for the constructive review; your two technical points led to the two most substantive changes in the revision.

W1 / Q1 (anchors with nonzero residual). Proposition 1(ii) now states the exact identity beta_uv = beta + (c_u - c_v)/(theta_u - theta_v). With Proposition 3 (below) the question becomes whether this lands in the safe interval, which we measure directly (App. B.2): using every pair of systems as anchors with their true gold, 63-100% of the pairs the panel separates land in the interval on every panel where rescaling helps (92% commercial, 95% frontier on Arena), except the deep small-open-leaning panel (43%), whose compressed scale leaves a narrow interval. The median over four anchors reduces, but does not remove, the selection bias of the widest pair. The budget experiment also reports pre-specified pairs selected from panel scores without observing gold; they save less (2.2-3.4x on Arena), and a pair drawn at random is worse than no gold, so anchors must be well separated on the panel.

W1 / Q2 (known factor and squared error vs estimated factor and median error). New Proposition 3 covers any factor exactly: rescaling by b > 0 gives squared gap error kappa[(beta/b - 1)^2 s_theta^2 + s_c^2/b^2] over all pairs (exact over the M systems, not an expectation, since the residual is orthogonal to gold by construction), so rescaling helps iff b lies strictly between 1 and b* = (beta^2 + rho)/(1 - (beta - 1)^2 - rho), and for every b > 1 when that denominator is not positive. The interval depends on the unknown beta and rho, so the two-anchor rule "rescale iff the estimate is at least 1" remains a heuristic: on MT-Bench's commercial panel the interval is (1, 1.63), and an estimate above 1.63 would hurt. Its practical content is how often an estimate lands inside: with four anchors and 64 labels, 78-100% of draws on 11 of the 12 panels where rescaling helps (58% on MT-Bench's mixed panel, whose slope is near one; Table 3). All budget results are now reported in squared error, the theorem's currency.

W2 / Q3 (global calibration baseline; PPI++ tuned for the design). We ran both, in a design-matched protocol (labels without replacement, finite-pool target): (i) a global calibration that regresses the labelled systems' gold on their panel scores (unattenuated, since the noise is in the response), (ii) an errors-in-variables-corrected forward fit, (iii) PPI++ with the finite-population-optimal coefficient and no shrinkage, and (iv) GREG. Every estimator choice uses only the labels in the budget. Your concern was partly right, and we revised the paper: on Arena the global calibration on uniformly allocated labels performs like four anchors, so the saving comes from pooling the two scale parameters across systems, not from concentrating the labels. Against the best per-system rectifier, four anchors need about 5x fewer labels on Arena 2024 (four anchors at n = 64 vs PPI++ at n = 256: squared error 5.4 vs 6.2 x 10^-3, difference CI [-0.0012, -0.0003]), 8x on Arena 2025, 1.5-2x on three of four MATH rosters, and about the same on MT-Bench. Without shrinkage a system with one to three labels keeps a nearly raw mean, so the design-tuned rectifiers are poor at small n; every saving is against the best of the three per-system estimators at each budget, including our original shrinkage version. At n = 512 (60% of the pool) PPI++ wins on effect size, and per-system rectifiers are needed for the ranking. Anchors keep one practical advantage: gold is needed on two to four systems and is reused across rosters.

Clarity. Restructured as described in the general response; theorem, measurement and recommendation are separated in the new claims table, and we replaced the rhetorical phrasing in section titles and body text with plain statements. The title is unchanged during the discussion period; §1 now states that its "sample" means per-system corrections.

Dear Reviewer H2ob,

Thank you again for your review; your two technical suggestions changed the paper. (1) Proposition 3 now covers rescaling by any estimated slope, exactly and in squared error, and Table 3 reports how often four anchors land in the safe interval. (2) With the global-calibration and design-tuned PPI++ baselines you proposed, we now attribute the label saving to pooling the scale rather than to concentrating labels, and report it for the four-anchor estimator: about 5x on Arena 2024 and 8x on Arena 2025, smaller on MATH and none on MT-Bench. The paper is also restructured around three research questions, with a claims table separating theorem, measurement and recommendation.

We would welcome any remaining questions before the discussion closes. If your concerns are resolved, we would appreciate it if you would consider updating your score.

Thank you to the authors for the thorough response. I went through the revised manuscript, and I appreciated the authors adding Prop. 3 and including additional experiments. These updates address my earlier concerns regarding theoretical alignment.

That said, I do not think the paper is currently fit for ICLR publication as it is currently. Although I appreciate the restructuring around the three research questions, I still found the main paper hard to read. Even after reading the paper several times, I still found myself going back and forth between sections and the appendix to understand the paper's contributions and main points. For example, in the revised manuscript line 89, without context, it's very difficult to understand what's the type of measurement you did and why the decision rule is significant alongside the budget cost; the abstract and introduction skips over various technical points that are understood in context through the rest of the paper.

Although I'm happy that the technical side of this paper is much more solid after the revision, I also don't think this paper is ready for publication in its current state due to the remaining issues in presentation. I will be keeping my score, but I do encourage the authors to keep polishing their writing and developing this research direction.

Reviewer_5REJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

his paper studies how panels of LLM judges deviate from gold-standard human evaluations, and argues that this deviation must be separated into two components with very different consequences: a slope term β that purely rescales all reported effect sizes (without changing any ranking) and a residual that is the only part capable of reordering the leaderboard. Through 99,487 judge verdicts across MT-Bench, MATH-500, and Chatbot Arena, the authors show that β is essentially a free parameter of the judge panel — with identical items and protocol, it ranges from 0.45 (open-weight panel, compressing gaps by 55%) to 1.74 (frontier panel, inflating gaps by 74%) on Arena, tracking the panel's mean member accuracy rather than the task. The practical upshot is that judged effect sizes have no unit unless the panel roster is reported, but the fix is cheap: two gold-anchored systems identify the scale, and a closed-form rule (rescaling always helps when β ≥ 1, but almost never when β < 1) tells practitioners when to apply it — 32 anchor labels outperform 512 uniformly spent labels against PPI++ on effect-size error. What no budget can buy is the ranking: the rank-relevant residual is statistically resolvable only with deep sampling, is unpredictable from surface features, and does not transfer across panels, so two anchors restore the unit but no gold-free procedure restores the order.

Critical strengths
  1. A clean and consequential conceptual split. Decomposing judge error into a scale component (β) and a rank-relevant residual (c̃) is simple but genuinely clarifying: it shows that much of what the literature treats as "judge bias" is a harmless unit error, while the truly dangerous part (the residual) is precisely what no gold-free method can reach. This reframing has immediate consequences for how benchmark results should be reported and consumed.
  2. Large-scale, well-controlled measurement. The study collects 99,487 verdicts from up to 26 judges across 14 model families in three heterogeneous settings (human preference, verifiable grading, varying leaderboard density). Crucially, the composition sweeps hold items and protocol fixed and vary only the roster, which isolates the panel as the causal factor behind β — supported by dose-response curves over 2,520–3,000 random sub-panels rather than just hand-picked tiers.
  3. Actionable theory with real content. Proposition 2 is a non-trivial, quantitative decision rule: rescaling always helps when β ≥ 1, but for β < 1 it helps only when σ²_c̃/σ²_θ < β²(1−β)/(1+β), a threshold that never exceeds 0.09 — so weak panels should be left alone. This is validated semi-synthetically, survives being fed estimated rather than true β (mean regret never positive), and corrects a mistake practitioners likely make routinely.
Critical weaknesses
  1. The central residual claims are established only on a panel practitioners would not field. The statistically significant flip-risk result (32%, p = 0.02) rests on the deepened 11-judge panel that is eight-elevenths small open-weight models at β = 0.625. On the commercial panel (β = 1.47) — the roster closest to deployed practice — the residual does not clear the matched null even at 368 battles/system (p = 0.077/0.12); it is "bounded rather than established." The abstract's headline that the residual is "18% of a median gold gap" therefore comes from the one comparison that cannot be statistically separated from noise, and the commercial deep panel further spans only 3 families, limiting the resolution of its family-level null.
  2. The systems are two years older than the judges. Using the public 2024 Arena snapshot means 2026-era judges (GPT-5.4, Claude-Opus-4.8) evaluate models they were trained after, opening a channel for prior-preference contamination from training data that the protocol does not control. Combined with a single pairwise prompt, one grading prompt, and English-only data, the quantitative claims (β values, residual magnitudes, flip risk) should be read as vintage-specific. Since the authors themselves show that denser leaderboards expose more pairs to the same residual, flip risk on today's leaderboards is plausibly worse than reported, not better.
Questions for authors
  1. Judge–system vintage mismatch. The 2024 Arena systems predate all judges by roughly two years, so judges may hold training-data priors about these specific models. Did you run any control for this — e.g., judging on paraphrased/anonymized outputs, or verifying that kin vs. non-kin effects do not simply track prior popularity of the system names? If not, why is prior-driven residual ruled out as an alternative explanation for the kinship contrast (which would not require provider kinship at all)?
  2. Widest-pair anchor bias. The widest-pair estimator shows a +0.22 upward bias in β̂ — enough to flip the rescaling recommendation on the mixed panel. Since this estimator drives the headline "32 beats 512" result, could you report the budget experiment using only the four-anchor median estimator you recommend in §8? If the honest comparison is 64 vs. 512, the abstract's framing seems to advertise the very estimator the paper advises against.
  3. Why family as the only exchangeable unit? Your family-level nulls widen intervals by 25–32% relative to item-only resampling, but resampling only 3 frontier families (or 3 commercial families) makes percentile intervals nearly discrete. Given that Appendix L resorts to exact permutation over families for tier separation, would a judge-level null with a within-family correlation correction (e.g., cluster-robust variance) be a better default? Does any conclusion change under it?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5REJ

Thank you for the positive assessment and the precise questions.

W1 (residual established only on a non-deployable panel; the 18%). Agreed that this was the gap. We deepened a deployable panel: eight judges from four families (GPT-5.4, GPT-5.1, GPT-4.1-mini, GPT-4o-mini, gpt-oss-120b, Llama-3.3-70B, Gemma-3-27B, Mistral-Small-3.2-24B; no small model) on 1,600 + 2,400 + 4,000 disjoint 2024 battles, composition fixed, 720 battles per system. The residual separates under the original, full-covariance and judge-clustered nulls, on flip risk (p < 0.001 / 0.006 / 0.002) and on RMS (p = 0.001 / 0.04 / 0.01). Since five of the eight judges are OpenAI models, we repeated the tests with one judge per family (each OpenAI judge in turn with the Meta, Google and Mistral ones): it separates in all five. In gold units it is 21% of a median gold gap, consistent with the 18% of the commercial panel, and we report it as an estimate rather than a headline. Its excess in pairs actually misordered is not statistically resolved (p = 0.11). On Arena 2025, at about 140-160 battles per system, it separates on the large open-weight roster but not, on flip risk, on the commercial or pooled deployable one.

W2 / Q1 (vintage, priors, kinship vs popularity).

  • New setting: 2,400 Arena battles from April-July 2025 among 30 systems (Claude-Opus-4, Gemini-2.5-Pro, o3, ...), within a year of the judges and 48% non-English. RQ1 replicates (small open-weight beta = 0.39, commercial 1.85; random sub-panels [0.77, 1.47], corr +0.90 with accuracy), and English and non-English halves give overlapping slopes for every roster.
  • Kinship on Arena 2025, as an intervention (add one provider's deployable judges to a deployable panel without any): adding the four OpenAI judges moves five of the six OpenAI systems up relative to the other 24, by +0.0156 (permutation p = 0.003, surviving Bonferroni over the five providers). Anthropic: 6 of 8 systems, +0.0035 (p = 0.035, which does not survive the same correction). DeepSeek, Google, Meta: not significant. So the kinship contrast replicates for OpenAI and is small; it is not the bulk of the residual.
  • Popularity (2024): kin systems are also the most battled (r = 0.70). Among non-kin systems the residual shift is unrelated to popularity (r = -0.02), and with log popularity as a covariate the kinship coefficient stays positive (+0.014), but the popularity-matched permutation is inconclusive because the two are collinear (p = 0.25). We therefore do not claim to rule out popularity.
  • Identity cues: judges see only conversation text, and responses on our items rarely name a model or provider (0.3% for kin systems, 0.1% for others); the share does not predict the shift. We did not run a paraphrase or anonymisation control, so training-time familiarity or stylistic identity cues are not ruled out; the paper now says so.
  • Beyond kinship: computed on disjoint halves of the battles and read against each panel's split-half reliability, residuals of deployable panels with no family in common reach a reliability-normalised similarity of 0.95 (2024) and 0.89 (2025), from raw correlations of 0.24 and 0.31 with reliabilities 0.25 and 0.35. These ratios are imprecise, but they point to a component that capable judges share, or equivalently a shared disagreement with the crowd gold.

Q2 (widest-pair bias). Done: the design-matched experiment now headlines the four-anchor median estimator you asked about, with each pooled estimator reported separately (App. B.1). With four anchors and 64 labels, per-system rectifiers need about 295 labels on Arena 2024's commercial panel to match its squared error (4.6x); the widest pair gives 6.1x and a pre-specified pair 2.2x. The widest-pair "32 vs 512" comparison is withdrawn (App. C).

Q3 (cluster-robust, judge-level inference). Done (App. B.5). Since beta = mean of judge-level slopes, we use CR1 with t_{G-1} and a Webb wild-cluster bootstrap. Because commercial and frontier share families, a tier contrast pools each family's contribution across both tiers (CR1 over the union of families, one wild weight per family). Every small-open-weight vs deployable contrast survives in every setting (p < 0.001); frontier vs commercial survives on Arena (p = 0.005), is borderline on MATH (p = 0.08; wild 0.004) and not significant on MT-Bench, matching Table 2's overlapping intervals. For the residual we report three nulls throughout (independent, full covariance across systems, judge-level family-clustered); modelling the covariance raises the small-open-leaning deep panel's p-values from 0.04/0.02 to 0.11/0.06 (RMS/flip), and we revised that claim accordingly.

Dear Reviewer 5REJ,

Thank you for your positive review and precise questions. As you asked, the budget experiment now headlines the four-anchor median estimator, the residual is tested on a deployable panel at depth, and judge-level cluster-robust inference is reported throughout. We would be glad to answer any further questions before the discussion period ends.

Reviewer_DY9sOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

LLM-based evaluation has increasingly shifted toward using a panel of judges to make collective judgments. This paper studies how scores from an LLM judge panel relate to human gold scores. The authors fit the panel score of each system as a linear function of its gold score plus a residual. The slope only stretches or shrinks the gaps between systems and never changes their order, while the residual can change the order and cannot be recovered without gold labels. Using MT-Bench, MATH-500, and Chatbot Arena, they find that the slope mostly depends on which judges are in the panel. To fix the scale, they label two "anchor" systems with gold scores and give a rule for when rescaling helps. They also show that the residual is hard to detect, cannot be predicted from simple features such as length, and does not carry over from one panel to another.

Critical strengths
  1. I enjoyed the problem statement and its motivation. Automated evaluation increasingly relies on a group of models to make collective decisions, yet a reported effect size from a judge panel means little unless the composition of the panel is also reported.
  2. The paper's analysis and core idea are simple but insightful: splitting the error into a scale part and a ranking part is a useful way to think about judge bias.
  3. I think the problem is interesting and important to the field, and the empirical studies are diverse enough to support the paper's claims.
Critical weaknesses
  1. This paper is very hard to follow. Numbers and new technical terms are mixed together and introduced without a clear roadmap. The results are scattered across the main text and many appendices, and important details are often only in the appendix.
  2. I'm not convinced the anchor fix is very useful in practice. It needs gold labels, and they have to be collected again every time the judge panel changes. It also only fixes the size of the gaps, so the ranking stays the same, while PPI++ can actually improve the ranking. In addition, the gold-budget experiment is only run on Chatbot Arena. It is unclear how well this fix generalizes to new tasks, and I didn't find results that support this.
  3. Many of the judges are small models that have been shown to be less capable, for example 1B–7B open-weight models such as OLMo-2-1B and StableLM-2-1.6B. The wide slope range in the abstract depends heavily on these judges. The panels people actually use, commercial and frontier, all have slopes above 1, and over random sub-panels the range is much narrower.
  4. The residual is only significant on panels made mostly of small open-weight judges, and on the commercial panel it doesn't clear the null even with more battles. A simple quadratic term also explains about 20% of it on some Arena panels, so part of the residual may just come from the linear fit being too simple. I think the claim that "no budget buys the ranking" needs stronger evidence than this.
Questions for authors
  1. Would the results hold with open-weight judges people actually use (for example 70B models), and with reasoning judges?
  2. Where does fixing the effect size matter if the ranking stays the same? Most leaderboards only report rankings or Elo, so a concrete example would help.
3 comments
AuthorsOpenReview ↗

Response to Reviewer DY9s (1/2)

Thank you for the detailed review and for the kind words about the problem and the core split. We took your concerns as the main guide for the revision. In short, what the revision adds: on identical items and protocol, deployable rosters are not calibrated either (over sampled deployable-only sub-panels, the central 90% means preference gaps inflated 19-86% and grading gaps compressed 29-48%); an exact safe interval for rescaling by any estimated slope (Prop. 3); and a design-matched comparison showing when pooling the scale beats per-system correction and when it does not.

W1 (hard to follow). Agreed. The revision is organised around three research questions; each section opens with its question and answer, a claims table (new Table 1) marks each claim as theorem, measurement or recommendation with its evidence and scope, and the densest number-laden paragraphs were replaced by tables.

W2 (practicality of the anchor fix).

  • "Gold must be collected again every time the panel changes." Fortunately it does not need to be. The gold value of an anchored system is a property of that system on the benchmark's items, not of the panel, so when the roster changes only the judging is re-run. New App. B.3: for each of the three original benchmarks, four anchors are chosen once without gold, and their gold rescales all four rosters of that benchmark; the estimate lands in Proposition 3's safe interval on 11 of the 12 panels (the exception is the panel our rule says not to rescale) and, over those 11, cuts squared error by a median 3.6x (using the anchors' full gold). Where systems with human ratings on the same items already exist, anchors cost nothing extra.
  • "It only fixes gap sizes, while PPI++ improves ranking." Agreed that scale and ranking need different budgets, and the ranking needs the larger one. On Arena 2024 our original PPI++ stays within a point of the raw panel's pair accuracy at small budgets and improves on it from 256 of 846 battles; the design-tuned rectifiers rank far worse at small budgets (0.62 vs 0.82 at n = 32) and improve on it only at 512. On MATH, where the panel order is already correct, every rectifier lowers it at every budget. For effect sizes at small budgets, four anchors need about 5x fewer labels than the best per-system rectifier on Arena 2024. The two are complementary, and the paper now says which to use for which purpose (§6).
  • "Only on Chatbot Arena." The budget experiment now runs on all four settings. The four-anchor saving is about 1x on MT-Bench (M=6), 1.5-2x on MATH (M=8), 5x on Arena 2024 (M=22) and 8x on Arena 2025 (M=30), largest on the larger boards. MATH also exercises the beta < 1 branch of our rule on real data: rescaling cuts squared error by 11-39x on three of four rosters there.

W3 (small judges drive the range). You are right that the widest range, 0.45-1.74, relies on including weak judges. We added the judges you name, on the same items and protocol: Llama-3.3-70B, gpt-oss-120b, Gemma-3-27B, Mistral-Small-3.2-24B, Kimi-K2.6, GPT-5-mini and DeepSeek-reasoner. Over random sub-panels drawn only from deployable judges, the central 90% of beta is [1.19, 1.29] on MT-Bench, [1.51, 1.71] on Arena 2024, [1.59, 1.86] on Arena 2025 and [0.52, 0.71] on MATH. So between deployable rosters the variation is much smaller, and we now say so. What remains is a large common miscalibration: a 4-point Arena gap is 2.3-2.7 gold points across deployable rosters, and the direction flips with the task.

AuthorsOpenReview ↗

Response to Reviewer DY9s (2/2)

W4 (residual evidence; curvature).

  • We narrowed the claims (abstract, §7, App. C) and added the evidence that was missing on deployable judges. On a panel of eight deployable judges from four families (OpenAI, Meta, Google, Mistral) at 720 battles per system, the residual separates from its matched null under all three nulls (flip risk p < 0.001 / 0.006 / 0.002; RMS p = 0.001 / 0.04 / 0.01), also when each family contributes a single judge, and it is 21% of a median gold gap. Its share of pairs actually misordered (8.7%) exceeds the matched null by an amount that is not statistically resolved (p = 0.11), and at the shallow headline densities the residual does not separate; the paper says both.
  • "No budget buys the ranking" is now stated as an identifiability result (Prop. 1(iii)): the scores alone do not identify a system's residual, and gold on a few anchor systems does not in general identify the residuals of the others. The empirical part is its magnitude.
  • Curvature (App. B.6), now on that deployable panel: a quadratic absorbs about 1% of the residual variance, and the residual still clears its full-covariance null under quadratic (RMS p = 0.034, flip p = 0.009) and logit (0.046, 0.010) calibration, with the null simulated through the same fitter. The 20% you cite is from shallow panels where we never claimed significance. A fully monotone (isotonic) fit removes about half of it; the remainder still separates on RMS (p = 0.031) but not on flip risk (p = 0.08). The rate at which a panel misorders systems is invariant to any monotone calibration.

Q1 (70B and reasoning judges). See W3 and the new rows of Table 2.

Q2 (where does fixing the effect size matter?). New App. B.4 gives three concrete cases, all with the ranking held fixed. (i) Elo, which Arena publishes: the human Elo gap between GPT-4-1106-preview and Koala-13B is 280; the small open-weight roster reports 108, the commercial roster 541, the frontier roster 735. (ii) Thresholded decisions (routing, release gates, reward thresholds: "act if the gap exceeds delta"): at delta = 0.10 the frontier roster gets 37% of the 231 pair decisions wrong, and rescaling with four anchored systems (using their full gold) cuts this to 22%. (iii) Cross-paper comparison: a 4-point judged win-rate gain is 2.3 gold points under a frontier roster and 8.9 under a small open-weight one.

Dear Reviewer DY9s,

Thank you again for your review. As the discussion period closes soon, we wanted to check whether our responses address your main concerns. In brief: (1) the scale error is not driven by small judges: random sub-panels drawn only from deployable judges, including the 70B-class open-weight and reasoning judges you asked about, still inflate preference gaps and compress grading gaps; (2) anchor gold does not need to be re-collected when the panel changes, and the budget experiment now covers all four settings; (3) the residual now separates from noise on a deployable panel at depth, also with one judge per family, and survives quadratic and logit calibration. The paper is also reorganised around three research questions, with a claims table (Table 1).

If any concern remains, we would be glad to clarify before the deadline. If the revision addresses your concerns, we would be grateful if you would consider updating your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers for careful reviews. DY9s and 5REJ find the split of judge error into a scale part and a rank part useful ("simple but insightful"; "clean and consequential"), and H2ob finds "strong empirical evidence of judge-dependent evaluation bias". The reviews converge on four requests: a clearer paper, stronger baselines and theory for the anchor fix, judges people actually deploy, and stronger evidence (or narrower claims) about the residual. All new or changed text in the revised PDF is in blue. The revision adds 107,202 verdicts (206,689 in total) from eight new judges and one new setting.

What changed, in brief. The central RQ1 finding is unchanged and now holds for deployable judges. Secondary claims were narrowed after the stronger baselines and nulls you asked for, and Appendix C lists each: the label saving comes from pooling the scale across systems, not from where the labels go; the "32 vs 512" comparison is withdrawn; the residual's significance is stated with its depth and roster; and, once gold noise is not shared, the residual looks shared across capable judges rather than panel-specific.

1. Presentation (DY9s, H2ob). The paper is reorganised around three research questions, each answered in its own section: RQ1, is the slope a property of the panel (§5); RQ2, can a small gold budget fix the scale, and when (§6); RQ3, what remains (§7). A new Table 1 lists every claim with its type (theorem / measurement / recommendation), evidence and scope. Rhetorical section titles were replaced by plain ones.

2. Theory for estimated slopes (H2ob). New Proposition 3: for any slope b used to rescale, including one estimated from data, the squared effect-size error over all pairs of systems is an exact quadratic form, and rescaling helps iff b lies strictly between 1 and an explicit b* (for every b > 1 when b*'s denominator is not positive). The two-anchor rule "rescale iff the estimate is at least 1" remains a heuristic; Table 3 reports how often four anchors with 64 labels land in the interval.

3. Design-matched budget experiment, stronger baselines (H2ob, DY9s, 5REJ). Labels without replacement, a finite-population target, PPI++ tuned for this design, the survey-regression (GREG) estimator, and a global calibration on uniformly allocated labels (H2ob's suggestion), on all four settings. Every estimator choice uses only the labels in the budget. To match four anchored systems at 64 labels, the best per-system rectifier needs about 5x the labels on Arena 2024 (M=22), 8x on Arena 2025 (M=30), 1.5-2x on three of four MATH rosters (M=8) and no more on MT-Bench (M=6). On Arena the global calibration does about as well as anchors, so pooling is what saves labels. Affine calibration cannot change rankings; per-system rectification improves Arena rankings at large budgets (at small budgets it depends on the estimator), and there it also wins on effect size.

4. Judges people deploy (DY9s, 5REJ). New tiers on the same items and protocol: large open-weight (Llama-3.3-70B, gpt-oss-120b, Gemma-3-27B, Mistral-Small-3.2-24B, Kimi-K2.6) and reasoning (GPT-5-mini, DeepSeek-reasoner). On sub-panels drawn only from deployable judges, the central 90% of beta lies in [1.19, 1.29] (MT-Bench), [1.51, 1.71] (Arena 2024), [1.59, 1.86] (Arena 2025) and [0.52, 0.71] (MATH). The between-roster range is narrower than with small judges, but the miscalibration persists: over these sub-panels the central 90% corresponds to 19-86% inflation on the preference settings and 29-48% compression on MATH.

5. A 2025 setting (5REJ). 2,400 Arena battles (April-July 2025, 30 systems, 48% non-English). RQ1 replicates: beta = 0.39 for small open-weight and 1.85 for commercial judges; English and non-English halves give overlapping slopes.

6. Residual (DY9s, 5REJ). (i) That judge scores alone, and in general a few gold anchors, do not identify the residual is a theorem (Prop. 1(iii)). (ii) A deployable panel of eight judges from four families, deepened to 720 battles per system, separates from its matched null under independent, full-covariance and judge-clustered nulls, also with one judge per family; the residual is about a fifth of a median gold gap. (iii) It survives quadratic and logit calibration on that panel. (iv) Disjoint-half analyses suggest a residual component shared across the tested deployable panels, although the reliability-normalised comparisons are imprecise. (v) Provider kinship replicates on Arena 2025 for OpenAI only and is a small part of it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.