acceptodds
Under review as a conference paper at ICLR 2027

Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges

Abstract

Multi-judge evaluation increasingly needs calibrated probabilities, not only verdicts. With a small labeled set, a natural practice is to curate: rank judges by individual accuracy and keep the best few. We argue that the labels are better spent to calibrate the panel: model the judges jointly and choose the panel by held-out proper-scoring risk, which may still select a subset, but never by marginal accuracy. We explain why curation appears to work. A conditionally independent aggregator double counts redundant judges, and in a two-block model we prove that this misspecification alone can make an accuracy-curated panel beat the full panel, while the correctly specified full panel beats both. On four labeled pairwise benchmarks with 38–144 judges, removing only the modeled dependence shifts the comparison against the full panel on every benchmark and split, and two pre-registered tests, a controlled simulation and three held-out panels, agree with this mechanism. Under conditionally independent aggregation the five most accurate judges beat the full panel by 0.09–0.13 NLL on both reward-model benchmarks; dependence-aware aggregation shrinks the gap to at most 0.015. Selecting panels by calibrated risk stays within 0.017 NLL of the best accuracy-ranked rule on every benchmark, without choosing the panel size by hand; the aggregator matters more than the panel rule. JudgeBench remains a counterexample: we recommend never pruning by accuracy alone, not keeping every judge.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM evaluation, multi-judge aggregation, panel selection, post-hoc calibration, proper scoring rules, judge dependence

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ksiKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper challenges the common practice of selecting LLM judge panels by individual accuracy ("curation"). The authors argue that curation's apparent success is an artifact of aggregator misspecification: conditionally independent (CI) models like Dawid-Skene double-count correlated judges, making a small curated subset look artificially better than the full panel. The paper proves this mechanism analytically in a two-block Gaussian model, isolates it empirically via a matched covariance intervention on four benchmarks with 38–144 judges, and proposes risk-based panel selection under dependence-aware aggregation as a practical alternative. CI aggregators favor Top-5 by 0.09–0.13 NLL; dependence-aware aggregators shrink this gap to ≤0.015 or reverse it. Risk-based selection stays within 0.017 NLL of the best accuracy-ranked rule without requiring manual panel size tuning.

Critical strengths
  • S1: It is really appreciated that this work studies a practical problem, extremely relevant in the industrial setting.
  • S2: It is appreciated that you tried to understand why in JudgeBench the results are not following the pattern and giving an explanation.
  • S3: The Experimental Setup is clearly described with all the necessary details.
Critical weaknesses
  • W1: Code is available. But running it is impossible. There is no requirements.txt file and there are lots of files. Navigating all of them to understand how code works is practically impossible.
  • W2: I think that the Related Work Section should be expanded a bit. This is a well-known topic in the literature and lots of works try to address this problem. Half of a page maybe is not enough.
  • W3: It is not clear to me why you are not comparing your results to FCPS, which is doing something very similar and is the most related work.
  • W4: With ϵ=10−3\epsilon = 10^{-3}, δ=0.05\delta = 0.05, M=70M = 70 candidates, and selection slices of 35–299 items, the bound is 1.6–4.7 nats. This means it is two to three orders of magnitude larger than any NLL difference reported (0.001–0.022).
Questions for authors
  • M1: All theory and experiments assume zt∈0,1z_t \in {0, 1}.
  • M2: Proposition 3 requires knowing which judges are "redundant" and which are "diverse." No procedure is given for estimating this from votes.

Also the text in the Weaknesses Section of the form can be considered as additional questions.

Reviewer_XZxrOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to spend a small labeled set when combining many LLM judges or reward models into a calibrated probability that response A beats response B. The common practice it targets is accuracy curation: rank judges by development accuracy, keep the top few, aggregate. The paper’s central claim is that curation only looks good because conditionally independent aggregators double-count correlated judges; once dependence is modeled, the advantage of the top-5 largely disappears. Supporting evidence: (i) a two-block Gaussian model in which the CI score on the full panel is beaten by the accuracy-curated block while the full-covariance score beats both; (ii) a matched intervention that zeroes the off-diagonal of a fitted LDA covariance and shows the All-vs-Top-5 contrast moves against the full panel on all four benchmarks, all 21 splits, and three held-out panels; (iii) a pre-registered simulation reproducing the sign flip near the predicted ρ; (iv) a panel-selection experiment in which choosing panels by held-out calibrated NLL lands within 0.017 NLL of the best accuracy-ranked rule.

Critical strengths
  1. The matched covariance intervention is a clean design. Holding means, priors, imputation and shrinkage fixed and toggling only the off-diagonal is the right way to isolate dependence from the other differences between aggregator families. The sign is consistent across 7 panels and every split, including three panels on which the intervention had not been run.

  2. Methodological care is well above the norm for this subfield. The label boundary is explicit, the two encoding pitfalls in Appendix C.1 are a real service (I suspect the first one afflicts a nontrivial fraction of published panel comparisons), the simulation and held-out analyses were pre-registered, and the authors are candid that their own earlier harness had both defects.

  3. Honest reporting of mixed results. JudgeBench is presented as a counterexample rather than explained away; the small gains of risk-based selection are stated plainly; the finding that half of the LLMBar gain comes from below-chance judges is surfaced rather than buried.

Critical weaknesses
  1. The title and framing overclaim relative to the evidence. What the experiments support is "use a dependence-aware aggregator." They do not support "don't curate": under ridge, the accuracy path (curation with kk tuned) is best or tied on 3 of 4 benchmarks (Table 2), and risk-based selection is strictly worse on JudgeBench and RewardBench 2 despite having the accuracy prefixes in its library. "Label-efficient" is likewise contradicted by Table 8: the CI + Top-5 comparison resolves at 100–250 labels, while the dependence-aware comparison resolves nowhere except ridge on RewardBench with all 1,492 development labels, and the proposed pipeline needs a third selection slice on top. The authors concede both points in Sec. 4.5 and the limitations, but the title and abstract do not.

  2. Table 1 confounds more than dependence, and key baselines are missing. The CI and dependence-aware families also differ in missing-vote handling (skipped vs. mean-imputed) and generative vs. discriminative fitting. On JudgeBench the added API judges have coverage of only 164–309 of 350 items, and Eq. (2) assumes availability is uninformative about ztz_t, which is doubtful for unparseable outputs on hard items. The intervention in Sec. 4.3 isolates dependence, but the Table 1 narrative attributes the whole 0.09–0.13 gap to it. A one-coin variant with mean imputation would close this gap. Separately, the paper never compares against semi-supervised Dawid–Skene (EM on the unlabeled evaluation votes) or an unsupervised dependence-aware aggregator such as CARE (Zhao et al. 2026), so the implicit claim that labels are what dependence-awareness requires is unexamined.

  3. The LLMBar gain depends on exploiting stably-wrong judges, and robustness to shift is untested. Half of the full-panel gain comes from negative weights on 25 below-chance judges; on the Natural subset All is worse (+0.093 NLL). Stable wrongness on adversarial items is a property of how LLMBar was constructed, and if the deployment distribution has fewer adversarial items, negative weights can flip from help to harm. A cross-subset test (fit on adversarial, evaluate on Natural, and the reverse) would show whether the dependence-aware full panel is robust or is fitting benchmark artifacts.

Questions for authors
  1. Risk-based selection (λ=0\lambda = 0) contains the accuracy prefixes in its library yet loses to the accuracy path on JudgeBench and RewardBench 2 (Table 2). Is this entirely selection-slice noise? Can you show evaluation NLL as a function of selection-slice size?

  2. Does missingness (JtJ_t) correlate with the label or with item difficulty on JudgeBench, where added judges have coverage of 164–309 of 350? Does a one-coin variant with mean imputation (or an LDA that skips missing judges) change the Table 1 gap?

  3. Cross-subset LLMBar: fit on the adversarial subsets and evaluate on Natural (and the reverse). Does the full panel's advantage survive, given that All is worse on Natural items (+0.093 NLL)?

  4. For RewardBench 2, what fraction of the 144 models share a base model, and does family deduplication (as done for PPE, where it shrank the effect to near zero) change ΔA−T\Delta_{A-T} or the covariance interaction?

Reviewer_JCLzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper talks about how we can efficiently use labels to calibrate a panel of LLM Judges rather than curating them by using the panel's accuracy scores i.e rather than picking the top-k judges based on some accuracy metric (curating), it's better to model the joint-judges first and choose based on some risk metric. The reason is because these judges are noisy, has biases (which are shared) and naive aggregators assume the scores are conditionally independent which can amplify the noise (double counting) resulting in inefficient and overconfident judge panels. For the actual calibration of the judge panels, they do a wide range of experiments on different choices with types of aggregators used (conditionally independent vs dependency-aware) when combining the judge panel scores, metrics used to define the final risk, selection criterias of panels (top-k, accuracy only etc;) and on various datasets. Some of the main findings are on how modeling LLM judge-dependence improves the overall panel scores, how choosing the judge panel by scores/risks is on-par/better than accuracy alone and how aggregator choices matter more than the panel rule.

Critical strengths

The motivation is clear - LLM-Judge panels are widely used and understanding how to improve their reliability methodically helps a lot in improving efficiency and understanding where to spend resources. And addressing over-confident LLM-judge panels is also a tricky problem because it's technically hard to understand whether it's double-counting or genuine judge agreement.

Critical weaknesses
  • Limited Contributions (L85 - 92): As the authors have mentioned, the main contribution is empirical diagnostic analysis of scaling to multiple judges on binary-only labels. Then, I feel it's a narrow contribution which focuses on specific diagnosis; for eg: conditionally independent (CI) aggregators double-count redundant judges, which makes accuracy curation look better than it is (contribution 1) and the rest are bringing in different {aggregators, panels, metrics} for LLM-Judge panel to understand the settings better. For the actual motivation, similar phenomenon is already present in weak supervision + already reported in LLM-Judges -- which the authors have already cited. So, I would like to hear authors opinions on the main contributions the paper is adding for practitioners and readers. How should a reader go about using these diagnostic results for their own aggregators?
  • (Experiments on baselines) While the paper is not proposing any method, having relevant baselines (like average, majority voting etc;) are important to understand the diagnosis better -- the authors can pick the weak-supervised papers that they have cited as relevant curated baselines.
  • (On clip and then calibrate; L149-150): Judge panel aggregator values are first clipped and then used for calibration - How will it impact the conditional independence and other runs as clipping can change the panel in unexpected ways.
Questions for authors

On presentation and sentence formations: I feel the paper can be presented a lot better. I strongly recommend the authors to consider polishing the writing and work on the presentation as the current form is very hard to understand at various sections. Some example sentences verbatim attached below from introduction section:

  • Every estimator we use is standard, and we do not claim that the full panel is uniformly best. FCPS (Zhu et al., 2026) already selects panels and aggregator families by validation on a seven-judge pool; our contribution is an empirical diagnosis for large, dependent LLM-judge and reward-model panels: we identify, prove and test the mechanism that makes accuracy curation look right.
  • In the sense of our title, a panel chosen by calibrated risk is calibration, not curation, even when it is a subset, and “label-efficient” means that one small labeled set is spent on fitting and selecting the panel’s predictive model rather than on ranking judges; how many labels that needs depends on panel size and redundancy (Sections 4.4 and 4.6).

While these are grammatically valid sentences, I somehow felt very hard to understand this format - it felt more like a blog post/informal reading - "in the sense of our title" etc;

Some suggestions on formatting

  • Have tables for protocols, metrics used in each experiment setting somewhere so it's easy to refer.
  • Have metrics defined at Appendix for more details.
  • The paper might be benefitted from notations table in Appendix.
  • Because there are multiple settings, having a table for aggregators, metrics, panel selection criterias etc; will be super helpful for readers for easier flow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.