acceptodds
Withdrawn from ICLR 2027

From Ordering Evaluators to Selecting One: What a Validation Statistic Is Worth Under a Label Budget

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Automatic evaluators are validated by pooled correlation with human labels over clustered items and used to choose inside a cluster. We measure what a validation statistic is worth when it selects one evaluator from a budget of labelled clusters, by the utility of the selected evaluator on clusters no rule used, with an estimand, a selection protocol and intervals whose coverage is measured inside worlds with a known or simulated truth. A statistic grouped on the decision's cluster orders evaluators better than pooling, and that better ordering is an unreliable guide to how much better an evaluator it selects, most visibly on machine translation, where large ordering advantages came with gains whose intervals included zero on every pool frozen in advance. Across public benchmarks for speech, summarization, story generation, machine translation and reward modelling, some of them read under protocols frozen in advance, grouped statistics usually order evaluator pairs better than pooled correlation, while the utility gain of the evaluator they select is uneven and its intervals often include zero. With few labelled clusters the sign of the gain differs by panel and rests on single tasks. A pre-registered, time-stamped manipulation of the candidates in a labelled cluster contradicted the account we had offered for that difference, which we withdraw, and on its replication set showed the grouped statistic's gain falling as candidates are removed. We recommend reporting, beside any validation statistic, the utility of the evaluator it selects with its interval, its leave-one-out and how the labels were spent.

Primary area:
datasets and benchmarks
Keywords:
meta-evaluation, evaluation methodology, evaluator selection, automatic metrics, LLM judges, reward models, best-of-n, machine translation metrics, clustered data, label budget, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RiP8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores how best to evaluate automatic evaluation methods, for example LLM-as-judge, machine translation metrics, or reward models. Such automatic evaluations are often themselves evaluated based on metrics like correlation to a ground truth (e.g., human label).

The starting observation is that automatic evaluation methods are often judged based on correlation pooled across groups. For example, the evaluator might assign a score to several generations for a single prompt. This pooling ignores the fact that we might care about picking the best candidate within a group rather than how well a model predicts an absolute score (across groups).

The paper then describes related work, some experimental protocols, and resulting numbers from comparing different ways of evaluating automatic evaluations over datasets like PPE, WMT25 MQM, and WMT25 ESA.

Critical strengths

The research topic is interesting: what value do we leave on the table by not fully considering the dependence structure and different use-cases of automatic evaluations?

The authors have correctly identified an important area for further research. What is the proper way to statistically account for within-cluster dependence? What automatic evaluations should we use when picking the best candidate from a cluster, as opposed to assessing the overall performance of a model? The topic has many important, under-studied directions.

Critical weaknesses

The paper does not present a coherent argument for any specific conclusion about a well-identified research question. Instead, the paper describes the topic, followed by related work, and then some experimental protocols and their results. This description is disjointed and fragmented, making it difficult to follow. Most of the text describes details about what was done thus it reads like a log rather than a research paper. Consequently, it is unclear what lessons, insights, facts, or discoveries one should take away from the paper, except perhaps that it is better to consider dependence structure when analyzing hierarchical / mixed-effects data, which is a well-researched area in statistics.

To be more specific, here are the contributions listed at the end of the introduction:

  • "Ordering is not choosing": This bullet comes closest to presenting a conclusion; however, the bullet's text is ambiguous: "... a statistic grouped on the decision's clusters orders evaluators better than pooling in the mean over a set's configurations, and the evaluator it selects is not reliably better than pooling's...". In other words, using the dependence structure orders evaluators better, but does not select a better evaluator.
  • "What a set can show, and where the advantage falls": The paragraph for this bullet just repeats numbers from the results but gives no research questions or conclusions.
  • "An estimand and its calibration": This bullet describes their evaluation metric.
  • "Choosing a judge from eight labelled prompts": Here, they say that on one dataset (PPE), their tie-calibrated accuracy score gave them a confidence interval which contains zero. The research question isn't stated, and the result did not have sufficient power to show something one way or the other.

The paper needs significant revision. The experimental protocols and outcomes are potentially well-documented (there are 102 pages of appendices); however, they have not been interpreted or assembled into a coherent argument.

Questions for authors

My main questions are about the authors' interpretation of the results, and the argument they are trying to make:

  • What core research question(s) did you want to ask with this work?
  • What were your hypotheses about these questions?
  • In your empirical results, which methods do you consider your baselines (the pooled Pearson correlation)? - Which methods do you view as the most important variants you wished to test (e.g., the utility from line 207)?
Reviewer_BYLrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that validating automatic evaluators (e.g., LLM judges) by how well a validation statistic orders them is a different question from how good the evaluator it selects turns out to be, and reports across several benchmarks and pre-registered protocols that the former does not reliably predict the latter.

Critical strengths

This work may provide some practical insight for borader works relying on automatic evaluators.

Critical weaknesses
  1. The novelty and contribution is limited. What this work provides is basically measurement, which might be practical but incremental.

  2. [Critical issue] The manuscript is almost unreadable and I believe this is not ready to be reviewed. The 9-page main text is organized as somewhat a chronological log of experiment rather than a structured report of findings and almost every sentence packs several numbers, whose meaning and interpretation is defined in the appendix. The substantive content is deferred to the appendix with more than 100 pages that is extremely poorly organized and presented. I could not read almost anything clearly from the manuscript about what it wants to claim, what the evidence for it etc. Presenting a paper like this makes it almost impossible to properly review a paper, leaving reviewers to carry the burden of evaluating all of it. This is not an acceptable research practice.

Questions for authors
  1. How was AI used for this work? Please disclose honsetly.
Reviewer_b9uGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether validation statistics that rank automatic evaluators more accurately also help select better evaluators when only a limited number of human-labeled examples are available. The authors compare pooled correlation with within-cluster and grouped validation statistics, evaluating the selected evaluator on held-out clusters using downstream measures such as best-of-n recovery and pairwise accuracy. Experiments cover speech, summarization, story generation, machine translation, reward models, and LLM judges. The main finding is that grouped statistics generally improve evaluator ranking, but these improvements do not consistently translate into better evaluator selection. The paper also examines how label budgets, candidate counts, and evaluator panels affect these outcomes. Several hypotheses are evaluated using protocols fixed before the corresponding experiments, and the authors explicitly report predictions that were not supported.

Critical strengths
  1. The paper makes an important distinction between correctly ranking evaluators and selecting an evaluator that performs well in practice. This is relevant for LLM judges and reward models, where validation metrics are frequently used to choose systems. Measuring the downstream utility of the selected evaluator, rather than relying only on correlation or ranking agreement, provides a more useful perspective on evaluator validation.
  2. The experimental analysis is extensive, covering multiple benchmarks, evaluator types, and label budgets. I appreciate the use of held-out clusters, uncertainty intervals, and protocols fixed before evaluating new datasets. The authors also acknowledge unsuccessful predictions and withdraw an explanation that was contradicted experimentally. This makes the empirical findings more credible, even when the results are not consistently positive.
Critical weaknesses
  1. While the paper demonstrates that grouped statistics improve evaluator ordering, the practical benefits for evaluator selection are considerably less convincing. On the exploratory configurations, the improvement in best-of-n recovery is only 1.6 percentage points, with a confidence interval of [-0.2, 3.5]. Similarly, most of the frozen evaluation sets do not establish a statistically clear improvement. This limits the strength of any practical recommendation to prefer grouped validation statistics over pooled correlation.
  2. The experimental conclusions are sensitive to the evaluator panel, candidate pool, and benchmark configuration. For example, the grouped statistic performs substantially worse on PRM800K with eight labeled problems, and some WMT25 results change when all available systems are included. The paper acknowledges these differences, but it remains difficult to determine when a practitioner should expect one validation statistic to outperform another. A clearer characterization of these conditions would strengthen the contribution.
Questions for authors
  1. Given that grouped statistics consistently improve evaluator ordering but often fail to produce significant downstream utility gains, under what conditions would the authors recommend using grouped statistics instead of pooled correlation for evaluator selection? Can the existing results identify observable characteristics of a benchmark or evaluator panel that would help make this decision?
  2. How much of the variability in the reported improvements can be attributed to the composition of the evaluator panel rather than the validation statistic itself? In particular, could the authors provide additional analysis explaining why the results differ between PRM800K, RewardBench 2, and the machine translation benchmarks?
  3. The paper finds that averaging the three highest-ranked LLM judges can outperform selecting a single judge, but this increases inference costs threefold. How does this approach compare with selecting the best individual judge when both methods are evaluated under the same total annotation and inference budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.