acceptodds
Under review as a conference paper at ICLR 2027

When Are Smaller Benchmarks Better?

Abstract

Automated evaluations often rank systems differently from humans. We show that agreement improves substantially when the benchmark is reduced to a small subset of its existing items, without adding new data. Sieve selects as small as 20 items on a Pareto front over two objectives: per-item accuracy match with the source benchmark, and correlation with the human system ranking. We evaluate on held-out systems and held-out items, so the gain cannot be circular. The 20-item subsets agree with humans better than the full 80- to 5,000-item benchmarks they came from, closing about half the remaining gap to perfect agreement on average and all of it for the weakest judges. Across the 36-cell task-judge grid the mean gain is +0.37 at one-sided Wilcoxon p=2.7×10⁻⁶. At the extreme, LLaMA-3B on MT-Bench swings from an inverted ranking, ρ_T=−0.49, to perfect agreement, ρ_Q*=+1.00. What a judge gains, it turns out, depends on how much room it had to improve. We call this the Headroom Law: gain scales with headroom at slope α=0.86, 95% CI [0.75, 0.93], with per-judge slopes in [0.65, 0.94] and seven independent selectors recovering α ∈ [0.87, 1.01]. So we can tell whether Sieve will help on a new task-judge pair before running the search at all: leave-one-out r=0.92 on fit cells, and live prediction r=0.935 on 66 cells we had never seen.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-Judge, benchmark compression, evaluation, human agreement, Headroom Law

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KB9ZOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the question of finding a subset of a benchmark such that automated evaluations on it correlate better with human judgments than when conducted on the full benchmark itself. The proposed method, called Sieve, selects a subset of predefined cardinality of items to evaluate, evaluates how much a fraction of rating systems tend to agree both in accuracy (M) and in ranking (C) with human judgments on these items, considers a Pareto front over (M,C) and takes the subset that reaches the closest point to maximum performances on that front. That subset is then evaluated by other systems, whose rankings appear to be better correlated with human ratings. The paper proposes the Headroom Law to explain that, generally speaking, the worse the systems, the more beneficial to them is the proposed subset selection mechanism. Experiments conducted on various benchmarks (systems, tasks) appear to support the benefits of the approach and the suitability of that law.

Critical strengths

The problem of interest is particularly well-articulated, with a clear research question that should draw many readers' curiosity.

The framing in terms of held-out and visible systems and ratings, while echoing classical machine learning practice, is sound and well-conducted in the proposed approach.

The willingness to select the subset that corresponds to an ideal tradeoff between performance and ranking faithfulness on a Pareto front is also sound and interesting. The results seem to validate the overall approach.

Critical weaknesses

The paper is hard to follow and has a strange structure. The introduction provides already many technicalities, that are repeated or redefined later, or not. For instance, HAG is defined line 72 as a very important metric while still being in the introduction, then HA is defined twice, in line 158 and 215, while HAG is then re-stated with different notations in Eq. (8), and later HAG-L and HAG-Q are undefined. Similarly, Eq. (4) mentions Acc, which is undefined until then, and then Eq. (5) defines Acc but it is unclear if it is the same, as well as Acc defined in Eq. (9). The tables seem to use notations undefined in the text.

The related work section is utterly minimal, and, likewise, the comparisons with other methods of the literature is missing (from table 3, I suppose?). Or are there methods of Table 3 that are actually the known competitors? No description is given about it, making it difficult to understand if the paper only compares with variants of itself, or with an established community.

The actual contributions are a bit difficult to grasp. They mention a non-circular protocol (multiple times in the paper) without providing a clear explanation (much more than what is already said) of what that is nor illustrating its existence with results from the literature. Then, contributions 2, 3 and 4 do not underline what makes Sieve novel, interesting, and useful, while I am sure there are impactful contributions that could be extracted from the paper. Unless it is simply using a classical "train/validation/test split" on items, systems, and ratings?

In the same vein, it is teased that the Headroom law can tell if Sieve will help on a new task-judge pair before running the search at all (see abstract), which seems great, but the contributions/the paper do not properly state how that is possible. It might be trivial from what is already presented, but the difficulty I had to follow the paper might have prevented me from getting it.

Questions for authors

Does the proposed method actually answer the question in the title? In particular, at which stage of a benchmark design should the proposed work be used to find the right baseline to support automated evaluation?

How exactly does Sieve compare to other compression methods from the literature (if Table 3 does not provide that comparison already, which is not clear)?

How practical is the proposed solution (Eq. 3), given that it is a brute-force approach (unless I am wrong?) on benchmark items, and how different from a simple "train/validation/test" split is the actual contribution of Sieve?

2 comments

Thank you for the careful reading. Your comments on clarity shaped this revision more than any others, and we hope you find the paper much easier to follow now.

  • Does the paper answer its title? Yes. A smaller test is better when it asks a weak judge better questions, not when it selects items.
  • Is it easier to read? The introduction is rewritten and short, each symbol is defined once, and the contributions are stated in four lines.
  • Is it compared with the literature? Yes, with two published methods (Table 2), and the related work has 27 more references.

W1. Structure and notation. Thank you for listing these. We fixed each case in the revised version. Human agreement (HA) and accuracy (Acc) are each defined once, at their first use. The gain in human agreement (HAG) is introduced in the introduction and formally defined once, in Section 3.3. HAG-L and HAG-Q appeared in a table that has been removed.

W2. Related work and published competitors. We added 27 references to the related work. We also compared with two published item-selection methods, Anchor Points (Vivek et al., 2024) and Arena-Hard (Li et al., 2024), next to random selection (Table 2). None of them beats the full benchmark on held-out systems, that is, the systems not used to select the items: with 20% of the items they reach 0.63 to 0.64 in agreement with the human ranking, against 0.73 for the full benchmark. Our Probe method, in which the judge answers yes/no check questions, was tested in its own experiments (Table 4) and beats the full benchmark by +0.41 when the judge is weak.

W3. What is new, and what is it useful for? A standard train/validation/test split divides the items only. Our protocol also divides the systems, at the same time, because a benchmark is meant to rank systems it has never seen. Dividing the systems is not new in itself: tinyBenchmarks and Anchor Points also test on models that were not used to build the subset. To our knowledge, two things are new.

(1) The target is the human ranking, not the full benchmark. Earlier work asks whether a subset reproduces the scores of the full benchmark. We ask whether it agrees with humans better than the full benchmark does.

(2) A shuffled-label control. We repeat the selection with the human labels shuffled, so that they carry no information. A gain that survives the shuffle is a fit to the systems, not a better test. This control exposed our own error: on the seen systems, that is, the systems used to select the items, the subset gains +0.08 with real labels and +0.43 with shuffled labels.

The contributions of the revised paper are:

Trading items for questions, and when it helps. At a fixed number of judge calls, a small judge answers yes/no check questions on fewer items, instead of rating every item once. This helps when the judge is weak, meaning that its own scores agree poorly with the reference ranking, and the task is scored against an answer key or a model's ratings. It does not help when the judge is strong.

What it is useful for. Many users can only afford a small open model as a judge. For such a weak judge, agreement with the reference ranking rises from 0.43 to 0.67 (Table 5), close to the 0.72 that strong judges reach by rating directly. The strong model is called once per item to write the questions, not once per response. Asking only the first question needs no human labels at all.

W4. Can one tell beforehand whether the method helps? Not from the headroom, which is 1 minus the benchmark's agreement with the human ranking. We withdrew that claim, because shuffled labels give the same slope (0.47 against 0.43). For Probe, a rule fitted only on the seen systems names the better test, Probe or the full benchmark, in 42 of 49 task-judge pairs (Section 5.2).

At which stage should it be used? After a benchmark exists and a judge has been chosen, when new systems are to be ranked. The main text now gives the recipe (paragraph "How to use it", end of Section 5.2): write the questions, check on a few labelled systems whether the judge is weak, choose the test, and confirm the choice on held-out systems.

Is Sieve brute force? Is it practical? No, it is not brute force. The equation you refer to states the objective, and the search does not enumerate subsets. It draws 60 random subsets of 20 items, scores each, and keeps the best. Brute force would have to examine about 4 x 10^15 subsets even on our smallest task, which has 60 candidate items. Sieve takes about 3,000 judge calls per task-judge pair, a few minutes on one GPU. Its limit is different: its gain does not carry over to held-out systems, while Probe does for weak judges.

Thank you again for your time and for the detailed list of problems, which made the paper better. If the revision resolves your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_KB9ZOpenReview ↗

Thank you

I thank the authors for the detailed answers, that are clearly formulated. I also appreciate that errors were transparently reported and corrected. I am happy that my questions and comments helped improve the paper.

I simply regret that some of the major changes were needed in the first place. As an example, I am not sure how fine it is to be able to add 27 references during the rebuttal, I feel that these references should have been there already in the first place if they were important, not only after a careful reviewer mentions it, as it could have gone through the system unnoticed.

Likewise, I am not sure how comfortable I am with the fact that the paper can significantly change its results and claims/contributions within the rebuttal period, as it feels like "a new paper" was produced somehow, and that would almost deserve a full new round of review + rebuttal (which cannot be done within the 8 hours remaining between the authors' submitted rebuttal and the deadline for it).

My concerns are not so much anymore about the paper itself (and anyway I still feel unconfident about my ability to rate it), they are more on what can be tolerated in terms of changes. Hence, I will maintain my rating for the sake of discussing it during the meta-review period.

Reviewer_bG9AOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper's main claim is that a small subset of a benchmark can have better human agreement on model ranking than the full benchmark. The paper proposes Sieve, a method that find such subset. The method find the subset by comparing two metrics on randomly samples subsets, one of the metrics M measures how much the subset reflects the human scores, and the other metrics C measure whether the benchmark reflect the quality of the prompts. The final subset is chosen based on both metrics. The experiments show that the selected subsets can improve agreement with human model rankings across several benchmarks.

Critical strengths

This paper shows an interesting and meaningful results, a compressed benchmark can better reflect human rankings than the full benchmark. I think the method is very useful when the goal is higher human agreement on model ranking.

The design of sieve method is simple and requiring only a small set of human annotations.

The experiments are comprehensive.

Critical weaknesses

The motivation of having C (ranking correlation) as the proxy is unclear, although the ablation shows that having C can lower the reproduction gap, however, it does not improve the overall correlation.

Questions for authors

Can the authors provide the details of the ranking setup? For example, how many models are actually ranked?

1 comment

Thank you for your thoughtful review, and for seeing the value in the idea that a smaller test can be better than the full benchmark. That idea is still the main claim of our paper, and the revision supports it with deeper and richer results. The gain now comes from asking the judge better questions rather than from selecting items, and for weak judges it holds on systems the method has never seen. The general response gives the full picture. Below we answer your two points.

Motivation of C. Thank you for raising this. C, the ranking-correlation objective, is used only in the experiments that measure the judge's accuracy on single items. There it narrows the gap between the subset and the full benchmark, as you noted, and the revision no longer claims that it improves the ranking. Our new method, Probe, does not need C at all. A stronger model writes yes/no check questions once per item, and the judge answers them. When the judge is weak, meaning that its own scores agree poorly with the reference ranking, Probe beats the full benchmark on new systems by +0.41 on the 8 weak task-judge pairs (Section 5.2, Table 4).

How many models are ranked? Between 7 and 30 per benchmark (Section 5.1): SummEval 16, PromptBench 15, HumanEval 28, HELM 30, AlpacaEval 16, UltraFeedback 16, Nectar 7, JudgeLM 13, MMLU-Pro 12. We divide the systems at random, 60% for the selection and 40% held out, which leaves 3 to 12 held-out systems. Each agreement value is a Spearman correlation over those systems, averaged over 10 such divisions. One update: the HumanEval responses were synthetic in the submission. They are now real completions of 28 open code models, labelled by the original unit tests.

Thank you again for your time. We hope the corrected and independently confirmed results address your concern about soundness.

Reviewer_AbQeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether carefully selected smaller benchmarks can produce model rankings that agree more closely with human judgments than the original, larger benchmarks. The authors propose Sieve, which selects a small subset of benchmark items using two objectives: maintaining item-level accuracy and preserving ranking discrimination across a reference judge panel. The method uses a held-out evaluation protocol to reduce the risk of selecting items that simply overfit to the human reference. The authors also introduce the Headroom Law, an empirical relationship suggesting that improvements in human agreement can be predicted from how poorly the original benchmark agrees with humans. Experiments across multiple benchmarks and LLM judges show substantial improvements, particularly for weaker judges. The paper also compares Sieve against existing benchmark compression methods and examines generalization across judges, systems, and random seeds.

Critical strengths
  1. I like that the paper goes beyond simply reducing benchmark size while preserving the original rankings. If the original benchmark does not agree well with human judgments, preserving its rankings is not necessarily useful. The idea of selecting a smaller set of items that actually improves human agreement is interesting and makes the work different from existing benchmark compression methods.

  2. The experimental evaluation is fairly thorough. The authors test across several benchmarks and judges, compare against existing compression methods, and evaluate on held-out systems. I particularly appreciate the effort to avoid circular evaluation by separating subset selection from the final human reference. The results are encouraging, especially for weaker judges, where the improvements are quite substantial.

Critical weaknesses
  1. I am not fully convinced that the Headroom Law is as general as the paper suggests. The relationship is empirically strong, but headroom is defined as 1 minus the original correlation, which also provides an upper bound on the achievable gain. Since both quantities depend on the same baseline correlation, some relationship is already expected. The experiments demonstrate a consistent pattern, but I would be more cautious about presenting it as a general scaling law. Also, many of the evaluation cells share the same underlying tasks and judges, so they are not entirely independent observations.

  2. The results on held-out systems are less consistent than the overall averages suggest. Table 2 contains several cases where improvements on seen systems are substantially reduced or become negative on held-out systems. For example, SummEval with LLaMA-3B goes from +0.61 on seen systems to -0.15 on held-out systems, and PromptBench also shows negative held-out gains for multiple judges. These results raise questions about how reliably a selected 20-item subset will generalize to new models. I would also like to see a clearer analysis of how much the selected subsets depend on the particular systems used during selection.

Questions for authors
  1. Since headroom is mathematically related to the maximum possible improvement, how much of the observed Headroom Law is explained by this constraint alone? Have the authors compared their results against a suitable null model or a baseline that accounts for this dependence?

  2. Table 2 shows some substantial differences between seen-system and held-out-system gains. What characteristics of these tasks or selected subsets explain the negative transfer? Is there a way to identify when Sieve is unlikely to generalize before deploying the subset?

  3. The ranking correlation objective uses three prompts of the same judge as a proxy for differences between systems. How sensitive are the results to this choice? In particular, would the performance change if the reference panel contained independently trained models instead of prompt variations?

  4. The method uses human item-level ratings during selection, while claiming to improve human agreement without directly optimizing the final human ranking. How much human annotation is actually needed to construct a reliable subset for a new benchmark? This seems important for understanding the practical cost of the method.

2 comments

Thank you for your insightful questions. We followed them and revised our paper.

W1, Q1. Is the Headroom Law explained by its own upper bound? Yes. Headroom is 1 minus the benchmark's agreement with the human ranking, and the gain cannot exceed it, so the two are tied together before any method is applied. We ran the null model you asked for: we shuffled the human labels across systems, which removes all signal, and repeated the whole procedure. The slope of the gain against headroom is 0.43 with real labels and 0.47 with shuffled labels (Table 3). A method that cannot work shows the same "law", so we withdrew it. Thank you also for pointing out that our task-judge pairs share tasks and judges and are not independent, which the revision now states.

W2. Held-out systems. Thank you. Your question led us to find a bug ourselves: held-out systems, which take no part in selecting the items, had received the scores of the seen systems, which do. With it fixed, a selected subset agrees with the human ranking less well than the full benchmark on held-out systems (-0.15 on average, positive in 2 of 27 task-judge pairs, Table 1). Your example, SummEval with LLaMA-3B, is now +0.00 on seen and -0.25 on held-out systems. What does work on held-out systems is our new method, Probe (Sections 3.5 and 5.2): instead of selecting items, a small judge answers yes/no check questions about each item. When the judge is weak, meaning that its own scores agree with the reference ranking at a correlation below 0.5 on the seen systems, Probe beats the full benchmark on held-out systems by +0.41 on the 8 weak task-judge pairs, with a gain in every pair, and by +0.23 when weakness is decided separately in each random division of the systems (Table 4).

Q2. Why the negative transfer, and can it be predicted? The subset fits the systems it was chosen on, not the task. It is worst on three tasks in which each system answers only some of the items, so that a 20-item subset holds only a few judge scores per held-out system (-0.30, against -0.08 on the other tasks). We have no rule that predicts when selection transfers, because on average it does not. For Probe, a rule fitted on the seen systems names the better test, Probe or the full benchmark, in 42 of 49 task-judge pairs (Section 5.2 of the revised paper).

Q3. Prompt panel or independent models? The panel is used only in the experiments on single-item accuracy, not in those that rank systems, which the submission did not state correctly. On held-out systems even selection on the human ranking itself does not transfer (-0.15), so we did not test the panel there. What does improve agreement on held-out systems is Probe (+0.41 on the 8 weak task-judge pairs, Table 4).

Q4. How much human annotation? Almost none. Asking only the first check question of each item needs no human labels and gains +0.48 on the 8 weak task-judge pairs. Choosing the best question for each item needs human labels on the seen systems (600 to 4,830 per random division of the systems) and gains +0.52. New systems need no labels in either case.

Reviewer_AbQeOpenReview ↗

Rebuttal Comments

Thank you for the detailed response and for taking my concerns seriously. I appreciate the transparency regarding the implementation errors and the effort to re-evaluate the results and acknowledging both findings I raised in my review. Considering the rebuttal and revisions together, I maintain my rating of Weak Acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.