- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper addresses the question of finding a subset of a benchmark such that automated evaluations on it correlate better with human judgments than when conducted on the full benchmark itself. The proposed method, called Sieve, selects a subset of predefined cardinality of items to evaluate, evaluates how much a fraction of rating systems tend to agree both in accuracy (M) and in ranking (C) with human judgments on these items, considers a Pareto front over (M,C) and takes the subset that reaches the closest point to maximum performances on that front. That subset is then evaluated by other systems, whose rankings appear to be better correlated with human ratings. The paper proposes the Headroom Law to explain that, generally speaking, the worse the systems, the more beneficial to them is the proposed subset selection mechanism. Experiments conducted on various benchmarks (systems, tasks) appear to support the benefits of the approach and the suitability of that law.
Critical strengths
The problem of interest is particularly well-articulated, with a clear research question that should draw many readers' curiosity.
The framing in terms of held-out and visible systems and ratings, while echoing classical machine learning practice, is sound and well-conducted in the proposed approach.
The willingness to select the subset that corresponds to an ideal tradeoff between performance and ranking faithfulness on a Pareto front is also sound and interesting. The results seem to validate the overall approach.
Critical weaknesses
The paper is hard to follow and has a strange structure. The introduction provides already many technicalities, that are repeated or redefined later, or not. For instance, HAG is defined line 72 as a very important metric while still being in the introduction, then HA is defined twice, in line 158 and 215, while HAG is then re-stated with different notations in Eq. (8), and later HAG-L and HAG-Q are undefined. Similarly, Eq. (4) mentions Acc, which is undefined until then, and then Eq. (5) defines Acc but it is unclear if it is the same, as well as Acc defined in Eq. (9). The tables seem to use notations undefined in the text.
The related work section is utterly minimal, and, likewise, the comparisons with other methods of the literature is missing (from table 3, I suppose?). Or are there methods of Table 3 that are actually the known competitors? No description is given about it, making it difficult to understand if the paper only compares with variants of itself, or with an established community.
The actual contributions are a bit difficult to grasp. They mention a non-circular protocol (multiple times in the paper) without providing a clear explanation (much more than what is already said) of what that is nor illustrating its existence with results from the literature. Then, contributions 2, 3 and 4 do not underline what makes Sieve novel, interesting, and useful, while I am sure there are impactful contributions that could be extracted from the paper. Unless it is simply using a classical "train/validation/test split" on items, systems, and ratings?
In the same vein, it is teased that the Headroom law can tell if Sieve will help on a new task-judge pair before running the search at all (see abstract), which seems great, but the contributions/the paper do not properly state how that is possible. It might be trivial from what is already presented, but the difficulty I had to follow the paper might have prevented me from getting it.
Questions for authors
Does the proposed method actually answer the question in the title? In particular, at which stage of a benchmark design should the proposed work be used to find the right baseline to support automated evaluation?
How exactly does Sieve compare to other compression methods from the literature (if Table 3 does not provide that comparison already, which is not clear)?
How practical is the proposed solution (Eq. 3), given that it is a brute-force approach (unless I am wrong?) on benchmark items, and how different from a simple "train/validation/test" split is the actual contribution of Sieve?
2 comments
Thank you
I thank the authors for the detailed answers, that are clearly formulated. I also appreciate that errors were transparently reported and corrected. I am happy that my questions and comments helped improve the paper.
I simply regret that some of the major changes were needed in the first place. As an example, I am not sure how fine it is to be able to add 27 references during the rebuttal, I feel that these references should have been there already in the first place if they were important, not only after a careful reviewer mentions it, as it could have gone through the system unnoticed.
Likewise, I am not sure how comfortable I am with the fact that the paper can significantly change its results and claims/contributions within the rebuttal period, as it feels like "a new paper" was produced somehow, and that would almost deserve a full new round of review + rebuttal (which cannot be done within the 8 hours remaining between the authors' submitted rebuttal and the deadline for it).
My concerns are not so much anymore about the paper itself (and anyway I still feel unconfident about my ability to rate it), they are more on what can be tolerated in terms of changes. Hence, I will maintain my rating for the sake of discussing it during the meta-review period.
Thank you for the careful reading. Your comments on clarity shaped this revision more than any others, and we hope you find the paper much easier to follow now.
W1. Structure and notation. Thank you for listing these. We fixed each case in the revised version. Human agreement (HA) and accuracy (Acc) are each defined once, at their first use. The gain in human agreement (HAG) is introduced in the introduction and formally defined once, in Section 3.3. HAG-L and HAG-Q appeared in a table that has been removed.
W2. Related work and published competitors. We added 27 references to the related work. We also compared with two published item-selection methods, Anchor Points (Vivek et al., 2024) and Arena-Hard (Li et al., 2024), next to random selection (Table 2). None of them beats the full benchmark on held-out systems, that is, the systems not used to select the items: with 20% of the items they reach 0.63 to 0.64 in agreement with the human ranking, against 0.73 for the full benchmark. Our Probe method, in which the judge answers yes/no check questions, was tested in its own experiments (Table 4) and beats the full benchmark by +0.41 when the judge is weak.
W3. What is new, and what is it useful for? A standard train/validation/test split divides the items only. Our protocol also divides the systems, at the same time, because a benchmark is meant to rank systems it has never seen. Dividing the systems is not new in itself: tinyBenchmarks and Anchor Points also test on models that were not used to build the subset. To our knowledge, two things are new.
(1) The target is the human ranking, not the full benchmark. Earlier work asks whether a subset reproduces the scores of the full benchmark. We ask whether it agrees with humans better than the full benchmark does.
(2) A shuffled-label control. We repeat the selection with the human labels shuffled, so that they carry no information. A gain that survives the shuffle is a fit to the systems, not a better test. This control exposed our own error: on the seen systems, that is, the systems used to select the items, the subset gains +0.08 with real labels and +0.43 with shuffled labels.
The contributions of the revised paper are:
Trading items for questions, and when it helps. At a fixed number of judge calls, a small judge answers yes/no check questions on fewer items, instead of rating every item once. This helps when the judge is weak, meaning that its own scores agree poorly with the reference ranking, and the task is scored against an answer key or a model's ratings. It does not help when the judge is strong.
What it is useful for. Many users can only afford a small open model as a judge. For such a weak judge, agreement with the reference ranking rises from 0.43 to 0.67 (Table 5), close to the 0.72 that strong judges reach by rating directly. The strong model is called once per item to write the questions, not once per response. Asking only the first question needs no human labels at all.
W4. Can one tell beforehand whether the method helps? Not from the headroom, which is 1 minus the benchmark's agreement with the human ranking. We withdrew that claim, because shuffled labels give the same slope (0.47 against 0.43). For Probe, a rule fitted only on the seen systems names the better test, Probe or the full benchmark, in 42 of 49 task-judge pairs (Section 5.2).
At which stage should it be used? After a benchmark exists and a judge has been chosen, when new systems are to be ranked. The main text now gives the recipe (paragraph "How to use it", end of Section 5.2): write the questions, check on a few labelled systems whether the judge is weak, choose the test, and confirm the choice on held-out systems.
Is Sieve brute force? Is it practical? No, it is not brute force. The equation you refer to states the objective, and the search does not enumerate subsets. It draws 60 random subsets of 20 items, scores each, and keeps the best. Brute force would have to examine about 4 x 10^15 subsets even on our smallest task, which has 60 candidate items. Sieve takes about 3,000 judge calls per task-judge pair, a few minutes on one GPU. Its limit is different: its gain does not carry over to held-out systems, while Probe does for weak judges.
Thank you again for your time and for the detailed list of problems, which made the paper better. If the revision resolves your concerns, we would be grateful if you would consider updating your assessment.