- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper looks at whether model rankings in leaderboards are robust to the specific composition of each benchmark (ie which specific samples are including in the benchmark). Their method builds subsets of the same benchmarks which consist of low differential functioning samples and compares scores on those benchmarks to scores on a random subsample of the same benchmarks. They find that while scores are very correlated in both settings, there is still a noticeable amount of ranking reversals, particularly within pairs of models that originally have close scores.
Critical strengths
This paper is very well presented, motivated, and easy to follow. For instance I found figure 1 very helpful and visually pleasing. From my understanding (not very familiar with this field), their method appears sound and results robust.
Critical weaknesses
- I am not very familiar with this field, but the authors' main finding, that rankings are not always stable, especially when initial score differences are small, does not seem very surprising. I can also therefore not properly assess novelty and impact of the work.
- The authors should further elaborate on takeaways from their method. Can this be used to inform either the construction of benchmarks or the interpretation of benchmark results?
- The authors' analysis does not include top frontier models like claude gemini or chatgpt. Even though from my understanding open models are not required for this analysis?
Questions for authors
- Can you please elaborate on what figure 2 shows? How correlated are ranking reversals to the initial difference in performance between both models?
- How do you explain that reversal rates are different for different benchmarks? For instance why is WinoGrande so different?
- Are their other methods that assess confidence in benchmark results that this work should compare to? Or even simple statistical methods like bootstrapping or confidence intervals?
Manuscript revisions to improve clarity and readability
Dear reviewers and Area Chair,
Thank you for the detailed reviews and follow-up comments. Over the past few days, we have worked through the manuscript section by section, with particular attention to the concerns about terminology, dense explanations, and readability.
To make the revisions easier to review, the marked manuscript uses:
The main changes are as follows:
Clarified the research question and contribution. We revised the abstract and introduction to explain more directly what the audit tests and why the comparison with matched-random subtests matters.
Simplified terminology and reduced repetition. We removed overlapping names for the same concepts, introduced key definitions earlier, and shortened repeated explanations.
Reworked the method explanations. We revised dense passages, made the connections between capability adjustment, DIF estimation, and ranking evaluation more explicit.
Incorporated additional analyses and clarified their interpretation. Appendix H now presents further analyses discussed in our responses, including bootstrap intervals for model-pair score differences, WinoGrande diagnostics, and prediction of DIF from held-out item text.
We recognize that our earlier wording sometimes made it hard for readers to follow the argument. Improving this has been a central priority of the revision. We appreciate the reviewers’ careful engagement and the specific examples that helped us identify where we needed clearer explanations.
Thanks again to all the reviewers, your opinions really helped us make the paper better!
Best regards, The Authors