- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how changes in grading rules affect model comparisons on SWE-bench Verified, EvalPlus, and LiveCodeBench. By regrading identical saved outputs, it shows that many previously clear model leads disappear or reverse, demonstrating that leaderboard conclusions can depend heavily on the grader. The paper therefore advocates “claim-first” reporting: researchers should state the specific conclusion they want to establish and verify the comparisons required for it, rather than drawing conclusions directly from leaderboard rank.
Critical strengths
None
Critical weaknesses
- The paper is not quite readable.
- The paper studies something that is trivial - if you change the grading of a benchmark, the eval outcome changes too. I can't see any value provided by this paper.
Questions for authors
None
Readability and the decisions affected by regrading
Dear Reviewer fDm4,
Thank you so much for taking the time to review our paper and for drawing attention to its presentation and contribution. Even when a critique is brief, we genuinely value any signal that helps us make the paper clearer to readers and more transparent in its motivation. With that spirit, we would like to share our revision on readability and clarify the core idea, novelty, and practical benefit of our framework.
Logic Mainline and Readability Improvements
In the submitted draft, we introduced formal notation before motivating why a researcher needs it. In the revision, we reorganized the logic so that a concrete research decision on page 1 leads naturally into the problem, the notation, and the joint inference:
We would be very grateful if you could let us know which specific passage, figure, or step would benefit most from further clarification—we are more than happy to refine it further.
Core Idea, Novelty, and Practical Benefit
You are right that changing how a benchmark is graded can change a score. The core idea and novelty of our paper lie not in observing that scores or rankings shift, but in providing a principled way to decide what scientific claims remain supported—and which baselines or comparisons are actually required—when an evaluation changes:
Thank you again for your time, your feedback, and your consideration of our revised manuscript. We warmly welcome any further comments or suggestions you may have.