- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes an approach to decide how conlcusions from a benchmark change or loose validity when the benchmark is updated. However, it is written quite badly and I cannot understand the exact problem statement and proposed solution.
Critical strengths
I cannot find any critical strength in this paper. The contribution seems marginal.
Critical weaknesses
The paper is not readable despite considerable effort. It does not formulate the problem clearly. The paper would thus need a full revision in order to be eligible for ICLR or similar top tier conferences.
Questions for authors
N/A
2 comments
Dear authors, thank you very much for the rebuttal. I do appreciate the effort. After reading the other reviews, I am convinced though that my judgement of the paper will not change and I unfortunately do not have the capacity to give the paper another full read (I would do this if it was borderline or going to be accepted).
Thanks for the comments.
You could not identify the problem or the proposed solution. We rewrote the introduction around one decision: after a benchmark is revised, can its maintainer keep the published top five, or the winner, without rescoring every stored answer?
The inputs are the old scores, the unchanged model outputs and a restriction on what the update may change. Compare the old lead with the most it could lose under that restriction. If the lead is larger, its sign cannot change. Otherwise the rule leaves the comparison open; it does not predict a reversal. Section 2 gives a five-question example.
On WorkBench, nine tasks have revised reference actions; the scorer, tools, initial state and unaffected judgments stay fixed. Each changed binary task could cost the old leader at most two points relative to a rival, hence at most 18 in total. The fifth model led the sixth by 19, so no model outside the top five could enter. The winner led by only 7. It did survive, but confirming that required rescoring.
The contribution is empirical, not a new algorithm. On the same saved arithmetic answers, changing extraction reversed 126 of the 1,634 comparisons above the original margin screen; correcting answers, removing items or doing both reversed none. The winner stayed put while the top-three and top-five sets changed. We also measured the work behind a reuse decision: with the changed WorkBench tasks identified and the same caches, resolving all pair relations took 92 action-list executions versus 132 executions when scoring every affected model–task cell. Generic intervals with cached version differences gave the same plan (Section 5, Table 4).