- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses an interesting question: how much does the leaderboard ranking of an AI model depend on the concete evaluation view of the person performing the benchmark experiments. Studying this question is also motivated by a convincing example: switching from a one-attempt to a four-attempt success rate analysis makes the #1 and #12 reverse for DeepSWE, indicating that leaderboards should always be taken with a pinch of salt and that the underlying evaluation view has to be made transparent.
To address this issue, the paper introduces a couple of measures (LAI, SLAI,d_ha) that are more robust under different evaluation views, in essence by relying on a worst case view over all different such views. The paper demonstrates the proposed measures on real-world benchmark suites and theoretically connects them with each other and also with alternative measures known from the related literaturs.
Critical strengths
-
The paper addresses an interesting and timely topic, which is well-motivated in its introduction.
-
The idea of introducing a measure that quantifies the robustness / stability of benchmark results / leaderboards seems promising and convincing.
Critical weaknesses
-
The paper introduces several different measures without giving enough contextual interpretation of those measures. In this way, at least for me, it was hard to identify the benefits of thoe measures over existing ones (and also among the introduced ones).
-
The experiments and results sections (Sections 4 and 5) are very confusing to me. (1) The description of the experimental setup is unclear to me: why the distinction between design panels and additional panels? what different evaluayion views ar concretely considered? I think the paper really should make an effort to explain this setup more carefully and to relate it to the formal model introduced in its previous sections. (2) I find it rather uncommon to mix the experimental results in Sections 5 with the theoretical results (Prop 1, Theos 1 and 2). Would it make sense to clearly separate those parts?
Questions for authors
-
Why not stop with LAI? What's the advantage of using SLAI rather than LAI directly? It would be nice to have this clearly explained in the main text. (In general, some intuition/explanation next to the introduced formulas would be nice).
-
I like the idea of checking leaderboards for robustness towards the evaluation view. Another aspect coming to mind here is the "sample dependence" of leaderboards, i.e., leaderboards changing under (slight) changes in the underlying benchmark suite. Is there any possibilities to obtain statistical guarantees for the proposed measures that apply beyond the concrete suite considered? In the spirit of recent works such as: https://doi.org/10.52202/079017-3115
Response to Reviewer 8Gk9: Appreciation and Clarification on Anonymity
Thank you for your thoughtful review and positive assessment. We sincerely appreciate your recognition of the problem’s importance, the compelling DeepSWE example, our theoretical results on ranking-position sensitivity and duplicate views, and the independent computational checks.
Regarding the anonymity concern, the cited passages record data provenance and analytical chronology rather than author identity. The bibliography entry cites Datacurve’s public third-party DeepSWE v1.1 benchmark, noting its release and snapshot dates to identify the exact public dataset evaluated. Similarly, the local protocol and public-data retrieval timestamps in Appendix B.2, with the UTC+8 offset specifying the time convention of those records, document the chronology of operations on previously public data, establishing that the SLAI definition, default weights, comparison measures, and verification approach were fixed before the additional numerical analyses.
The quoted material contains no author names, affiliations, or personal contact details, and the cited URL points to a public third-party benchmark resource. Together, these records serve to make the evidence and analysis chronology verifiable. We therefore respectfully clarify that the cited examples document research provenance and do not disclose the authors’ identities.