- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper claims to measure how much information a new benchmark adds vs. all (or a large number of known) existing benchmarks.
Critical strengths
I like the core idea of the paper, if it were executed well.
Critical weaknesses
This paper is in my opinion unreadable. I strongly suspect the writing was heavily AI-generated. Even if not, the writing is not clear and not up to the standards of science, let alone a top-tier AI conference. It is simply not written in a way where one can fully understand what is going on. There are frequent references, even as early as the abstract and first few sentences, to concepts that have not been introduced or explained. My suspicion is that is because the AI is referring to topics in its context, either in its own reasoning traces or things it has discussed with the authors. The AI use section admits heavy reliance on AI for much of (most of?) the process that led to this paper, and it shows.
A sample size of two human raters is not a large enough to tell signal from noise.
Questions for authors
How much did you use AI to write the paper? Did it do the initial drafts, or did you? Did you let it rewrite sentences? Paragraphs? Whole sections? If this represents your writing, please work on the skill of more clearly describing what you have done and not making reference to lots of not-yet-explained concepts.
How much did you use AI for the rest of the science? Have you checked all the code that it wrote? Are you an expert in the statistical methods such that you know if they were properly done? Etc?