- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
LLM benchmark scores obviously strongly correlate with release date. This paper conducts a very careful statistical analysis of LLM benchmark scores, with the main focus of isolating whether economic benchmarks like GDPVal and Terminal-Bench pick up on additional "economic capability" signal beyond overall capabilities. They answer this both with a factor analysis and with predictive validity tests.
Critical strengths
Many robustness checks are conducted.
Critical weaknesses
-
The research question is not interesting. It is already well-accepted in the community that benchmark scores improve with release date, and it is also obvious that there is secondary signal in domain-specific benchmarks. Moreover there are no methodological contributions or interesting experiments. The paper just takes existing LLM benchmark data and applies a wide battery of known techniques, mostly things related to factor analysis and leave-one-out prediction power, to answer its questions.
-
The writing of the paper is obscure and could use some more careful human attention.
Questions for authors
None
Response to Reviewer h6UD
Thank you for the review. We agree that benchmark scores rising with release date is expected, and that domain-specific benchmarks are likely to carry secondary signal. The paper does not claim either as a discovery, and it introduces no new estimator. Its contribution is narrower, an audit with thresholds fixed in advance. It measures how large these effects are on a current frontier leaderboard, finds that the release-date reduction misses its pre-specified threshold on the primary grid, and shows that whether economic benchmarks count as distinct depends on which test and which retention rule a reader applies. Leaderboards now publish economic columns as separate rankings, so we think the size of these effects is worth stating with intervals, though we accept that this is a modest contribution. On the writing, several reviewers found the paper hard to follow, and we will shorten the abstract, add background on the factor model, and cut methodological detail from the main text.
AI use. The investigation, the analytic decisions and the positions taken here are ours. Generative AI tools assisted with polishing our text drafted for this response and with running analyses we specified, as set out in our shared response.