- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a detailed statistical analysis of dataset and benchmark papers published at three major machine learning conferences (NeurIPS, ICML, and ICLR) over the past five years. The authors examine various factors that may be associated with the popularity and academic impact of these papers, providing a comprehensive overview of historical publication trends.
However, the work primarily focuses on descriptive and correlational analyses of existing publication data. It does not introduce new methodologies or provide substantial new insights into machine learning research or the development and evaluation of datasets and benchmarks. As a result, I find the novelty and significance of the contributions rather limited. Overall, while the analysis is thorough and potentially informative, I believe the paper is better suited as a bibliometric study or a report than as a research contribution to ICLR.
Critical strengths
- The paper provides a relatively comprehensive and detailed statistical analysis of dataset and benchmark papers published at the 3 major machine learning conferences over 2021-2025.
- It also presents several interesting empirical observations about the factors associated with the popularity and academic impact of these papers, which may provide useful references for future research.
Critical weaknesses
- The novelty is limited, and maybe it's not so related to ICLR conference. According to the ICLR 2027 Reviewer Guidelines, submissions are expected to demonstrate "new, relevant, impactful knowledge" to the machine learning community. However, it's not a paper about methodological, theoretical, or empirical studies. Although the paper provides a comprehensive statistical analysis, most of its findings are relatively intuitive or consistent with existing observations, such as the positive correlations between citation counts and the number of authors or GitHub stars. It is still unclear how these statistical findings could meaningfully inform the development or evaluation of datasets and benchmarks.
- Although the paper presents extensive statistical analyses, some conclusions are not sufficiently supported by the evidence. In particular, the authors emphasize the additional citation impact of Oral/Spotlight recognition, but the regression discontinuity analysis in Appendix E.3 (Table 22) does not show statistically significant effects. Moreover, citation counts may not accurately reflect the quality of datasets and benchmarks, and the substantial missing data in several analyzed features further limits the reliability of the conclusions.
Questions for authors
- Do similar findings hold for conferences in other research fields? For example, is the relationship between conference recognition (e.g., Oral/Spotlight awards) and academic impact also observed in other disciplines? A broader cross-disciplinary analysis might help reveal more general patterns in conference review and recognition systems, rather than findings specific to machine learning dataset and benchmark papers.
We are grateful for the comments on novelty and broader relevance.
W1: The novelty is limited, and maybe it's not so related to ICLR conference. According to the ICLR 2027 Reviewer Guidelines, submissions are expected to demonstrate "new, relevant, impactful knowledge" to the machine learning community. However, it's not a paper about methodological, theoretical, or empirical studies. Although the paper provides a comprehensive statistical analysis, most of its findings are relatively intuitive or consistent with existing observations, such as the positive correlations between citation counts and the number of authors or GitHub stars. It is still unclear how these statistical findings could meaningfully inform the development or evaluation of datasets and benchmarks.
R1: This is an empirical meta-research study of ML dataset and benchmark publications. We agree that several marginal correlations are familiar and should be presented as replication findings rather than the main novelty. The contribution is the curated cross-venue corpus and the analysis of how recorded reviewer assessments, reported paper characteristics and conference recognition relate to citation outcomes in this setting. We believe these analyses will be valuable to the community.
W2: Although the paper presents extensive statistical analyses, some conclusions are not sufficiently supported by the evidence. In particular, the authors emphasize the additional citation impact of Oral/Spotlight recognition, but the regression discontinuity analysis in Appendix E.3 (Table 22) does not show statistically significant effects. Moreover, citation counts may not accurately reflect the quality of datasets and benchmarks, and the substantial missing data in several analyzed features further limits the reliability of the conclusions.
R2: The recognition finding is supported by both fixed effects regressions and exact-score comparisons. In an additional analysis comparing spotlight/oral papers with posters within conference–year–track–score groups, the ratio of geometric means of 1+citations is 1.59 (95% CI [1.34, 1.88]; N=3,163). The ratio remains above one near the thresholds and after outlier and cohort-influence checks. Both reported Table 22 estimates are positive but imprecise; their statistical insignificance leaves the causal effect unresolved without overturning this association. We will clearly distinguish these roles.
Q1: Do similar findings hold for conferences in other research fields? For example, is the relationship between conference recognition (e.g., Oral/Spotlight awards) and academic impact also observed in other disciplines? A broader cross-disciplinary analysis might help reveal more general patterns in conference review and recognition systems, rather than findings specific to machine learning dataset and benchmark papers.
A1: We think this is an excellent question to ask. Indeed, we believe one of the contributions of our paper is precisely uncovering and calling out this association in ML datasets and benchmarks. Yes, there is evidence about similar effects in other domains. Merton (1968) cited in Section 2 documents a broad “Matthew effect” where well-regarded papers attract disproportionately more citations and Azoulay et al. (2014), cited in Section 5, document a modest citation increase following prestigious recognition in biomedical research, although it is about author recognition rather than paper recognition itself. Our work provides complementary and novel evidence for ML datasets and benchmarks by linking recognition to reviewer assessments and comparing papers with identical recorded score summaries. We will happily expand the related work discussion to make these connections explicit.