acceptodds
Under review as a conference paper at ICLR 2027

What Matters in Dataset and Benchmark Papers? A Large-Scale Analysis

Abstract

To investigate the influence and importance of different factors in dataset and benchmark publications, we first systematically analyze 1,615 papers accepted to NeurIPS Datasets and Benchmarks Track between 2021 and 2025, combining OpenReview metadata, Semantic Scholar citations, GitHub repository statistics, and information extracted from paper PDFs. We then extend our analysis to the 3,951 papers concerning benchmarks and datasets published at NeurIPS, ICML, and ICLR. We consider numbers about the papers, such as the author counts and citations, as well as numbers in the papers, such as dataset sizes and the number of tasks. Papers with more authors generally receive more citations, and the number of stars and forks on GitHub is positively correlated with citation counts. Special recognition such as spotlight or oral strongly implies higher, even after controlling for paper's underlying quality using a regression discontinuity design. However, dataset size and the number of baselines show little correlation with citation counts. We hope these findings encourage a more rational use of such numbers when writing and reviewing dataset and benchmark papers, and we release our curated dataset to support further research.

Primary area:
datasets and benchmarks
Keywords:
Science of Science

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_isXUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a detailed statistical analysis of dataset and benchmark papers published at three major machine learning conferences (NeurIPS, ICML, and ICLR) over the past five years. The authors examine various factors that may be associated with the popularity and academic impact of these papers, providing a comprehensive overview of historical publication trends.

However, the work primarily focuses on descriptive and correlational analyses of existing publication data. It does not introduce new methodologies or provide substantial new insights into machine learning research or the development and evaluation of datasets and benchmarks. As a result, I find the novelty and significance of the contributions rather limited. Overall, while the analysis is thorough and potentially informative, I believe the paper is better suited as a bibliometric study or a report than as a research contribution to ICLR.

Critical strengths
  1. The paper provides a relatively comprehensive and detailed statistical analysis of dataset and benchmark papers published at the 3 major machine learning conferences over 2021-2025.
  2. It also presents several interesting empirical observations about the factors associated with the popularity and academic impact of these papers, which may provide useful references for future research.
Critical weaknesses
  1. The novelty is limited, and maybe it's not so related to ICLR conference. According to the ICLR 2027 Reviewer Guidelines, submissions are expected to demonstrate "new, relevant, impactful knowledge" to the machine learning community. However, it's not a paper about methodological, theoretical, or empirical studies. Although the paper provides a comprehensive statistical analysis, most of its findings are relatively intuitive or consistent with existing observations, such as the positive correlations between citation counts and the number of authors or GitHub stars. It is still unclear how these statistical findings could meaningfully inform the development or evaluation of datasets and benchmarks.
  2. Although the paper presents extensive statistical analyses, some conclusions are not sufficiently supported by the evidence. In particular, the authors emphasize the additional citation impact of Oral/Spotlight recognition, but the regression discontinuity analysis in Appendix E.3 (Table 22) does not show statistically significant effects. Moreover, citation counts may not accurately reflect the quality of datasets and benchmarks, and the substantial missing data in several analyzed features further limits the reliability of the conclusions.
Questions for authors
  1. Do similar findings hold for conferences in other research fields? For example, is the relationship between conference recognition (e.g., Oral/Spotlight awards) and academic impact also observed in other disciplines? A broader cross-disciplinary analysis might help reveal more general patterns in conference review and recognition systems, rather than findings specific to machine learning dataset and benchmark papers.
1 comment

We are grateful for the comments on novelty and broader relevance.

W1: The novelty is limited, and maybe it's not so related to ICLR conference. According to the ICLR 2027 Reviewer Guidelines, submissions are expected to demonstrate "new, relevant, impactful knowledge" to the machine learning community. However, it's not a paper about methodological, theoretical, or empirical studies. Although the paper provides a comprehensive statistical analysis, most of its findings are relatively intuitive or consistent with existing observations, such as the positive correlations between citation counts and the number of authors or GitHub stars. It is still unclear how these statistical findings could meaningfully inform the development or evaluation of datasets and benchmarks.

R1: This is an empirical meta-research study of ML dataset and benchmark publications. We agree that several marginal correlations are familiar and should be presented as replication findings rather than the main novelty. The contribution is the curated cross-venue corpus and the analysis of how recorded reviewer assessments, reported paper characteristics and conference recognition relate to citation outcomes in this setting. We believe these analyses will be valuable to the community.


W2: Although the paper presents extensive statistical analyses, some conclusions are not sufficiently supported by the evidence. In particular, the authors emphasize the additional citation impact of Oral/Spotlight recognition, but the regression discontinuity analysis in Appendix E.3 (Table 22) does not show statistically significant effects. Moreover, citation counts may not accurately reflect the quality of datasets and benchmarks, and the substantial missing data in several analyzed features further limits the reliability of the conclusions.

R2: The recognition finding is supported by both fixed effects regressions and exact-score comparisons. In an additional analysis comparing spotlight/oral papers with posters within conference–year–track–score groups, the ratio of geometric means of 1+citations is 1.59 (95% CI [1.34, 1.88]; N=3,163). The ratio remains above one near the thresholds and after outlier and cohort-influence checks. Both reported Table 22 estimates are positive but imprecise; their statistical insignificance leaves the causal effect unresolved without overturning this association. We will clearly distinguish these roles.


Q1: Do similar findings hold for conferences in other research fields? For example, is the relationship between conference recognition (e.g., Oral/Spotlight awards) and academic impact also observed in other disciplines? A broader cross-disciplinary analysis might help reveal more general patterns in conference review and recognition systems, rather than findings specific to machine learning dataset and benchmark papers.

A1: We think this is an excellent question to ask. Indeed, we believe one of the contributions of our paper is precisely uncovering and calling out this association in ML datasets and benchmarks. Yes, there is evidence about similar effects in other domains. Merton (1968) cited in Section 2 documents a broad “Matthew effect” where well-regarded papers attract disproportionately more citations and Azoulay et al. (2014), cited in Section 5, document a modest citation increase following prestigious recognition in biomedical research, although it is about author recognition rather than paper recognition itself. Our work provides complementary and novel evidence for ML datasets and benchmarks by linking recognition to reviewer assessments and comparing papers with identical recorded score summaries. We will happily expand the related work discussion to make these connections explicit.

Reviewer_9cLQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors examine factors associated with citation impact in dataset and benchmark papers. They find that author count, GitHub activity, and paper awards correlate with higher citation counts, while dataset size and baseline count show limited correlation. The authors advocate for more nuanced evaluation beyond quantitative metrics.

Critical strengths
  • The paper aligns with ongoing efforts to analyze datasets from perspectives such as ethical disclosure and distribution, which can contribute to the field.
  • The paper reconfirms several well-established findings.
Critical weaknesses
  • Above all, the choice of paper metadata as the analysis factors does not clearly show how the results contribute to the academic community. For example, why should researchers know that GitHub repository popularity is strongly correlated with citations? The paper does not make a sufficient effort to persuade readers that this information matters.
  • The description of the dataset does not clarify which conference papers are included. The paper initially states that it covers NeurIPS, ICLR, and ICML papers, but the section titles and body suggest that it covers only NeurIPS (as the title also implies). The main contribution of the paper is analyzing paper metadata, yet it does not even clarify what data it uses.
  • The paper mainly relies on correlation analysis to show relationships among paper metadata. Although it also includes multiple regression analyses and an RDD-inspired analysis of recognition effects, the analysis could be strengthened with more thorough treatment.
Questions for authors
  • What is the intended takeaway of this paper for the benchmark and dataset community? For example, how should dataset creators or reviewers use the finding that GitHub repository popularity is strongly correlated with citations?
  • How do the metadata factors (e.g., author affiliations, author counts) relate to the quality, usage, or validity of the datasets themselves?
  • Could the authors discuss potential confounders in the correlation and regression analyses (e.g., paper age, research area, or author reputation)? How robust are the results when these factors are controlled?
3 comments

We appreciate the reviewer’s focus on practical relevance and clarity.

W1: The practical value of the metadata findings is unclear.Above all, the choice of paper metadata as the analysis factors does not clearly show how the results contribute to the academic community. For example, why should researchers know that GitHub repository popularity is strongly correlated with citations? The paper does not make a sufficient effort to persuade readers that this information matters.

R1: The practical goal is to interpret popularity indicators and paper metadata more carefully. GitHub stars may provide an early signal of community interest, while author count and affiliation should be considered alongside direct evidence of dataset quality. Dataset creators can use repository attention and citations to describe reach while reporting validity, coverage, documentation and maintenance separately. Reviewers should assess those direct properties rather than infer quality from team size, institutional affiliation or popularity alone.


W2: The description of the dataset does not clarify which conference papers are included. The paper initially states that it covers NeurIPS, ICLR, and ICML papers, but the section titles and body suggest that it covers only NeurIPS (as the title also implies). The main contribution of the paper is analyzing paper metadata, yet it does not even clarify what data it uses.

R2: The corpus includes 1,615 NeurIPS D&B papers and 3,951 additional main-track papers from NeurIPS, ICML, and ICLR. Section 4 focuses on the D&B track; Section 5 extends the recognition analysis across venues. We will clarify this structure prominently.


W3: The paper mainly relies on correlation analysis to show relationships among paper metadata. Although it also includes multiple regression analyses and an RDD-inspired analysis of recognition effects, the analysis could be strengthened with more thorough treatment.

R3: The submitted paper includes within-year multivariable regressions (§4.8, Figure 5), rating-adjusted recognition regressions (§5, Table 4), and exact-score comparisons (Appendix E.2, Table 20). We have further strengthened the recognition analysis. On the same 4,190 papers used in Table 4, the recognition coefficient remains positive with conference-by-year fixed effects (0.297, SE 0.046) and cohort-specific rating slopes (0.311, SE 0.048), using ln(1+citations). Extending the exact-score approach in Appendix E.2, we compare recognized papers with posters within conference–year–track–score groups, obtaining a geometric-mean ratio of 1+citations of 1.59 (95% CI [1.34, 1.88]; N=3,163), robust to citation outliers and leave-one-cohort-out checks. These additional analyses support an association beyond recorded reviewer scores and cohort differences, while allowing for unobserved quality differences.


Q1: What is the intended takeaway of this paper for the benchmark and dataset community? For example, how should dataset creators or reviewers use the finding that GitHub repository popularity is strongly correlated with citations?

A1: GitHub stars may capture community interest earlier than citations, providing a potential early signal of dataset/benchmark popularity. Our correlation results motivate this possibility, although predictive performance requires longitudinal validation. Author count and affiliation should likewise be interpreted cautiously rather than treated as direct indicators of dataset quality.


Q2: How do the metadata factors (e.g., author affiliations, author counts) relate to the quality, usage, or validity of the datasets themselves?

A2: These factors describe collaboration and institutional context; they do not directly measure dataset quality, usage, or validity. We will distinguish citation reception and repository attention from these properties and narrow our conclusions accordingly.


Q3: Could the authors discuss potential confounders in the correlation and regression analyses (e.g., paper age, research area, or author reputation)? How robust are the results when these factors are controlled?

A3: These factors can confound the observed associations. Within-year analyses and the added conference-by-year recognition specifications account for broad cohort differences, but they do not equalize exact publication exposure or control fine-grained research area and prepublication author reputation. The Section 4 models therefore estimate conditional associations under their reported covariates, not effects adjusted for all of these factors.

I thank the authors for the clarifications. However, I agree with the other reviewers, that the novelty is still limited and that contributions are not the best fit for ICLR. As such, I still cannot recommend acceptance.

We thank the reviewer for the further feedback. The novelty lies in the curated cross-venue corpus and in connecting how ML datasets and benchmarks are reported, evaluated and recognized with the attention they receive. Since benchmarks and datasets help define what counts as progress in ML, understanding how they gain visibility is also a question about the field's evaluation infrastructure and research priorities.

Reviewer_LUrjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates which characteristics of dataset and benchmark publications are associated with reviewer ratings and citation impact. It compiles a multi-source corpus of 5,566 papers accepted at NeurIPS, ICML, and ICLR between 2021 and 2025, including 1,615 papers from the NeurIPS Datasets and Benchmarks Track. The analysis combines OpenReview metadata, Semantic Scholar citations, GitHub statistics, and LLM-extracted paper attributes. The authors report positive associations between citations and author count, reviewer ratings, and repository popularity, but little consistent association with reported dataset size or baseline count. They also examine whether spotlight and oral recognition are associated with additional citations beyond reviewer ratings, using regression adjustment, exact-score comparisons, and a fuzzy regression discontinuity analysis.

Critical strengths
  1. The paper studies a relevant question and compiles a large, multi-source corpus of 5,566 papers. The curated dataset could support useful research on dataset and benchmark publications.
  2. The analyses are generally well structured and include year-level comparisons and robustness checks. The finding that dataset size alone is not a reliable proxy for impact is useful.
Critical weaknesses

1.LLM extraction is insufficiently validated. The model sometimes adds information absent from the abstract, and no human-validated accuracy is reported. Substantial missing data also raises concerns about measurement error and selection bias. 2.The claims are stronger than the evidence. Accepted-paper analyses cannot establish effects on acceptance, and controlling for reviewer scores does not remove unobserved quality differences. The RD analysis lacks key diagnostics, has non-significant reduced-form and 2SLS estimates, and uses very few clusters. 3.Reporting and reproducibility need improvement. Sample counts, inclusion criteria, and reviewer-score definitions differ across analyses without sufficient explanation. Data collection, matching, and missingness handling need clearer documentation, and no anonymous dataset or code link is provided.

Questions for authors
  1. Can you report human-validated accuracy for the extracted variables and show whether the main results hold on verified data or with alternative extraction models?
  2. Please report the RD thresholds, bandwidth selection, manipulation and continuity tests, and robustness checks. How do the non-significant reduced-form results support the causal claims?
  3. Please reconcile the different sample counts across tables and clarify citation missingness, data collection date, and the planned dataset/code release and licensing.
2 comments

We thank the reviewer for the careful methodological feedback.

W1: LLM extraction is insufficiently validated. The model sometimes adds information absent from the abstract, and no human-validated accuracy is reported. Substantial missing data also raises concerns about measurement error and selection bias.

R1: All affiliation records for the NeurIPS Datasets and Benchmarks Track were manually verified before the initial submission. We corrected the initial submission’s extraction results using GPT-6.1 Sol. The current output should no longer include details that are not present in the abstract.

We manually verified 100 randomly sampled papers across five fields: dataset_size (dataset size), num_tasks (task count), num_baselines (baseline count), dataset_name and benchmark_name (names of newly introduced datasets and benchmarks). Accuracy was 92.8% overall, including correct blanks, and 95.8% among non-empty predictions. Measurement error and potential selection bias from non-random missingness remain limitations.


W2: The claims are stronger than the evidence. Accepted-paper analyses cannot establish effects on acceptance, and controlling for reviewer scores does not remove unobserved quality differences. The RD analysis lacks key diagnostics, has non-significant reduced-form and 2SLS estimates, and uses very few clusters.

R2: Our main finding is a positive recognition citation association beyond recorded reviewer scores. On the same 4,190 papers, the coefficient remains positive with conference by year fixed effects (0.297, SE 0.046) and cohort-specific rating slopes (0.311, SE 0.048), using ln(1+citations). Exact-score comparisons reinforce this finding, while allowing for unobserved quality differences. We will align the abstract and conclusion with this evidence and clarify that our accepted paper corpus does not estimate acceptance effects.


W3: Reporting and reproducibility need improvement. Sample counts, inclusion criteria, and reviewer-score definitions differ across analyses without sufficient explanation. Data collection, matching, and missingness handling need clearer documentation, and no anonymous dataset or code link is provided.

R3: The collected dataset was already uploaded with the initial submission. We will clarify inclusion criteria, sample exclusions, score definitions, matching, and missingness handling, and specify the release and licensing arrangements.

Q1: Can you report human-validated accuracy for the extracted variables and show whether the main results hold on verified data or with alternative extraction models?

A1: All author-affiliation records for the NeurIPS D&B track were manually verified. After the GPT-6.1 Sol correction pass, we manually evaluated the five extracted fields on 100 randomly sampled papers. Field-level agreement was 92.8% overall, including correctly blank fields, and 95.8% among non-empty predictions. The key conclusions have not changed.


Q2: Please report the RD thresholds, bandwidth selection, manipulation and continuity tests, and robustness checks. How do the non-significant reduced-form results support the causal claims?

A2: Table 22 is a supplementary threshold analysis. Its two reported reduced form and IV estimates are positive, consistent with the idea that recognition has additional positive impact on citation. While the confidence intervals at 95% do include zero. This is not evidence against our finding. Due to smaller sample sizes about cutoffs, a more charitable interpretation is that we simply did not have enough statistical power to detect the effect. We include the analysis for transparency, especially when it is directionally aligned with our main findings from fixed effects.

The reported specifications use locally constant regression to account for the fact that our running variable, ratings, is itself somewhat discrete. Half-windows are 0.5 raw score points for spotlight versus poster (N=867; eight conference-year clusters) and 1 point for oral versus spotlight (N=398; six clusters). These are fixed comparison windows, without a formal optimal-bandwidth selector. Cutoffs were selected empirically from first-stage relationships and frozen before the final analysis. The spotlight/poster cutoffs are ICLR 2021/2022/2024/2025: 6.75/7.5/7.5/7.75; NeurIPS 2021/2023/2024/2025: 7.25/8.25/8.25/5.25. Oral/spotlight cutoffs are ICLR main 2024/2025: 6.75/7.5; NeurIPS main 2021: 6.75; NeurIPS D&B/main 2023: 8.25/7.75, 2024: 8.25/7.25, and 2025: 5.25/5.25.

We would also like to point out that exact score comparison is statistically significant and consistently shows a positive impact from recognition on citation, against a variety of robustness checks including citation outlier adjustments, and leave-one-cohort-out checks.


Q3: Please reconcile the different sample counts across tables and clarify citation missingness, data collection date, and the planned dataset/code release and licensing.

A3: The dataset was uploaded with the initial submission, and data collection ended on September 26, 2026. It will be publicly released upon acceptance or alongside the arXiv version.

The combined corpus contains 5,566 papers, but each analysis has its own eligibility and overlap restrictions. Table 3 uses 4,430 contributing papers from cohorts containing all three recognition categories. The combined-recognition total-citation model in Table 4 uses 4,190 papers with the required ratings, outcomes and eligible recognition labels. The preferred-total spotlight comparison in Table 20 uses 3,317 papers, and the oral/spotlight comparison uses 452; the two Table 22 windows contain 867 and 398 papers.

Comments on the paper

1 comment

We thank the reviewers for their comments and careful reading. The main changes and additions are marked in blue in the revised manuscript.

We clarified the two analysis scopes: 1,615 NeurIPS D&B papers and a broader recognition analysis covering 5,566 papers. We corrected the abstract extractions and updated the related analyses. A manual check of 100 papers across five fields found 92.8% agreement overall, including correct blanks, and 95.8% among non-empty predictions.

Our point-by-point responses address questions about the data, methods and interpretation of the findings. For example, we clarify the setup and interpretation of the supplementary RD analysis in Appendix E.3 (Table 22). Separately, our exact-score comparisons show a citation gap between spotlight/oral papers and posters with the same recorded scores. This association makes conference recognition relevant to understanding which datasets and benchmarks gain visibility in the ML community.

The novelty lies in the curated cross-venue corpus and in connecting how ML datasets and benchmarks are reported, evaluated and recognized with the attention they receive. Since benchmarks and datasets help define what counts as progress in ML, understanding how they gain visibility is also a question about the field's evaluation infrastructure and research priorities.

The updated data and code has been accompanied with the revision and will be made public upon acceptance or alongside the arXiv version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.