acceptodds
Under review as a conference paper at ICLR 2027

Safeguarding Publication Caliber Amidst Massive AI Conference Submissions in the Era of LLMs

Abstract

The explosion of AI research, accelerated by Large Language Models (LLMs) and agent-powered tools, has led to an unprecedented surge in conference submissions. This deluge imposes a tremendous burden on the peer-review process, triggering a vicious cycle characterized by low-quality reviews, stochastic decision-making, and a reproducibility crisis. In this paper, we analyze the structural failures of current AI reviewing paradigms and contrast them with established practices in other computer science subfields. We find that existing mechanisms in AI conferences fail to effectively constrain irresponsible behavior from both authors and reviewers in the LLM era. To break this cycle, we propose the “carrot and stick" methods that introduce lightweight yet high-impact modifications for authors, reviewers, and conference organizers. Our method incorporates mandatory revision plans, a real-name shepherding system, and artifact evaluation (AE) for high-tier papers including oral and spotlight. By optimizing workflows and UI features, such as explicit score-increase justifications, our practical, cost-effective approach aims to safeguard publication caliber and restore the academic reputation of AI conferences.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
AI conference peer review, research integrity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8LaAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes an urgent challenge facing AI conferences in the LLM era: rapidly growing submission volumes and the resulting deterioration in review quality, reproducibility, and trust in the publication process. The authors frame this as a vicious cycle and propose a lightweight “carrot-and-stick” reform of the peer-review system to improve accountability and safeguard publication quality at scale.

Critical strengths

This paper is easy to follow and takes on an important, urgent problem for the AI community, with a concrete and thought-provoking proposal for reforming peer review.

Critical weaknesses
  • The paper appears to modify the standard ICLR formatting, particularly the vertical spacing around subsection headings (e.g., around Lines 85, 225, and similar locations), which may circumvent the intended page-limit constraints.
  • The proposed reforms are mostly argued qualitatively rather than backed by empirical evidence. The paper does a nice job pointing out real problems in peer review, but it remains unclear whether these mechanisms would actually improve publication quality. It is also unclear what costs and unintended consequences they might introduce.
Questions for authors
  • It would also help to discuss more systematically the possible unintended consequences of these reforms, including increased workload, strategic behavior, and risks introduced by stronger accountability and punishment mechanisms.
  • Given the broad set of proposed reforms, how would the authors operationalize and evaluate them in practice? In particular, which components should be prioritized for pilot deployment, and what measurable criteria should be used to assess whether they actually improve publication quality?
  • I wonder if the paper could better acknowledge ongoing complementary efforts to improve the peer-review ecosystem, such as improved reviewer assignment [1,2], detection of LLM-generated or low-quality reviews [3,4], and reviewer feedback or incentive mechanisms [5]. Acknowledging these efforts would give a more complete picture of how the community is already addressing the challenges identified in this paper.

[1] RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems. https://aclanthology.org/2026.acl-long.2035/

[2] MERIT: Matching Expertise via Rubric-Informed Training for Reviewer Assignment. https://arxiv.org/abs/2605.27865

[3] Detecting LLM-generated Peer Reviews. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0331871

[4] LazyReview: A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews. https://aclanthology.org/2025.acl-long.165/

[5] Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards. https://openreview.net/forum?id=l8QemUZaIA

Reviewer_CPa8OpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper first analyses the vicious cycle problems in AI conference submissions in the Era of LLMs, and then proposes a "carrot-and-stick" style solutions. The work is significant for improving the current AI conference organization but is out of the scope of the ICLR conference since it focuses on paper review strategy instead of research issues.

Critical strengths

None

Critical weaknesses

Few research issues are considered in this paper. In addition the proposal is not verified and validated with simulation or real world scenarios.

Questions for authors

None

Reviewer_cXxKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The article argues that the growth in submissions, the burden on reviewers, low-quality feedback, and issues with reproducibility create a self-reinforcing cycle. The authors propose strengthening reviewer accountability, requiring post-acceptance revision plans, introducing named "shepherds," and implementing artifact evaluation for papers selected for oral or spotlight presentations. While the appendix compares scores for certain papers across ICLR and ICML, it does not evaluate the actual effectiveness of the proposed reforms.

Critical strengths

• A method has been proposed, aimed at ensuring the quality of academic conferences. • Suggest limit the artifact evaluation and spotlight papers.

Critical weaknesses

• It is necessary to distinguish the differences. Kim et al. [1] already propose author feedback, reviewer accountability, and formal reviewer rewards. Pineau et al. [2] report an established NeurIPS reproducibility program. • The paper provides no empirical evaluation of the proposed reforms or quantitative estimate of the workload. • Regarding the statistics for papers rejected by ICLR but subsequently accepted by ICML, a score correlation near zero fails to distinguish between reviewer randomness, selection effects, or differences between the two conferences. [1] Kim, J., Lee, Y., and Lee, S. Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards. ICML, 2025. [2] Pineau, J., et al. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). JMLR, 2021.

Questions for authors

• What evidence supports the claim that the proposal adds little workload? Please estimate the recurring costs of shepherd checks, artifact evaluation, and complaint handling. • How does selecting only ICLR-rejected and ICML-accepted papers affect the correlations?

• What criteria would chair use to distinguish malicious score changes from justified changes in judgment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.