- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes an urgent challenge facing AI conferences in the LLM era: rapidly growing submission volumes and the resulting deterioration in review quality, reproducibility, and trust in the publication process. The authors frame this as a vicious cycle and propose a lightweight “carrot-and-stick” reform of the peer-review system to improve accountability and safeguard publication quality at scale.
Critical strengths
This paper is easy to follow and takes on an important, urgent problem for the AI community, with a concrete and thought-provoking proposal for reforming peer review.
Critical weaknesses
- The paper appears to modify the standard ICLR formatting, particularly the vertical spacing around subsection headings (e.g., around Lines 85, 225, and similar locations), which may circumvent the intended page-limit constraints.
- The proposed reforms are mostly argued qualitatively rather than backed by empirical evidence. The paper does a nice job pointing out real problems in peer review, but it remains unclear whether these mechanisms would actually improve publication quality. It is also unclear what costs and unintended consequences they might introduce.
Questions for authors
- It would also help to discuss more systematically the possible unintended consequences of these reforms, including increased workload, strategic behavior, and risks introduced by stronger accountability and punishment mechanisms.
- Given the broad set of proposed reforms, how would the authors operationalize and evaluate them in practice? In particular, which components should be prioritized for pilot deployment, and what measurable criteria should be used to assess whether they actually improve publication quality?
- I wonder if the paper could better acknowledge ongoing complementary efforts to improve the peer-review ecosystem, such as improved reviewer assignment [1,2], detection of LLM-generated or low-quality reviews [3,4], and reviewer feedback or incentive mechanisms [5]. Acknowledging these efforts would give a more complete picture of how the community is already addressing the challenges identified in this paper.
[1] RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems. https://aclanthology.org/2026.acl-long.2035/
[2] MERIT: Matching Expertise via Rubric-Informed Training for Reviewer Assignment. https://arxiv.org/abs/2605.27865
[3] Detecting LLM-generated Peer Reviews. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0331871
[4] LazyReview: A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews. https://aclanthology.org/2025.acl-long.165/
[5] Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards. https://openreview.net/forum?id=l8QemUZaIA