- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper conducts a study of peer reviews associated with 300 ICLR submissions spanning 2025-2026 to assess the prevalence of full-AI generation and AI editing in peer reviews. An existing claim-level semantic detection framework (SemDetect) is applied to classify reviews into three categories: human-associated, rewrite-associated, and AI-associated. The results suggest that only 0.1% of reviews are AI associated, with 65.9% classified as rewrite-associated and 34% as human associated.
Critical strengths
- The problem that this work aims to address is important and timely: to manage the increasing reliance on AI tools in peer review, it is critical to distinguish AI editing from more extensive AI writing of peer reviews.
Critical weaknesses
-
The technical contributions of this work are very weak. It is limited to the application of a single existing approach (SemDetect) to a relatively small number of peer reviews for 300 ICLR papers from 2025-2026. The method and the ICLR reviews are already publicly available so the only contribution is the analysis, which itself is flawed (see weakness #2).
-
The approach used to conduct the analysis was not validated against any ground truth data, such as pre-2022 peer reviews (which can be reasonably assumed to be human written). Only a single model (gpt-5.4-nano) is used to generate reference AI reviews, which is not representative of the various frontier models that are most likely to be used in practice. These methodological flaws make the finding that only ~0.1% of reviews are “AI associated” (i.e., not human written or merely AI-rewritten) highly suspicious.
Questions for authors
-
What is the purpose of the “two manuscript verification tiers” (line 212)?
-
How many “Human associated”, “Rewrite associated”, and “AI associated” reviews are flagged for reviews for which you can infer a ground truth label? E.g., pre-2022 reviews which can be labeled as human, and AI-edited or fully AI-generated reviews that are produced for the same papers.
Control Results and Manuscript Verification
Thank you for requesting the control counts. For the original nano-only and three-reference configurations, respectively, the counts below are in the order human-associated / rewrite-associated / AI-associated / unscorable.
The verification tiers document manuscript provenance. For 2025, we downloaded and hashed the source PDFs and checked titles and anonymity. For 2026, we used title- and anonymity-checked text from a pinned, hash-verified benchmark. Each tier corresponds to one cohort, so their effects cannot be estimated separately. We could not verify the exact submission-cutoff revision for every paper.