acceptodds
Under review as a conference paper at ICLR 2027

Measuring LLM-Associated Signatures in Peer Review: A Comparative Study of ICLR 2025 and 2026

Abstract

Large Language Models (LLMs) are increasingly used to support academic peer review through linguistic refinement and content generation. However, detectors based primarily on surface-level writing patterns may conflate language editing with automated scientific judgment. This paper presents a comparative empirical study of 300 ICLR 2025–2026 submissions and 1,201 linked official reviews, including 75 accepted and 75 rejected papers from each year. SemDetect, a claim-level semantic detection framework, is used to distinguish human-associated, rewrite-associated, and AI-associated signatures. Of 1,178 scorable reviews, 776 (65.9%) are classified as rewrite associated, 401 (34.0%) as human associated, and one (0.1%) as AI associated; 23 reviews are retained separately as unscorable. Human-associated and rewrite-associated reviews have similar mean ratings (5.11 versus 5.06) and reviewer-reported confidence (3.58 versus 3.57), but rewrite-associated reviews exhibit lower raw rating dispersion (1.69 versus 1.89). At the paper level, rewrite proportion is negatively correlated with within-paper rating variance, although the association does not meet the conventional significance threshold after adjustment for mean rating, review count, and conference cohort (). Rewrite prevalence is also similar for accepted and rejected papers (66.3% versus 65.5%) and is not independently associated with final decisions. The findings indicate that the dominant detected signature is refinement rather than complete review generation. They also show that rating dispersion provides complementary information to average ratings, although the adjusted evidence for variance compression remains inconclusive. Because detection cannot establish authorship, all outputs are interpreted as probabilistic indicators rather than proof of individual LLM use.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models (LLMs), AI-Assisted Peer Review, Semantic Text Detection, Rating Dispersion, Scientific Evaluation, Responsible AI, Computational Metascience, OpenReview

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iwKTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper conducts a study of peer reviews associated with 300 ICLR submissions spanning 2025-2026 to assess the prevalence of full-AI generation and AI editing in peer reviews. An existing claim-level semantic detection framework (SemDetect) is applied to classify reviews into three categories: human-associated, rewrite-associated, and AI-associated. The results suggest that only 0.1% of reviews are AI associated, with 65.9% classified as rewrite-associated and 34% as human associated.

Critical strengths
  1. The problem that this work aims to address is important and timely: to manage the increasing reliance on AI tools in peer review, it is critical to distinguish AI editing from more extensive AI writing of peer reviews.
Critical weaknesses
  1. The technical contributions of this work are very weak. It is limited to the application of a single existing approach (SemDetect) to a relatively small number of peer reviews for 300 ICLR papers from 2025-2026. The method and the ICLR reviews are already publicly available so the only contribution is the analysis, which itself is flawed (see weakness #2).

  2. The approach used to conduct the analysis was not validated against any ground truth data, such as pre-2022 peer reviews (which can be reasonably assumed to be human written). Only a single model (gpt-5.4-nano) is used to generate reference AI reviews, which is not representative of the various frontier models that are most likely to be used in practice. These methodological flaws make the finding that only ~0.1% of reviews are “AI associated” (i.e., not human written or merely AI-rewritten) highly suspicious.

Questions for authors
  1. What is the purpose of the “two manuscript verification tiers” (line 212)?

  2. How many “Human associated”, “Rewrite associated”, and “AI associated” reviews are flagged for reviews for which you can infer a ground truth label? E.g., pre-2022 reviews which can be labeled as human, and AI-edited or fully AI-generated reviews that are produced for the same papers.

1 comment
AuthorsOpenReview ↗

Control Results and Manuscript Verification

Thank you for requesting the control counts. For the original nano-only and three-reference configurations, respectively, the counts below are in the order human-associated / rewrite-associated / AI-associated / unscorable.

  • Historical reviews: 15 / 3 / 0 / 2 and 16 / 2 / 0 / 2
  • Controlled rewrites: 19 / 17 / 0 / 4 and 19 / 17 / 0 / 4
  • Fully generated reviews: 26 / 13 / 1 / 0 and 28 / 5 / 7 / 0 The historical reviews serve as a proxy for human-written reviews. Across all four reference configurations, full-generation recognition in the primary diagnostic was 1/40 with nano, 1/40 with GPT-4.1-mini, 0/40 with GPT-6.2-Astra, and 7/40 with all three references. The results lead us to withdraw our inference about the rarity of full generation.

The verification tiers document manuscript provenance. For 2025, we downloaded and hashed the source PDFs and checked titles and anonymity. For 2026, we used title- and anonymity-checked text from a pinned, hash-verified benchmark. Each tier corresponds to one cohort, so their effects cannot be estimated separately. We could not verify the exact submission-cutoff revision for every paper.

Reviewer_ekbWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes 1,201 reviews of 300 ICLR 2025–2026 submissions using SemDetect to identify human-associated, rewrite-associated, and AI-associated patterns. It examines how these categories relate to review scores, reviewer confidence, rating dispersion, and acceptance decisions. The detector classifies most reviews as rewrite-associated. These reviews have similar average scores and confidence to human-associated reviews, while evidence linking rewrite prevalence to lower reviewer disagreement remains inconclusive after adjustment.

Critical strengths
  1. The paper addresses a relevant question about LLM assistance in peer review and examines rating dispersion alongside average scores.
  2. The study distinguishes LLM-assisted editing from fully generated reviews, an important distinction when assessing whether AI supports reviewers’ own judgments or substitutes for them.
Critical weaknesses
  1. The original Sem-Detect study reports that 35.38% of human-written reviews are incorrectly classified as LLM-refined. This raises concerns about this paper’s finding that 65.9% of reviews are rewrite-associated. The authors do not validate the detector on their data or examine how these errors might affect their results.
  2. The study covers only 300 papers from two editions of ICLR, despite the availability of public reviews from multiple conferences and years. This limited scope makes it difficult to assess whether the findings generalize beyond the selected sample. A larger and more diverse evaluation would also provide stronger evidence for the weak association between rewrite-associated reviews and rating dispersion.
  3. The evidence that rewrite-associated reviews are linked to lower reviewer disagreement is weak. The correlation is small and remains uncertain after controlling for other factors. Although the authors acknowledge this limitation, the study provides limited evidence about the relationship between LLM-associated review patterns and reviewer disagreement.
Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Misclassification, Sample Scope, and Rating Disagreement

Thank you for highlighting the reported 35.38% human-to-refined misclassification rate. In our diagnostic, the original configuration assigned rewrite-associated labels to 3 of 18 scorable historical reviews and 17 of 36 scorable controlled rewrites. The historical sample is small and differs from the contemporary reviews, so we cannot use these rates to correct the 65.9% figure.

The observational analysis still covers 300 papers from two ICLR cohorts sampled to balance accepted and rejected papers. The historical controls assess detector performance. The adjusted disagreement result remains inconclusive (p = 0.067). We also found that the two cohorts used different rating categories, including zero in 2026, and propose correcting the rating-scale description.

Reviewer_RXgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper reports a comparative study of ICLR paper reviews and argues it is incorrect to treating LLM "refinement" and LLM "generation" as one category in measurement. It applies SemDetect, an claim-level semantic detector, to official reviews linked to 300 ICLR 2025-2026 submissions, classifying each review as human-associated, rewrite-associated, AI-associated, or unscorable. The research results show that the dominant detected signature of review is refinement, and that the variance result is suggestive but inconclusive.

Critical strengths

The motivating argument is correct and well stated. Collapsing "reviewer wrote the judgment and polished the prose" together with "reviewer delegated the judgment" is a measurement error in this literature. The three-way framing is the right conceptual move. In addition, the unscorable reviews are handled properly and well explained.

Critical weaknesses

The central instrument is never validated, and the headline finding may be an artifact of the classifier rather than a fact about peer review. In addition, similar work has been reported in many papers including one paper this reviewer recently reviewed. However, the experimental study and the experimental results are relatively trivial.

Questions for authors

none

1 comment
AuthorsOpenReview ↗

Detector Validation and Scope of Contribution

Thank you for recognizing the distinction between editing a review and delegating scientific judgment. We agree that our original configuration needed validation. The new controls show that it missed most deliberately generated reviews, so we withdraw our inference about the rarity of full generation.

Our narrower contribution is the analysis of whether the fraction of rewrite-associated labels relates to disagreement among reviewers of the same paper, accounting for mean rating, review count, and cohort. Prior work has examined AI assistance in relation to scores and acceptance, and Sem-Detect has reported ICLR 2026 category frequencies. The adjusted disagreement result remains inconclusive, and we propose clarifying this scope in the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author response and additional detector validation

Thank you for asking us to validate our use of Sem-Detect. We tested the original nano-only configuration on 100 controls from 20 ICLR 2017 papers: 20 historical reviews used as a proxy for human-written reviews, 40 controlled rewrites, and 40 fully generated reviews. We kept the pretrained classifier and thresholds fixed.

The original configuration identified 1 of 40 fully generated reviews as AI-associated and 17 of 36 scorable rewrites as rewrite-associated. It also labeled 3 of 18 scorable historical reviews as rewrite-associated. Six targets were unscorable. With three reference models, full-generation recognition rose to 7 of 40. In a separate check using older model snapshots, the corresponding counts were 2 of 40 for nano and 22 of 40 for three references.

We therefore withdraw the inference that full generation was rare. The 65.9% rewrite-associated and 0.1% AI-associated figures describe detector outputs among 1,178 scorable official reviews. They do not estimate how often reviewers used LLM editing or full generation. The adjusted association with within-paper rating variance remains inconclusive (β = −1.077, 95% CI [−2.229, 0.074], p = 0.067). We propose revising the abstract and conclusion to reflect this interpretation. The PDF currently on OpenReview is the original submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.