acceptodds
Under review as a conference paper at ICLR 2027

AI-Likeness in the Peer-Review Ecology: Evaluation-Response Asymmetry

Abstract

AI-like language has been linked to favorable peer-review evaluations, but whether that association extends to the author–reviewer exchange remains unclear. We analyze 150,635 ICLR reviews from 2024–2026, linking reviews to author responses and reviewer follow-up. Using EditLens and CoCoDet as complementary continuous proxies, we benchmark their separation of matched human and directly generated reviews and compare downstream associations with earlier ICLR cohorts. Across both measurements, higher review-side AI-likeness is associated with higher ratings but less subsequent positive reviewer updating. In 2025–2026, its relation to author participation depends on the measurement: EditLens is associated with less rebuttal submission, whereas CoCoDet is associated with more. Pooled analyses generally link higher AI-likeness scores to more visible response organization, while within-paper comparisons attenuate or reverse the response-side association, suggesting that part of the pooled pattern reflects differences among papers. We call this pattern evaluation–response asymmetry: favorable evaluation does not extend uniformly to participation, organization, and reconsideration. These findings show that ratings alone do not characterize how AI-like language relates to peer review as an exchange.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
peer-review ecology, AI-likeness, evaluation–response asymmetry, author participation, author response organization, reviewer updating, measurement validation, historical controls

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QEAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between AI-like review text, review scores, and subsequent discussion using 150,635 ICLR reviews from 2024–2026. Using two AI-text detectors, the authors find that reviews with higher AI-likeness scores tend to give higher ratings but are less likely to be followed by a rating increase. They also examine whether authors respond and how many tables their responses contain, with results varying across detectors.

Critical strengths
  1. The paper includes extensive sensitivity analyses, examining how the findings change with detector choice, score transformations, and sample restrictions. These checks help clarify which results are robust and which depend on the analysis setup.
Critical weaknesses
  1. Prior work such as the ai lottery has already examined the association between AI-like reviews and higher ratings. While this paper extends the analysis to author responses and reviewer updating, these additional analyses provide relatively little new insight. Participation results are inconsistent across detectors, and the relevance of response table counts to meaningful peer-review interaction is unclear. The association with less positive updating is potentially interesting, but the analysis offers limited insight into what distinguishes these exchanges or why reviewers retain their scores.
Questions for authors

n/a

Reviewer_dMQfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how AI-like review text relates to peer-review outcomes. Using 150,635 official ICLR reviews from 2024–2026, linked to author responses and reviewer follow-ups, the authors measure AI-likeness using two existing detectors (EditLens and CoCoDet).

Four outcomes are examined: review scores, author response rates, the number of tables in responses, and post-rebuttal score increases. A one-standard-deviation increase in AI-likeness is associated with 0.03–0.22 higher review scores and a 1–3 percentage-point lower probability of score increases. Author response effects differ across detectors, while responses generally contain more tables.

The authors characterize these patterns as evaluation–response asymmetry, using 2022–2023 reviews as historical controls.

Critical strengths
  • Linking over 150,000 ICLR reviews to author responses and reviewer follow-ups provides a valuable resource. The inclusion of 2022–2023 as historical controls is thoughtful.
  • The study extends beyond initial review scores to author engagement and reviewer updates. The negative association between AI-likeness and score increases is interesting and persists after controlling for initial scores in 2024–2025.
  • Rank normalization, quadratic models, tail exclusions, within-paper comparisons, and scale transformations are examined. The authors also acknowledge limitations and generally avoid causal claims.
Critical weaknesses
  • The reported detector performance (AUROC 0.999 and 0.997) concerns the relatively easy distinction between fully human-written and fully AI-generated reviews. Performance on realistic AI-assisted or AI-edited human reviews is not established.

  • False-positive rates, calibration, and correlations between detectors on actual reviews are missing. Because detector scores are treated as continuous variables, even small differences among clearly human-written reviews influence the regressions. Some associations disappear when CoCoDet scores are converted to their original probability scale (Table 20).

  • Non-native English speakers may use LLMs for translation or language polishing, and existing detectors are known to misclassify non-native writing as AI-generated (Liang et al., 2023).

  • Differences in scoring behavior or discussion participation among these reviewer groups could explain the observed pattern of higher scores but fewer updates. The historical controls do not resolve this issue, since translation and proofreading tools were already available in 2022.

  • Detectors were selected based on historical associations, CoCoDet scaling was determined after preliminary analyses, and the EditLens implementation deviates beyond the stated tolerance from the original repository's validation scores (Appendix C).

  • Removing the top 5% of scores eliminates or reverses key 2024 associations (Table 18). Using collection-time scores, EditLens shows no score association in 2024–2025 (Table 3). Moreover, 2026 differs in scoring scale, score timing, and update measurement. In particular, score increases are inferred from Qwen-classified comments rather than actual score changes, producing substantially different rates (5.8% versus 16.6% and 22.9% in 2024 and 2025). Claims of consistent negative associations or increasing effects across years are therefore difficult to justify. The provenance of pre-discussion score snapshots for 2024–2025 is also unclear.

  • Text length is the only major control. Reviewer confidence, paper area, and review structure are omitted despite their potential influence on scores and subsequent revisions (Louis et al., 2026).

Questions for authors
  • How do the detectors perform on human-written reviews polished or translated using LLMs? What are their false-positive rates and correlations on actual reviews?
  • Can the detectors distinguish translated non-native writing from AI-generated reviews? What evidence rules out translation-related confounding?
  • Does the negative association with score updates persist when restricting the analysis to reviewers who actively participated in discussions? What fraction of 2026 reviews were classified as "no increase" due to empty replies?
  • Can the authors verify that the analyzed review texts precede rebuttal? How many were subsequently edited?
  • Do the main associations persist after controlling for reviewer confidence and paper area?
  • Do additional tables contain new experimental evidence or merely reorganize existing results?
Reviewer_vBa5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how AI-like language in ICLR reviews relates to the score a reviewer gives and to what happens afterwards. The authors collected the public ICLR Official Reviews from 2024-2026, score both review text and author-response text with two detectors, EditLens and CoCoDet. The research studied the relation among the scores on: the reviewer's rating, whether authors submitted a rebuttal, the number of Markdown tables in the response, and whether the reviewer subsequently raised the overall rating. The reported pattern evaluation-response asymmetry: higher review-side AI-likeness goes with higher ratings but with less positive reviewer updating, while author participation has no shared direction and table-count associations depend on detector and text side.

Critical strengths

The linkage among the AI likeness language, the review score, and interaction among reviewers and authors is interesting. This research links each review to its paper, to the author response, and to the reviewer's follow-up, then reports four outcomes on the same records. Historical controls are used as a real disqualification criterion ensure the soundness of the proposed experimental study.

Critical weaknesses

The asymmetry construct partly rests on disagreement between two uncalibrated proxies. Two of the four legs of the asymmetry are "the detectors point opposite ways" (participation) and "it depends on detector and text side" (organization). It notes that agreement between differently-trained detectors "is not independent replication", but it is a serious concern. There are lots of inconsistency between the text and other evidences. For example, "The rating pattern also holds with recorded ratings throughout (Appendix B).", but please check Appendix B.

Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.